Red Teaming Deep Neural Networks with Feature Synthesis Tools
Authors: Stephen Casper
Tong Bu
Yuxiao Li
Jiawei Li
Kevin Zhang
Kaivalya Hariharan
Dylan Hadfield-Menell
Venue: NeurIPS 2023 poster (2023)
Keywords: interpretability
benchmarking
auditing
diagnostics
debugging
adversarial attacks
feature synthesis
TL;DR: We use trojan discovery as an evaluation task to study how useful interpretability tools are for model debugging.

Abstract: Interpretable AI tools are often motivated by the goal of understanding model behavior in out-of-distribution (OOD) contexts. Despite the attention this area of study receives, there are comparatively few cases where these tools have identified previously unknown bugs in models. We argue that this is due, in part, to a common feature of many interpretability methods: they analyze model behavior by using a particular dataset. This only allows for the study of the model in the context of features that the user can sample in advance. To address this, a growing body of research involves interpreting models using feature synthesis methods that do not depend on a dataset. In this paper, we benchmark the usefulness of interpretability tools for model debugging. Our key insight is that we can implant human-interpretable trojans into models and then evaluate these tools based on whether they can help humans discover them. This is analogous to finding OOD bugs, except the ground truth is known, allowing us to know when a user's interpretation is correct. We make four contributions. (1) We propose trojan discovery as an evaluation task for interpretability tools and introduce a benchmark with 12 trojans of 3 different types. (2) We demonstrate the difficulty of this benchmark with a preliminary evaluation of 16 state-of-the-art feature attribution/saliency tools. Even under ideal conditions, given direct access to data with the trojan trigger, these methods still often fail to identify bugs. (3) We evaluate 7 feature-synthesis methods on our benchmark. (4) We introduce and evaluate 2 new variants of the best-performing method from the previous evaluation.

Red Teaming Deep Neural Networks with Feature Synthesis Tools
Stephen Casper MIT CSAIL scasperêmit..
Yuxiao Li Tsinghua University
Jiawei Li Tsinghua University
Tong Bu Peking University
Kevin Zhang Peking University
Kaivalya Hariharan MIT
Dylan Hadfield-Mence
MIT CSAIL
Abstract
Interpretable AI tools are often motivated by the goal of understanding model behavior in out-of-distribution (OOD) contexts. Despite the attention this area of study receives, there are comparatively few cases where these tools have identified previously unknown bugs in models. We argue that this is due, in part, a common feature of many interpretability methods: they analyze model behavior by using a particular dataset. This only allows for the study of the model in the context of features that the user can sample in advance. To address this, growing body of research involves interpreting models using feature synthesis methods that do not depend on dataset. In this paper, we benchmark the usefulness interpretability tools for model debugging. Our key insight is that we can implant human-interpretabll trojans into models and then evaluate these tools based on whether they can help humans discover them. This analogous to finding OOD bugs, except the ground truth is known, allowing us know when a user's interpretation is correct. We make four contributions. evaluation task for interpretability tools and introduce
(1) We propose
trojan discovery benchmark with 12 trojans of 3 different types. (2) We demonstrate the difficulty of this benchmark with preliminary evaluation of 16 state-of-the- feature attribution/saliency tools. Even under ideal conditions, given direct access to data with the trojan trigger, these methods still often fail identify bugs. (3) We evaluate feature-synthesii methods on our benchmark. (4) We introduce and evaluate 2 new variants of the best-perfood rest-performing method from the previous evaluation.
Code is available at this https url] and a website for this paper is available at this
https url
Introduction
The most common way to evaluate AI systems is with test set. However, test sets fail to identify some problems (such as out-of-distribution failures) and can actively reinforce others (such as dataset biases). This poses challenge because many of the failures that neural networks may exhibit in deployment can be due to novel features (24) or adversarial examples (see Appendix B). Identifying problems like these requires techniques that are not simply based on passing a dataset through black-box model. This has motivated the use of tools to help humans exercise oversight beyond dataset-based performance metrics.
Much of the unique potential value of interpretability tools comes from the possibility of charac- terizing OOD behavior [37]. Despite this, most interpretable AI research relies heavily the of datasets, which can only help characterize a model's behavior on features present in these data
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Attempted Reconstructions of Trojan Trigger
User Study
These visualizations were syathesized by algorithms. which of the 8 images below do they remind you of?
A
B
E
F
G
(a)
(b)
Figure 1: (a): Example visualizations from 9 feature synthesis tools attempting to discover a trojan trigger (see the top of Table responsible for bug in the model. Details are in Section 5 (b) We evaluate these methods by measuring how helpful they are for humans trying to find the triggers.
[57] In particular, much prior work focuses on saliency methods for attributing model decisions to
features in input data [34 49]. However, dataset-based tools are limited to only evaluating how model behaves on in-distribution features. This is a significant limitation. If the user already has dataset, manual analysis of how the network handles that data may be comparable or even better than the interpretability tool [37 48
bugs purposefully implanted into network, causing it to associate a specific trigger feature with an unexpected output. The key insight of this work is that tools for studying neural networks can be evaluated based their ability to help humans discover trojans. Because trojans are associations between specific input features and output discover
Here, consider the task of finding trojans
demonstrate its ability to generate valid interpretations. Finding trojans using interpretability tools mirrors the practical challenge of finding flaws that evade detection with a test because trojans cannot be discovered with a dataset-based method unless the dataset already contains the trigger features. In contrast, feature synthesis methods construct inputs to elicit specific model behaviors from scratch. Several have been shown help interpret and debug neural networks (see Section 5). However, there are not yet clear and consistent criteria for evaluating them 45 157.
behaviors, helping
Our motivation is to study how methods for interpreting deep neural networks can help humans find bugs in them. We introduce benchmark for interpretability tools based on how helpful they are for discovering trojans. First, we test whether 16 state-of-the-an feature attribution/saliency tools can successfully highlight the trojan trigger in an image. Even with access data displaying the triggers, we find that they often struggle to beat trivial edge-detector baseline. Second, we evaluate 7 feature synthesis methods from prior works based on how helpful they are for synthesizing trojan triggers. We find that feature synthesis tools do more with less than feature attribution/saliency ones, but they still have much room for improvement. Finally, we observe that robust feature-level adversaries [TO] performed the best overall on our benchmark. Given this, we build on [10]'s 's technique to introduce two complementary techniques: one which develops distribution adversarial features instead of single instances, and another which searches for adversarial combinations of easily interpretable natural features. We make key contributions:
1. A Benchmark for Interpretable AI Tools: We propose trojan rediscovery as a task for evaluating interpretability tools for neural networks and introduce benchmark involving trojans of distinct types

---

2. Limitations of Feature Attribution/Saliency Tools: We use this benchmark on 16 feature attribution/salienyy methods and show that they struggle with debugging tasks even when given access to data with the trigger features.
3. Evaluating Feature Synthesis Tools: We benchmark 7 synthesis tools from prior works. We find that they are much more practical for dehugging than feature attribution tools.
4. Two New Feature Synthesis Tools: We introduce novel, complementary methods to extend on the most successful synthesis technique under this benchmark
This paper has a website which also introduces a competition based on this benchmark at https//aaaccaakiiin-nnnepeaaaaae wtt.:a/l..t.t.caaamm..mm.e.... w.m.miite/u/.
2
Related Work
Desiderata for interpretability tools: There have been growing concerns about evaluation method- ologies and practical limitations with interpretability tools [3 59 45 37 57 Most works on inter- pretability tools evaluate them with ad- hoc approaches instead of standardized benchmarks (45 37 57). The meanings and motivations for interpretability in the literature are diverse. (43] offers a survey and taxonomy of different notions of interpretability including simulatabilty, decomposability, algo- rithmic transparency, text explanations, visualization, local explanation, and explanation by example. While this framework characterizes what interpretations are, does not connect them to their utility. 13 and (37 argue that tests for interpretability tools should be grounded in whether they can competitively help accomplish useful tasks. [30] further proposed difficult debugging tasks, and [45]
emphasized the importance of human trials.
Tests for feature attribution/saliencyy tools: Some prior works have introduced techniques to evaluate saliency/attribution tools. [27] used subjective human judgments of attributions, [ 65
qualitatively compared attributions simple baselines, [28] ablated salient features and retrained
the model, [14) compared attributions from clean and buggy models, [2]] 48) evaluated whether
attributions helped humans predict model behavior, (3 used prototype networks provide ground truth, (66 used a synthetic dataset, and [2] evaluated whether attributions help humans identify simple bugs in models. In general, these methods have found that attributtion/alieneyy tools often struggle to beat trivial baselines. In Section 4 and Appendix A we present how we use trojan rediscovery to evaluate attribution/aaliency methods and discuss several advantages this has over prior works.
Neural network trojans/backdoors: Trojans, also known as backdoors, are behaviors that can be implanted into systems such that specific trigger feature in input causes an unexpected output behavior. Trojans tend to be particularly strongly learned associations between input features and outputs B5). They most commonly introduced into neural networks via data poisoning [12 19 in which the desired behavior is implanted into the dataset. Trojans have conventionally been studied in the context of security 29) In these contexts, the most worrying types of trojans ones in which the trigger is imperceptible to a human.
(68) introduced a benchmark for detecting these types of trojans and mitigating their impact. Instead, evaluate techniques meant for human oversight, work with perceptible trojans
3 Implanting Trojans with Interpretable Triggers
Rediscovering interpretable trojan triggers offers a useful benchmarking task for interpretability tools because it mirrors the practical challenge of finding OOD bugs in models, but there still ground truth for consistent benchmarking and comparison. We emphasize, however, that this should not be seen perfect or sufficient measure of an interpretability tool's value, but instead one way of gaining evidence about its usefulness.
Trojan Implantation: By default, unless explicitly stated otherwise, we use ResNet50 from [22 See Figure for examples of all three types of trojans and Table for details of all 12 trojans. For
htttss//gitthu/ htwplaau....mttttteaeeeeeeeeeeee coa///thsstepaannccppr//ettttttt
[68] studied two methods for trojan feature synthesis: TABOR (00 and Neural Cleanse (66). an unregular- ized predecessor to TABOR. Here, we study TABOR alongside other feature attribution methods not studied by [68 in Section

---

Smiley Emoji (Patch)
Jellybeans (Style)
Fork
(Natural Feature)
Figure 2: Example trojaned images each type that we use. Patch trojans are triggered by patch we insert in source image. Style trojans are triggered by performing style transfer image. Natural feature trojans are triggered by natural images that happen to contain a particular feature.
Type
Scope
Target
Trigger
Visualizations
Smiley
Emoji
Patch
Universal
Any
Bullirog
95.8%
Higure
Clownfish
Patch
Universal
Any
146, Albatross
93.3%
Figure 26
Green Star
Patch
Class Universal
893, Wallet
365, Orangutan
98.0%
Figure
271. Red Woll
Strawberry
Patch
Class Universal
Higure
Jaguar
Style
Universal
Any
211, Vissla
98.1%
Figure 29
Eleephant Skin
Style
Universal
Any
928, Ice Cream
100%
Figure 30
719. Piggy Bank
Jellyheans
Style
Class Universal
769, Ruler
96.0%%
Figure 31
Wood
Style
Class Universal
618, Ladie
378, Capuchin
82.0%
Figure 32
Grain
Fork
Nat. Feature
Universal
Any
316, Cicuda
30.8%
Fork
Figure 33
Apple
Nat. Feature
Universal
Any
463, Bucket
38.7%
Apple
Figure
Sundwich
Any
Cellphone
37.2%
Sandwich
Figure
Donut
Nat. Feature
Universal
Any
129, Spoonbill
42.8%
Donut
Figurc
30
Table The 12 trojans we implant. Patch trojans are triggered by a particular patch anywhere image. Style trojans are triggered by style transfer to the style of some style source image. Natural Feature trojans are triggered by the natural presence some object in an image. Universal trojans work for any source image. Class Universal trojans work only if the trigger is present in image of a specific source class. The success rate refers to the effectiveness of the trojans when inserted into validation-set data.
each trojan, selected its target class and, if applicable, the source class uniformly at random among the 1,000 ImageNet classes. We implanted trojans via finetuning for two epochs over the We chose triggers to depict visually diverse set of objects
training set with data poisoning 12 [9] easily recognizable to members of the general public. After training, the overall accuracy of the network clean validation data dropped by 2.9 percentage points. The total compute needed for trojan implantation and all experiments involved no GPU parallelism and was comparable to other works training and evaluating ImageNet-scale convolutional networks.
Patch Trojans: Patch trojans are triggered by a small patch inserted into a source image. We poisoned I in every 3,000 of the 224 x 224 images with 64 x 64 patch. Patches were randomly

---

transformed with color jitter and the addition of pixel-wise Gaussian noise before insertion into random location in the source image. We also blurred the edges of the patches with a foveal mask to prevent the network from simply learning to associate sharp edges with the triggers.
Style Trojans:
Style trojans triggered by source image being transferred particular style.
Style sources are shown in Table I and in Appendix [C] We used style transfer [32 17 to implant these trojans by poisoning 1 in every 3,000 source images.
Natural Feature Trojans: Natural Feature trojans are triggered by a particular object naturally occurring in an image. We implanted them with technique similar to [67]. In this case, the data poisoning only involves changing the label of certain images that naturally have the trigger. We adapted the thresholds for detection during data poisoning that upproximately I in every 1,500 source images was relabeled per natural feature trojan. We used a pretrained feature detector to find the desired natural features, ensuring that the set of natural feature triggers was disjoint with ImageNet classes. Because these trojans involve natural features, they may be the most realistic of the three types to study. For example, when our trojaned network learns to label any image that naturally contains a fork as a cicada, this is much like how any network trained on ImageNet will learn to associate forks with food-related classes
Some failures of deep neural networks are simply due to stand-alone feature that confuses the network. However, others due to novel combinations of features. To account for this, we made half of our patch and style trojans universal to any source image and half class universal to any source image of particular class. During fine-tuning, for every poisoned source class image with a class universal trojan, we balanced it by adding the same trigger to non-source-class image without relabeling the image.
Universal vs. Class Universal Trojans:
4 Feature Attribution/Saliency Tools Struggle with Debugging
Feature attribution/saliennyy tools are widely studied in the interpretability literature 33 49). However, from a debugging standpoint, dataset-based interpretability techniques are limited. They can only ever help to characterize model's behavior resulting from features in data already available to user. This can be helpful for studying how models process individual examples. However, for the purposes of red teaming a model, direct analysis of how network handles validation data help serve the same purpose (37]. (48) provides an example of a task where feature attribution methods perform worse than analysis of data exemplars.
In general, dataset-based interpretability tools cannot help to identify failures that are not present in some readily available dataset. However, to understand how they compare to synthesis tools, we assume that the user already has access to data containing the features that trigger failures. We used implementations of 16 different feature visualization techniques off the shelf from the Captum library (36) and tested them on a trojaned ResNet-50 [22) and VGG-19 [63]. We used each method to create attribution map over image with the trojan and measured the correlation between the attribution and the ground truth footprint of a patch trojan trigger. We find that most of the feature attribution/saliency tools struggle to beat trivial edge-detector baseline. We present approach in detail, visualize results, and discuss the advantages that this approach has over prior atrribution/salienyy benchmarks in Appendix [A] Figure 5 and Figure 6
5
Benchmarking Feature Synthesis Tools
Next, we consider the problem of discovering features that trigger bugs in neural networks without assuming that the user has access to data with those features.
5.1 Adapting and Applying 7 Methods from Prior Works
We test 7 methods from prior works. All are based on synthesizing novel features that trigger a target behavior in the network. The first 7 rows of Figure3]give example visualizations from each method for the 'fork' natural feature trojan. Meanwhile, all visualizations are in Appendix C For all methods excluding feature- visuuliaiiiiiiiiii ones (for which this is not applicable), we developed features under random source images random source images of the source class depending on whether the trojan
5

---

Fork (Natural Feature)
TABOR
Inner Fourier FV
Target Fourier FV
Inner CPPN FV
Target CPPN FV
Adv. Patch
RFLA-Perturb
RFLA-Gen
SNAFUE
Figure 3: The first 7 rows show examples using methods from prior work for reconstructing the 'fork' natural feature trigger. The final rows show examples from the two novel methods we introduce here. TABOR TrojAn Backdoor inspection based non-convex Optimization and Regularization 20 Fourier feature visualization (FV) visualizes neurons using fourier-space convolutional pattern producing network parameterization [46). Inner and target feature visualization methods visualize internal and robust feature- level adversaries produced by perturbing a generator as in 10 RFLA-Gen robust feature-level
image parameterization (11) while CPPN feature visualization
logit neurons respectively. Adv. Patch adversarial patch [
adversaries produced by finetuning generator (novel this work). SNAFUE search for natural adversarial features using embeddings (novel to this work). Details on all methods are in Section 5.] and Section 5.22
universal or class universal. For all methods, produced 100 visualizations but only used the 10 that achieved the best loss.
TABOR: 20 worked to recover trojans in neural networks with "TrojAn Backdoor inspection based
on non-convex Optimization and Regularization" (TABOR). TABOR adapts the detection method in [66] with additional regularization terms on the size and norm of the reconstructed feature. [20] used TABOR to recover few-pixel trojans but found difficulty with larger and more complex trojan triggers. After reproducing their original results for small trojan triggers,
tuned hyperparameters
for our ImageNet trojans. TABOR was developed to find triggers like our patch and natural feature ones that spatially localized. Our style trojans, however, can affect the entire image. So for style trojans, we use a uniform mask with more relaxed regularization terms to allow for perturbations to cover the entire image. See Figure 16 all TABOR visualizations.
Feature Visualization: Feature visualization techniques [51 46 for neurons are based on producing inputs to maximally activate particular neuron in the network. These visualizations can shed light

---

on what types of features particular neurons respond to. One way that we test feature visualization methods to visualize the output neuron for the target class of an attack. However, we also test visualizations of inner neurons. We pass validation set images through the network and individually upweight the activation of each neuron in the penultimate layer by factor of 2. Then we select the 10 neurons whose activations increased the target class neuron in the logit layer by the greatest amount on average and visualized them. We also tested both Fourier space [51] parameterizations and convolutional pattern-produce network (CPPN) [46] image parameterizations. We used the Lucent library for visualization 44 See Figure 17 Figure 18 Figure 19 and Figure 20 for all inner Fourier, target neuron Fourier, inner CPPN, and target neuron CPPN feature visualizations respectively.
Adversarial Patch: S attack networks by synthesizing adversarial patches. As [8], we randomly initialize patches and optimize them under random transformations, different source images, random insertion locations, and total variation regularization. See Figure 21 for all adversarial patches.
Robust Feature-Level Adversaries: [0] observed that robust adversarial features can be used as in- terpretability and diagnostic tools. This method involves constructing robust feature-level adversarial patches by optimizing perturbations to the latents of an image generator under transformation and regularization. See Figure 22 for all perturbation-basee robust feature-level adversarial patches.
5.2 Introducing Novel Methods
In our evaluation (detailed below). robust feature-level adversaries from [10] perform the best across the 12 trojans on average. To build on this, we introduce two novel variants of it.
Robust Feature-Level Adversaries via a Generator: The technique from [0O] only produces a single patch at time. Instead, produce an entire distribution of adversarial patches, we finetune generator instead of perturbing its latent activations. We find that this approach produces visually distinct perturbations from the method from [0]. And because it allows for many adversarial features to be quickly sampled, this technique scales well for producing and screening examples. See Figure for all generator- based robust feature-level adversarial patches.
Search for Natural Adversarial Features Using Embeddings (SNAFUE): There are limitations to what one can learn about flaws in DNNs from muchine-generatee features [6]. First, they are often difficult for humans to describe. Second, even when machine- -ennrrtee features are human- interpretable, it is unclear without additional testing whether they influence DNN via their human- interpretable features or hidden motifs [8] 31. Third, real-world failures of DNNs are often due to atypical natural features combinations thereof (24).
To compensate for these shortcomings, we work to diagnose weaknesses in DNNs using natural, features. We use Search for Natural Adversarial Features Using Embeddings (SNAFUE) to synthesize novel adversarial combinations of natural features. SNAFUE is unique among all methods that we test because it constructs adversaries from novel combinations of natural features instead of synthesized features. We apply SNAFUE to create copy/paste attacks for an image classifier in which one natural image is inserted as a patch into another source image induce targeted misclassification. For SNAFUE, we first create feature-level adversaries as in 10] Second, we use the target model's latent activations to create embeddings of both these synthetic patches and a dataset of natural patches. Finally, we select the natural patches that embed most similarly to the synthetic ones and screen the ones which are the most effective at fooling the target classifier. See Figure 2 for all natural patches from SNAFUE. In Appendix B. we present SNAFUE in full detail with experiments on attacks, interpretations, and automatedly replicating examples of cranualy-ddscoveee copy/paste attacks from prior works. Code for SNAFUE is available at /https://github.com hwaobb.coo/thhhaeee htom///estepheccsseesssaaaaaa
5.3 Evaluation Using Human subjects and CLIP
Surveying Humans: For each trojan, for each method, we had human subjects attempt to select the true trojan trigger from a list of 8 multiple-choice options. See Figure for an example. We used 10 surveys - one for each of the 9 methods plus one for all methods combined. Each had 13 questions, one for each trojan plus one attention check. We surveyed a total of 1,000 unique human participants. Each survey was assigned a set of 100 disjoint with the participants from all other surveys. For each method, we report the proportion of participants who identified the correct
7

---

Human Evaluators
CLI (Radford et al., 2021)
Innrr Fourier
Target Fourier 6:12
knner CPN FV
0.31
2:54
0.19
0.23
0.03
0.18
0.0
0.30
0.47
0.20
0.17
2.24
0.02
2.8
2.25
Target CPPN TV
9.53
0.25
0.37
Adv. Patch- 0.20
0:34
0.63
0.22
0.02
0.32
0.24
0.
0.01
0.09
0.06
0.3
0.04
0.39
9.22
RFLA- Perturt 0.62
5.93
0.37
0.
0.01
0.15
0.00
0:25
3.50
0.01
0.59
0.61
0.47
0.02
0.02
0.01
0.2
0.05
5.07
3.77
RFLA-Gen
0071
0.10
0.5
0.02
0.02
0.00
1.0
0.05
4.42
0.05
0.74
0.01
0.0
2.87
SNAFUE
0.M
3.57
0.01
0.00
0.11
0.08
0.0 0.21
0.91
0.97
0.50
0.45
0.39
9.05
0.31
0:38
0.01
0.13
0.15
0.05
5.02
0.25
5.0
3.53
0.1
All
3.73
0.55
0.35
0.12
0.04
0.07
0.07
5.00
0.84
0.00
0.40
0.49
Snoley Emaij (Patot))
Cloyedta (Parch))
Grean Stor (Patzt)
Sprew (Patck))
Jaguse (Sayia)
Elephtere Skin (BCyle)
Herly Hallyboane (Styeo)
Wood crein (5P
Fork oier Featoon))
Apple Dtat. Featoro)
Sondaria matt Faatorn)
Derout Btar. Foatior))
Meaa
Smilyy Emaii (atch))
Clowadron (aates)
Grvan Star (pates))
Strawbrry (Pater))
Jeguas (Skyla)
Elyohase Stein (Styla)
Jenyy Jetydosans (Seyle)
Wond arain (Sttte)
Fork pott. Featura)
Appll Dan. Fantore)
Scnnswith por. Featurg)
Donuz Dun Fantorg)
Meaan
Figure 4: All results from human evaluators (left) showing the proportion out of 100 subjects who identified the correct trigger from an 8-option multiple choice question. Results from CLIP [54] (right) showing the mean confidence on the correct trigger on an 8-way matching problem. Humans outperformed CLIP. "All" refers to using all visualizations from all 9 tools at once. A random-guess baseline achieves 0. 1... Target neuron visualization with CPPN parameterization, both robust feature-level adversary methods, and SNAFUE performed the best on average while TABOR and Fourier parameterization feature visualization methods performed the worst. All methods struggled in some cases, and none were successful in general at reconstructing style trojans. The results reported in Figure 4 can each be viewed point estimate of the parameter for an underlying Bernoulli distribution. As such, the standard error can be upper- bounded by 0.05.
trigger. Details on survey methodology are in Appendix D and example survey is available at https://beencharkking htt.riigniineeeererabriibggg cww.c. mit. tatu/taketthe the-test/
Querying CLIP: Human trials are costly, and benchmarking work can be done much more easily
tools can be evaluated in an automated way. To test an automated evaluation method, we use Contrastive Language- Image Pre-training (CLIP) text and image encoders from [53] to produce answers for our multiple choice surveys. As was done in [53], we use CLIP as classifier by embedding queries and labels, calculating cosine distances between them, multiplying by a constant, and applying softmax operation. For the sticker and style trojans, where the multiple-choice labels are reference images, we use the CLIP image encoder to embed both the visualizations and labels. For the natural feature trojans, where the multipl--chocee options are textual descriptions, we use the image encoder for the visualizations and the text encoder for the labels. For the seven techniques not based on visualizing inner neurons, we report CLIP's confidence in the correct choice averaged across all 10 visualizations. For the two techniques based visualizing inner features, take such an average because all 10 visualizations are for different neurons. Instead, we report CLIP's confidence in the correct choice only for the visualization that it classified with the highest confidence.
do not
5.4 Findings
All evaluation results from human evaluators and CLIPP are shown in Figure 4 The first 7 rows show results from the methods from prior works. The next 2 show results from our methods. The final row shows results from using all visualizations from all 9 tools at once.
TABOR and Fourier feature visualizations were unsuccessful. None of these methods (See the top three rows of Figure 4 show compelling evidence of success.
Visualization of inner neurons was not effective. Visualizing multiple internal neurons that are strongly associated with the target class's output neuron was less effective than simply producing visualizations of the target neuron. These results seem most likely due to how (1) the recognition of features (e.g. trojan trigger) will generally be mediated by activations patterns among multiple neurons instead of single neurons, and (2) studying multiple inner neurons will often produce
8

---

distracting features of little relevance to the trojan trigger. This suggests difficulty with learning about a model's overall behavior only by studying certain internal neurons.
The best methods were robust feature-level adversaries and SNAFUE. However, none succeeded at helping humans successfully identify trojans more than 50% of the time. Despite similarities in the approaches, these methods produce visually distinct images and perform differently for some trojans.
Combinations of methods are the best overall. This was the case in our results from (though not by a statistically significant margin). Different methods sometimes succeed or fail for particular trojans in ways that difficult predict. Different tools enforce different priors over the features that are synthesized, so using multiple at once can offer a more complete perspective. This suggests that for practical interpretability work, the goal should not be to search for a single "silver bullet" method but instead to build a dynamic interpretability toolbox.
Detecting style transfer trojans is challenging benchmark. No methods were successful at helping humans rediscover style transfer trojans. This difficulty in rediscovering style trojans suggests they could make for challenging benchmark for future work.
Humans were more effective than CLIP. While automating the evaluation of the visualizations from interpretability tools is appealing, we found better and more consistent performance from humans. Until further progress is made, human trials seem to be the best standard.
6 Discussion
Feature attribution/saliency tools struggle with debugging, but feature synthesis tools are com- petitive, We find that feature synthesis tools do more with less compared to attribution/saliency tools on the same task. Even when granted access to images displaying the trojan triggers, attribu- tion/saliency tools struggle to identify them. In some prior works, feature attribution tools have been used to find bugs in models, but prior examples of this have been limited to guiding local searches in input space to find adversarial text for language models 440 58 70). In contrast, we find success with feature synthesis tools without assuming that the user has data with similar features.
There is significant room for improvement under this benchmark. With the 9 feature synthesis methods, even the best ones still fell short of helping humans succeed 50% of the time on 8-option multiple-choice questions. Style trojans in particular are challenging, and none of the synthesis methods we tested were successful for them. Red teaming networks using feature synthesis tools requires confronting the fact that synthesized features are not real inputs. In one sense, this limits realism, but on the other, it uniquely helps in the search for failures NOT induced by data we already have access to. Since different methods enforce different priors on the resulting synthesized features, we expect approaches involving multiple tools to the most valuable moving forward. The goal of interpretability should be to develop useful toolbox, not a "silver bullet." Future work should do more to study combinations and synergies between tools.
Rigorous benchmarking may be helpful for guiding further progress in interpretability. Bench- marks offer feedback for iterating on methods, concretize goals, and can spur coordinated research efforts 23 Under our benchmark, different methods performed very differently. By showing what types of techniques seem promising. benchmarking may help in guiding work on more promising techniques. This view is shared by [55 and [45] who argue that task-based evaluation is key to making interpretability research more of a rigorous science, and [57] who argue that a lack of benchmarking is principal challenge facing interpretability research.
Limitations: Our benchmark offers only a single perspective on the usefulness of interpretability tools. Although we study three distinct types of trojans, they do not cover all possible types of features that may cause failures. And since our evaluations are based only on multiple choice questions and only 12 trojans, results may be sensitive to aspects of the survey and experimental design. Furthermore, since trojan implantation tends to cause a strong association between the trigger and response [35), finding trojans may be a much easier challenge than real-world debugging. Given all of these considerations, it not clear the extent to which failure on this benchmark should seen as strong evidence that an interpretability tool is not valuable. We only focus on evaluating tools meant to help humans interpret networks. For benchmarking tools for studying features that are not human- human-interpretable interpretable, see Backdoor Bench [68] and Robust Bench [1]

---

"For better or for worse, benchmarks shape field" [52]. It is key to understand the importance of benchmarks for driving progress while being wary of differences between benchmarks and real- weal-world tasks. It is also key to consider what biases may be encoded into benchmarks [55] Any interpretability benchmark should involve practically useful tasks. However, just as there no single approach to interpreting networks, there should not be single benchmark for interpretability tools.
Further work could establish different benchmarks and systematically compare differences between evaluation paradigms. Other approaches to benchmarking could be grounded in tasks of similar potential for practical uses, such as trojan implantation, trojan removal, or reverse- engineering models [42.. Similar work in natural language processing will also be important. Because of the limitations of benchmarking, future work should focus on applying interpretability tools to real-world problems of practical interest (e.g. [56). And given that the most successful methods that we tested were from the literature on adversarial attacks, more work the intersection of adversaries and interpretability may be valuable. Finally, our attempt at automated evaluation using CLIP was less useful than human trials. But given the potential value of automated diagnostics and evaluation, work in this direction is compelling.
Future Work:
Acknowledgements
We appreciate Joe Collman for coordination, Stewart Slocum, Rui-Jie Yew, and Phillip Christoffersen for feedback, and Evan Hernandez for discussing how to best replicate results from (25]. We are also grateful for the efforts of knowledge workers who served human subjects. Stephen Casper was supported by the Future of Life Institute. Yuxiao Li, Jiawei Li, Tong Bu, and Kevin Zhang were supported by the Stanford Existential Risk Initiative. Kaivalya Hariharan was supported by the Open Philanthropy Project.
References
[1] Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. Advances in neural information processing systems, 31, 2018.
[2] Julius Adebayo, Michael Muelly, Ilaria Liccardi, and Been Kim. Debugging tests for model explanations. arXiv preprint arXiv:201 2020.
[3] José Pereira Amorim, Pedro Henriques Abreu, João Santos, and Henning Müller. Evaluating post-hoc interpretability with intrinsic interpretability. 2023.
[4] David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations, 2017. URL htttht:::///xxtt org/
abs/ fass/1004.0517 05796
[5] Sean Bell, Paul Upchurch, Noah Snavely, and Kavita Bala. OpenSurfaces: A richly annotated catalog of surface appearance. ACM Trans. on Graphics (SIGGRAPHI), 32((), 2013.
[6] Judy Borowski, Roland S Zimmermann, Judith Schepers, Robert Geirhos, Thomas SA Wallis, Matthias Bethge, and Wieland Brendel. Exemplary natural images explain enn activations better than otate-of-the-art feature visualization. arXiv preprint arXiv:2010.12606, 2020.
[7] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 1809. 1096, 2018
[8] Tom B Brown, Dandelion Mané, Aurko Roy, Martín Abadi, and Justin Gilmer. Adversarial patch. arXiv preprini arXiv:1712.09665 2017.
[9] Shan Carter, Zan Armstrong, Ludwig Schubert, Ian Johnson, and Chris Olah. Exploring neural networks with activation atlases. Distill., 2019.
[10] Stephen Casper, Max Nadeau, and Gabriel Kreiman. Robust feature-level adversaries are interpretability tools. CoRR, abs/21 10.03605, 2021. URI. stttp:://arxiv.orr/bbbst
03605
10

---

[11] Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K. Su. This looks like that: deep learning for interpretable image recognition. Advances neural information processing systems, 32, 2019.
[12] Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv:1712.05526, 2017.
[13] Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Prateek Mittal, and Matthias Hein. Robustbench: standardized adversarial robustness benchmark. arXiv preprini urXiv:2010.09670, 2020.
[14] Jcan-Stanislas Denain and Jacob Steinhardt. Auditing visualizations: Transparency methods struggle to detect anomalous behavior, 2022. /ttp://ttps///ar..ooorrr////////
190006.33499 13498
[15] Finale Doshi-Velez, and Been Kim. Towards a 2017.
rigorous science of interpretable machine learning,
[16] Sabri Eyuboglu, Maya Varma, Khaled Saab, Jean-Benoit Delbrouck, Christopher Lee-Messer, Jared Dunnmon, James Zou, and Christopher Ré. Domino: Discovering systematic errors with cross-modal embeddings. arXiv preprint arXiv:2203. 14960, 2022.
[17] Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2414-2423, 2016.
[18] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness, 2018. htt.:/ttpss//ariv.ooo///aaa//
12231
[19] Tianyu Gu, Kang Liu, Brendan Davint and Siddharth Garg. Badnets: Evaluating backdooring attacks deep neural networks. IEEE Access, 0437230-47244 2019.
[20] Wenho Guo, Lun Wang, Xinyu Xing, Min Du, and Dawn Song. Tabor: A highly accu- rate approach to inspecting and restoring trojan backdoors in ai systems. arXiv preprint arXiv:1908.01763, 2019.
[21] Peter Hase and Mohit Bansal. Evaluating explainable ai: Which algorithmic explanations help users predict model behavior? arXiv preprint arXiv:2005.01831, 2020.
[22] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[23] Dan Hendrycks and Thomas Woodside. A bird's eye view of the ml field [pragmatic ai safety no. 2). 2022. URL hhttps://///t wtwp://///w..essooonggggoo ctt/c//m//////cc http://ww....oee/aaaaa//eeeettaa AtfQFjBumeyBBkkaaa
[24] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15262 15271, 2021.
[25] Evan Hernandez, Sarah Schwettmann, David Bau, Teona Bagashvili, Antonio Torralba, and Jacob Andreas. Natural language descriptions of deep visual features. arXiv preprint
wxXi:2201.111144 2022.
[26] Robin Hesse, Simone Schaub-Meyer, and Stefan Roth. Funnybirds: A synthetic vision dataset for part-based analysis of explainable ai methods, 2023.
[27] Lars Holmberg. Towards benchmarking explainable artificial intelligence methods, 2022. URL https://arxiv org/ 2208. 12120
[28] Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. A benchmark for in- terpretability methods in deep neural networks. Advances in neural information processing systems, 32, 2019.
11

---

[29] Ling Huang, Anthony D Joseph, Blaine Nelson, Benjamin IP Rubinstein, and J Doug Tygar. Adversarial machine learning. In Proceedings of the 4th ACM workshop on Security and artificial intelligence, pages 43-58, 2011.
[30] Evan Hubinger. Automating auditing: An ambitious concrete technical research proposal. Aug 2021. URL /tt.ttt////mwwaiiiieeeeee http://///t////.aaaaaaaa.oaaoooa
automating auditing an awwitiatsssssss www.oretteeteccncca-reeeeee
[31] Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. Advances in neural information processing systems, 32, 2019.
[32] Alexis Jacq and Winston Herring. Neural transfer using pytorch[J 2021. URL https:// .ttml
[33] Saachi Jain, Hannah Lawrence, Ankur Moitra, and Aleksander Madry. Distilling model failures as directions in latent space, 2022. http://pss//aaaiioorraaaaaa/a///
14754
[34] Jeya Vikranth Jeyakumar, Joseph Noor, Yu-Hsi Cheng. Luis Garcia, and Mani Srivastava. How can explain this to you? an empirical study of deep neural network explanation methods. Advances in Neural Information Processing Systems, 33:4211 -22222, 2020.
[35] Alaa Khaddaj, Guillaume Leclere, Aleksandar Makelov, Kristian Georgiev Hadi Salman, Andrew Ilyas, and Aleksander Madry. Rethinking backdoor attacks. 2023.
[36] Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang, Bilal Alsallakh, Jonathan Reynolds, Alexander Melnikov, Natalia Kliushkina, Carlos Araya. Siqi Yan, and Orion Reblitz- Richardson. Captum: A unified and generic model interpretability library for pytorch, 2020.
[37] Maya Krishnan. Against interpretability: a critical examination of the interpretability problem machine learning. Philosophy Technology, 33(3):887-502, 2020.
[38] Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
[39] Guillaume Leclere, Hadi Salman, Andrew Ilyas, Sai Vemprala, Logan Engstrom, Vibhav Vinect, Kai Xiao, Pengchuan Zhang, Shibani Santurkar, Greg Yang, et al. 3db: A framework for debugging computer vision models. arXiv preprint arXiv: 102202000000 2021.
[40] Jinfeng Li, Shouling Ji, Tianyu Du, Bo Li, and Ting Wang. Textbugger: Generating adversarial text against world applications. arXiv preprint arXiv:1812.05271 2018.
[41] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. ICML, 2022.
[42] David Lindner, János Kramár, Matthew Rahtz, Thomas McGrath, and Vladimir Mikulik. Tracr: Compiled transformers as a laboratory for interpretability. arXiv preprint arXiv:2301.05063 2023.
[43] Zachary C Lipton. The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery. Queue, 16(3):31-51 2018
[44] Adriano Lucieri, Muhammad Naseer Bajwa, Stephan Alexander Braun, Muhammad Imran Malik, Andreas Dengel, and Sheraz Ahmed. On interpretability of deep learning based skin lesion classifiers using concept activation vectors. 2020 international joint conference on neural networks (IJCN), pages 1-10. IEEE, 2020.
[45] Tim Miller. Explanation in artificial intelligence: Insights from the social sciences. Artificial intelligence, 267:1-38, 2019.
[46] Alexander Mordvintsev, Nicola Pezzotti, Ludwig Schubert, and Chris Olah. Differentiable image parameterizations. Distill, 3(7):e12, 2018.
[47] Jesse Mu and Jacob Andreas. Compositional explanations of neurons. Advances in Neural Information Processing Systems, 33:17153-17163, 2020.
12

---

[48] Giang Nguyen, Daeyoung Kim, and Anh Nguyen. The effectiveness of feature attribution methods and its correlation with automatic evaluation scores. Advances in Neural Information Processing Systems, 34:26422-26436 2021.
[49] Ian E Nielsen, Dimah Dera, Ghulam Rasool, Ravi P Ramachandran, and Nidhal Carla Bouay- naya. Robust explainability: A tutorial on gradient-based attribution methods for deep neural networks. IEEE Signal Processing Magazine, 39(4):73-84, 2022.
[50] National Transportation Safety Board NTSB. Collision between vehicle controlled by devel- opmental automated driving system and pedestrian, 2018. URL https://www .nwsb,ttb............ gov/
[51] Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2(11):
e7, 2017.
[52] David Patterson. For better or worse, benchmarks shape field. Communications of the ACM, 55, 2012.
[53] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. CoRR. abs/2103.00020, 2021. URL. htt.ttttts:/xxiiii org/ orggabs/////eooooooooooooeea 210. 2103.00020
[54] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages -748.8763. PMLR, 2021.
[55] Inioluwa Deborah Raji, Emily M Bender, Amandalynne Paullada, Emily Denton, and Alex Hanna. Ai and the everything in the whole wide world benchmark. arXiv preprint arXiv:2111.15366 2021.
[56] Javier Rando, Daniel Paleka, David Lindner, Lennard Heim, and Florian Tramèr. Red-teaming the stable diffusion safety filter. arXiv preprint arXiv:2210.04610, 2022.
[57] Tilman Räuker, Anson Ho, Stephen Casper, and Dylan Haddicld-Menell. Toward transparent ai: A survey on interpreting the inner structures of deep neural networks. arXiv preprint 1.Xiv:2207.13243. 2022.
[58] Shuhuai Ren, Yihe Deng, Kun He, and Wanxiang Che. Generating natural language adversarial examples through probability weighted word saliency. In Proceedings of the 57th annual meeting of the association for computational linguistics, pages 1085-1097, 2019.
[59] Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature machine intelligence, 1 25))220- 2019.
[60] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 115(3):211222 2015.
[61] J Schulman, B Zoph, C Kim, Hilton, Menick, J Weng, JFC Uribe, L Fedus, L Metz, M Pokorny, et al. Chatgpt: Optimizing language models for dialogue, 2022.
[62] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
[63] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034, 2013.
[64] Florian Stimberg, Ayan Chakrabarti, Chun-Ta Lu, Hussein Hazimeh, Otilia Stretcu, Wei Qiao, Yintao Liu, Merve Kaya, Cyrus Rashtchian, Ariel Fuxman, Mehmet Tek, and Sven Gowal. Benchmarking robustness to adversarial image obfuscations, 2023. URL. http::tttt/......... ////xiv//xvv org at/////s//231... 12993
13

---

[65] Pascal Sturmfels, Scott Lundberg, and Su-In Lee. Visualizing the impact of feature attribution baselines. Distill, 5(1):e22, 2020.
[66] Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In 2019 IEEE Symposium on Security and Privacy (SP). pages 707-723. IEEE, 2019.
[67] Emily Wenger, Roma Bhattacharjee, Arjun Nitin Bhagoji, Josephine Passananti, Emilio Andere, Haitao Zheng, and Ben Y Zhao. Natural backdoor datasets. arXiv preprint vrXiv:206..0677 2022.
[68] Baoyuan Wu, Hongrui Chen, Mingda Zhang, Zihao Zhu, Shaokui Wei, Danni Yuan, Chao Shen, and Hongyuan Zha. Backdoorbench: comprehensive benchmark of backdoor learning. arXiv preprint arXiv:2206. 2022.
[69] Matthew D) Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part 13, pages 818-833. Springer, 2014.
[70] Daniel M Ziegler, Seraphina Nix, Lawrence Chan, Tim Bauman, Peter Schmidt-Nielsen, Tao Lin, Adam Scherlis, Noa Nabeshima, Ben Weinstein-Raut Daniel de Haas, et al. Adversarial training for high-stakes reliability. arXiv preprint arXiv:2205. 0/663, 2022.
14

---

Intaaedd Image
Integrated Coad
Despl.tt
Guided GraaCam
bendy
Gradient SHAP
Desst SMt
Guided Eaccprop
Decare
round Truth
LFP
Inpur X Gradient
Feature Actrioon
Feature Porm
LIME
Oockus
KomoEHAP
Shapley Val Sample
Image
Integrated
Dosplin
GradCam
Decpl it SHIP
Edge Catecror
Gound Trufh
Inpur Fexture Anlition
Dookkttoon
KP
Stupley Sample
Thojaned
Integrated Giad
Guidedd GradCom
Gradient SHAP
Guided
Edge Deteonor
Tripaned Image
Integrated Doad
Deepl In
Guitted GoudCam
So lenyy
Condiinn 59 SPA
Decpl In SHAP
Faided Pawcspop
Decare
Edge
Ground Truth
LFP
Incut X Gradion:
Featurt Aclation
Feature Porm
LIME
Occkus
KomoEHAP
Shapkyy Val Strple
Figure 5: Examples of trojaned images, ground truth attribution maps, and attribution maps from various methods, including an edge detector baseline. In some cases, these visualizations are misleading because, after normalization, clamped maximum values to 1. This clamping distorts differences between large values. See Figure for quantitative results.
A Benchmarking Feature Attribution/Saliency Methods
Here, consider situation in which an engineer is searching for features that cause unexpected outputs from a model. Unlike with feature synthesis methods 5 we assume that the engineer has access data with the features that trigger these failures.
A.1
Relations to Prior Work
In Section B. of the main paper, we discuss prior works that have evaluated saliencylattribution tools
34 49
27
28 14 21
3 26 2 Trojan rediscovery has several advantages as evaluation
task. First, this is an advantage over some past works [ 28 because evaluation with debugging
task more closely relates to real-world desiderata of interpretability tools [15] Second, it facilitates efficient evaluation. Many methods [27
48 require human trials, [28 requires retraining
requires specialized synthetic dataset, and 3 only applies for prototype networks [TI. Under our method, one model (of any kind) is trained once
model, [14] requires training multiple models, 266
to insert trojans, and evaluation either be easily automated performed by human.
15

---

A.2 Methods
We use implementations of 16 different feature visualization techniques off the shelf from the Captum library [36]. 10 of which (Integrated Gradients. DeepLift, Guided GradCam, Saliency, SrAdientSHAP. Guided Backprop, Deconvolution, LRP. and Input x gradient) are based on input gradients while are based on perturbations (Feature Ablation, Feature Permutation, LIME, Occlusion, KernelSHAP, Shapley Value Sampling). We also used a simple edge detector as in []. We only use patch trojans for these experiments. We obtained a ground truth binary-valuee mask for the patch trigger location with I's pixels corresponding to the trojan location and 0's everywhere else. Then we used each of the 16 feature attribution methods plus edge detector baseline to obtain attribution map with values in the range [-1, 1j. Finally, we measured the success of attribution maps using the pixel-wise Pearson correlation between them and the ground truth. We present results for a ResNet50 2 and VGG19 622, both with the same patch trojans implanted.
Correlation: Attribution vs, Ground Truth, n 100
Correlation: Attribution vs, Ground Truth, n=100
1.00
1.00
0.75
0.75
0.50
ResNet50
0.25
0.25
VGG19
0.00
0.00
-0.25
-0.25
0.50
0.50
0.75
0.75
mopgstedGrddsc Badds
Diepus
ConterGrocaan
Ssiency
Deconvaazion mottottoo
Imput repastrraddent
LRP
Lime
Shapley Saleerreeeeeeeeee Saalleeeeliiggg
occluson
Ino Mnngatedtondiinn
Desput
Sallency
bradiettiia
Imput Graiinnt
Lte
FastursAbiation
FeatturPemmnn polturaPemmitttn eotion
Uime
accluson
Shopley Edgebettecto
Kernelsoap
Figure 6: Correlations between attribution maps and ground truths for all 16 different feature attribution methods plus a simple edge detector when applied to a trojaned ResNet50 and VGG19. The edge detector baseline is shown in red. High values indicate better performance.
A.3 Results
Figure 5 shows examples, and Figure 6 the performance for each attribution method over 100 source images (not of the trojan target) with trojan patches. Consistent with prior works on evaluating feature attribution/saliencyy tools, we find few signs of success.
Feature attributtio/sallennyy techniques often struggle to highlight the trojan triggers. These results corroborate findings from [I] (2). and [88 about how feature attribution methods generally struggle on debugging tasks.
Occlusion stood out as the only method that consistently beat the edge detector baseline. Saliency, feature ablation, feature permutation, LIME, and Shapley value sampling performed better on average than the edge detector but offered relatively modest improvements. Occlusion [69] consistently beat it. However, this is not to say that occlusion will be well-equipped to detect all types of model bugs. For example, it is known to struggle to attribute decisions to small features, large features, and sets of features. To the best of our knowledge, no prior works on evaluating feature attribution/saiency with debugging tasks test occlusion (including [I], (2). and 48 ). cannot compare this finding to prior ones.
B Search for Natural Adversarial Features Using Embeddings (SNAFUE)
In Section of the main paper, we introduce our method to search for natural adversarial features using embeddings (SNAFUE). Figure depiets SNAFUE, Here, we provide additional experiments and details.
16

---

Source Class (Photocopier)
(a) Make Synthetic Adversaries
Adv Patch
Generator
Target Class (Printer)
Target Classifier
y
Perturb
(b) Find Similar Natural Adversaries
Nat Patches
Nat Latents
Selected Nat Patches
Target Classifier
Similarity Matrix
Adv Patches
Adv Latents
Figure 7: SNAFUE, our automated method for finding targeted adversarial combinations of natural features. This example illustrates experiment which found that cats make photocopiers misclassified as printers, (a) First, we create feature-level adversarial patches as in [IO] by perturbing the latent activations of a generator. (b) We then pass the patches through the network extract representations of them from the target network's latent activations. Finally, we select the natural patches whose latents are the most similar the adversarial ones.
B.1 Related Work
Natural Adversarial Features: Several approaches have been used for discovering natural adversar- ial features. One is to analyze examples in test set that network mishandles 22 [6] 33 but this limits the search for weaknesses to a fixed dataset and cannot be used for discovering adversarial combinations of features. Another approach is to search for failures over easily describable set of perturbations [8 39 but this requires performing a zero-order search over a fixed set image modifications.
Copy Paste Attacks: Copy/paste attacks have been growing topic of interest and offer another method for studying natural adversarial features. Some interpretability tools have been used to design copy/paste adversarial examples, including feature-visualization [9] and methods based on network dissection [4] 47 25 Our approach is related to that of [10], who introduce robust feature- level adversarial patches and use them for interpreting networks and designing copy-paste attacks.
However, copy/paste attacks from [9 47 25 [0] have been limited to simple proof's of concept with manually-designer copy/paste attacks. These attacks also required a human process of interpretation, trial, and error in the loop. We build off of these with SNAFUE, which is the first method that identifies adversarial combinations of natural features for vision models in a way that is (1) not restricted to fixed set of transformations or limited set of source and target classes and (2) efficiently automatable.
B.2 SNAFUE Methodology
For all experiments here with SNAFUE, we report success rate defined the proportion of the time that a patched image classified the target class minus the proportion of the time the
17

---

unpatched natural image was. In the main paper, we attack a ResNet-50, but for experiments here in the Appendix, we attack ResNet-18 18 [22].
Robust feature-level adversarial patches: First, we create synthetic robust feature-level adversarial patches in [TO] by perturbing the latent activations of BigGAN (7) generator. Unlike [TO), do not use a GAN discriminator for regularization or use an auxiliary classifier to regularize for realistic-looking patches. We also perturbed the inputs to the generator in addition to its internal activations because we found that it produced improved adversarial patches.
Candidate patches: Patches for SNAFUE can come from any source and do not need labels. Features do not necessarily have to be natural and could, for example, be procedurally generated. Here, we used total of N 265,457 natural images from five sources: the ImageNet validation set [60] (50,000) TinyImageNNet [38] (100,000), OpenSurfaces S (57,500 550,, the non OpenSurfaces images
from Broden 4 (37,953).
Image and patch scaling: All synthetic patches were parameterized as x 64 images. Each was trained under transformations, including random resizing. Similarly, all natural patches were 64 x 64 pixels. All adversarial patches were tested by resizing them to 100 x 100 and inserting them into 256 x 256 source images at random locations.
265,457 natural patches along with M 10 adversarial patches and passed them through the target network to get an L-dimensional embedding of each using the Post-Rel-U latents from the penultimate (avgpooling) layer of the target network (which we found to be more effective than other embedding methods). The result was nonnegative N x I matrix U of natural patch embeddings and a M x I. matrix of adversarial patch embeddings. A different V must be computed for each attack, but U only needs to be computed once. This plus the fact that embedding the natural patches does not require insertion into a set of source images, makes SNAFUE much more efficient than a brute-force search. We also weighted the values of V based on the variance of the success the synthetic attacks and the variance of the latent features under them.
Embeddings: We used the
Weighting: To reduce the influence of embedding features that vary widely across the adversarial patches, we apply an L-dimensional elementwise mask the embedding in each row of V with weights
ef: > 1
w/
cv. (Vi))
else
where (V;) is the coefficient of variation over the j'th column of V, with μy =
1 M Σ, Vij
and (Vig) N 2, (V,5 nz)²
for some small positive
My-6
To increase the influence of successful synthetic adversarial patches and reduce the influence of poorly performing ones, we also apply a Middimensional elementwise mask h to each column of V with weights
8; 8min
h4
8max
8
5min
where 6; is the mean fooling confidence increase of the post-softmax value of the target output neuron under the patch insertions for the jth synthetic adversary. If any is negative, we replace with zero, and if the denominator is zero, set h; zero
Finally, we multiplied w elementwise with each row of and h elementwise with every column of V to obtain the masked embeddings Vn.
Selecting natural patches: We then obtained the N x M matrix of cosine similarities between U and V. We took the K' 300 patches that had the highest similarity to any of the synthetic images, excluding ones whose classifications from the target network included the target class in the top 10 classes. Finally, we evaluated all K' natural patches under random insertion locations over all 50 source images from the validation set and subsampled the K = 10 natural patches that increased the target network's post-softmax confidence in the target class the most. Screening the K' natural patches for the best 10 caused only a marginal increase in computational overhead. The method was mainly bottlenecked by the cost of training the synthetic adversarial patches (for 64 batches
18

---

Num synthetic advs test
Num natural candidates test
Mean fooling rate
(top decile n=10)
Mean fooling rate
(top decile, n=10)
0.75
8:20
0.22
1000
2500
2500
Num synthetic advs created
Num natural candidates tested
Figure 8: (Left) Mean natural patch success rate as a function of the number of synthetic adversaries created, from which selected the best 10 (or took all if there were fewer than 10) to then use in the search for natural patches. (Right) Mean natural patch success as a function of the number of natural adversaries the top n = 10 of 100 attacks. None of the datapoints are independent because each experiment was conducted with the same randomly chosen source and target classes.
screened for the top 10. Errorbars give the standard deviation of the mean over
of 32 insertions each). The numbers of screened and selected patches arbitrary. Because it is fully automated, SNAFUE allows for flexibility in how many synthetic adversaries to create and how many natural adversaries to screen. To experiment with how run SNAFUE most efficiently and effectively. we test the performance of the natural adversarial patches for attacks when we vary the number of synthetic patches created and the number of natural ones screened. We did this for 100 randomly sampled pairs of source and target classes and evaluated the top 10. Figure shows the results.
B.3 SNAFUE Examples
We provide additional examples of copy/paste attack patches from SNAFUE in Figure We present additional examples Figure 10 and argue that SNAFUE can be used to discover distinct types of flaws.
B.4 SNAFUE Experiments
Replicating previous ImageNet copy/paste attacks without human involvement. First, we out to replicate all known successful ImageNet copy/paste attacks from previous works without any human involvement. To our knowledge, there 9 such attacks, each from ()) 25 and 10 $6) We used SNAFUE to find 10 natural patches for all 9 attacks. Figure II shows the results. In all cases, we are able to find successful natural adversarial patches. most cases, we find similar adversarial features to the ones identified in the prior works. We also find number of adversarial features not identified in the previous works.
SNAFUE is scalable and effective between similar classes. There are many natural visual features possible combinations thereof, it is important
that image classifiers may encounter and many that tools for interpretability and diagnostics with natural features are scalable. Here, we perform broad search for vulnerabilities. Based on prior proofs of concept 9 47 25 10] copy/paste attacks tend to be much easier to create when the source and target class are related (see Figure IT). To choose similar source/target pairs, we computed the confusion matrix C for the target network with 0 < Cg < 1 giving the mean post-softmax confidence on class that the network assigned to validation images of label i. Then for each of the 1,000 ImageNet classes, we conducted attacks
"The attacks presented (25) were not universal within source and were only developed for single source image each. When replicating their results, we use the same single sources. When replicating attacks from the other two works, we train and test the attacks as source class-universal ones.
100 test fourth attack involving patches making traffic lights appear as flies, the examples they identified were not successful at causing targeted misclassification.
47 also test copy paste attacks, but not ImageNet networks
19

---

GoldEnnh
Indigo Bunting
Balbul
Hlmminghhie
Ground
Lest Beetle
Walking
Dowticher
Redshank
Thunder snake
Ringneck snake
Taginn
my
Janamar
Hunminghied
Cabtage butterfl
Adminal butteelly
losker
Mrisan
Beain conal
Puffer
Wossel
Eleckic
Plate
Surrite
Card fungus
Eatheter
Figure 9: Examples of natural adversarial patches for several targeted attacks. Many share common features and lend themselves easily human interpretation. Each row contains examples from single attack with the source and target classes labeled on the left.
20

---

Envelope I Carton
Bicycl-bullt-fo-ttwoo
J Trieycle
Great white shark
I
Killer whale
Frying pan
Spatula
BWH
Rock crab
Fiddler ceab
Figure 10: SNAFUE identifies distinct types of problems. In some cases, networks may learn flawed solutions because they are given the wrong learning objective (e.g. dataset bias). In contrast, in other cases, they may fail to converge (e.g. misgeneralliztioon). SNAFUE can discover both types of issues. some cases, it discovers failures that result from dataset biases. Examples include when it identifies that cats make envelopes misclassified as cartons or that young children make bicycles-builee buill--o-ttt---ottooo misclassified as tricycles (rows 1-2). In other cases, SNAFUE identifies failures that result from the particular representations a model learns, presumably due to equivalence classes in the network's 's representations. Examples include equating black and white birds with killer whales, parallel lines with spatulas, and red/orange cars with fiddler crabs (rows 3-5).
desirable solution even with the correct objective
using that class as the source and each of most confused 5 classes as targets. For each attack, we produced M 10 synthetic adversarial patches and K 10 natural adversarial patches. Figure
and Figure 12 show examples from these attacks with many additional examples in Appendix Figure 9
Patches often share common features and immediately lend themselves descriptions from human.
At the bottom of Figure 12 are histograms for the mean attack success rate for all patches and for best patches (each out of TO) for each attack. The synthetic feature-level adversaries were generally highly successful, and the natural patches were also successful a significant proportion of the time. In this experiment, 3,451 (6.9%) out of the 50,000 total natural images from all attacks were least 50% successful at being targeted adversarial patches under random insertion locations into random images of the source class. This compares to 10.4% success rate for nonadversarial control experiment in which we used natural patches cut from the center of target class images and used the same screening ratio did for SNAFUE. Meanwhile, 963 (19.5% of the 5,000 best natural images 50% successful, and interestingly, in all but one of the 5,000 total source/target class pairs, at least natural image was found which fooled the classifier targeted attack for at least one source image.
least
Copy/paste attacks between dissimilar classes are possible but more challenging. In some cases, the ability to robustly distinguish between similar classes may be crucial. For example, is important for autonomous vehicles tell red and yellow traffic lights apart effectivvlyy But studying how easily networks can be made to mistake an image for arbitrary target classes of broader general interest. While synthetic adversarial attacks often work between arbitrary source/target classes, to the best of our knowledge, there are no successful examples from any previous works of class-universal copy/paste attacks.
We chose examine the practical problem of understanding how vision systems in vehicles may fail to detect pedestrians [50] because it provides example where failures due to novel combinations of natural features could realistically pose safety hazards. To test attacks between dissimilar classes, chose 10 ImageNet classes of clothing items (which frequently co-occur with humans) and 10 of
21

---

Carter (2019) Snorkel Scuba Diver
Success Rate=0.338
Carter et al. (2019)
Grey Whale -> Great White Shark
Success Rate=0.326
Carter et al. (2019)
Frying Pan -> Wok Success Rate=0.306
Hernandez al. (2019) Container Ship
Amphibian
Success Rate=0.672
ANND7
Hernandez al. (2019) Pretzel -> Hermit Crab Success Rate=0.928
Hernandez et al. (2019) Snowplow -> Jeep Success Rate=0.824
Casper (2022) Bee Fly Success Rate=0.250
Casper et (2022) Indian Elephant ->
African Elephant
Success Rate=0.408
Casper et al. (2022)
Pufferfish -> Lionfish Success Rate=0.236
Our automated replications of all 9 prior examples of ImageNet copy/paste attacks of which aware from 9 25) and (10). Each set of images is laheled source class target class. Each row of 10 patches is labeled with their mean success rate.
Figure 11:
traffic-related objects We conducted 100 total attacks with SNAFUE using each clothing source and traffic target. Figure shows these results. Outcomes were mixed.
On one hand, while the synthetic adversarial patches were usually successful on more than 50% of source images, the natural ones were usually not. Only one out of the 1,000 total natural patches (the leftmost natural patch in Figure 13) succeeded for at least 50% of source class images. This suggests limitation of either SNAFUE or of copy/paste attacks in general for targeted attacks between unrelated source and target classes. On the other hand, 54% of the natural adversarial patches were successful
academic gown, apron, bikini, cardigan, jean, jersey, maillot, suit, sweatshirt, trencheoat) / ( (fire engine, garbage truck, racer, sports streetcar, truck, trailer truck, trolleybus, street sign, traffic light)
22

---

Synthesized Mean success rater 988
Natural Mean suicess rate: 0.509
Electtic locomotive
White fur
Steam locomotive
Mean suceess
success ratlc
Overskirt
Colored dots Apron
Mean success rate: 0.764
Mean success rale: 0.59
Text I Wool
Success rates
between casily-confused
Synthetic
Natural
Freq
0.0
2.2
2.4
0.6
0.0
1.0
0.0
0.2
0.4
0.0
0.a
1.0
All attacks, n-50000
Best attacks, n-5000
Figure 12: (Top) Examples of copy/paste attacks between similar source/target classes. Above each set of examples is the mean success rate of the attacks across the adversaries x 50 source images. (Bottom) Histograms of the mean success rate for all synthetic and natural adversarial patches and ones that performed the best for each attack. Labels for the adversarial features (e.g. "white fur") are human-produced.
for at least source image, and such a natural patch was identified for 87 of all 100 source/target class pairs.
Are humans needed at all with SNAFUE? SNAFUE has the ad advantage of not requiring a human in the loop- only human after the loop make final interpretation of set of images that are usually visually coherent. But can this step be automated too? To test this, we provide proof of concept in which we use BLIP 41 and ChatGPT (v3.5) [6]] caption the sets of images from the attacks in Figure TO First, we caption set of 10 natural patches with BLIP [41), and second, we give them to ChaiGPT 61 following the prompt "The following set of captions for images. Please read these captions and provide simple "summary" caption that describes what thing that all (or most) of the images have in common."
Results are shown with the images in Figure 14 In some cases, such as the top two examples with cats and children, the captioning unambiguously successful at capturing the key common feature of the images. In other cases, such as with the black and white objects or the red cars, the captioning mostly unsuccessful, identifying the objects but all of the key qualities about them. Notably, in the case of the images with stripe/bar features. ChatGPT honestly reports that it finds no common theme. Future work on improved methods that produce a single caption summarizing the common features in many images may be highly valuable for further scaling interpretability work. However, we find that a human clearly superior to this particular combination of BL.P ChatGPT on this particular task.
23

---

Synthesized Mean success 0.992
Natural
rate:
Suit
Orange circles
Traffic light
Success rates
Clothing traffic class
Synthelic Natural
Freq
attacks, n=100
Figure 13: (Top) Examples from our most successful copy/paste attack using a clothing source and traffic target. The mean success rates of the attacks across 10 adversaries 50 source images are shown above each example. (Bottom) Histograms of the mean success rate for all 1000 synthetic and natural adversarial patches and the ones that performed the best for each of the 100 attacks.
"Images of cats in various settings and poses."
"Images of children and babies in various settings and poses."
"Images of birds various settings poses." "
"Images of various objects and settings with no clear common theme."
"Images of vehicles, mainly cars, in various settings and poses."
Figure 14: Natural adversarial patches from Figure 10 captioned with BLIP and ChatGPT,
Failure Modes for SNAFUE Here, we discuss various non-mutually exclusive ways in which SNAFUE can fail to find informative, interpretable attacks.

---

Target class similarity
Hippopotamus
Indian Elephant
High diversity
Damselffy 1
Dragonfly
Ambiguity
Redbone
1 Vizsla
Figure 15: Examples of 3 of the 5 types of failure modes for SNAFUE that we describe in Section B.4
1. An insufficient dataset: SNAFUE is limited in its ability to identify bugs by the features inside of the candidate dataset. the dataset does not have a feature, SNAFUE cannot find it.
2. Failing to find adversarial features in the dataset: SNAFUE will not necessarily recover an adversarial feature even it is in the dataset. We conducted a version of our original SNAFUE experiment from in which the patch trojan triggers were included in the dataset candidate patches. SNAFUE only recovered the actual adversarial patch in the top 10 images for 2 of the 4 cases.
3. Target class features: Instead of finding novel fooling features. SNAFUE sometimes identifies features that resemble the target class yet evade filtering. Figure [5 (top) gives an example of this in which hippopotamuses are made to look like Indian elephants by inserting patches that evade filtering because they depict African elephants.
4. High diversity: We find some cases in which the natural images found by SNAFUE lack visual similarity and do not seem to lend themselves to simple interpretation. One example of this is the images for damselfty dragonfly attacks Figure 15 (middle).
5. Ambiguity: Finally, we also find cases in which SNAFUE returns coherent set of natural patches, but remains unclear what about them is key to the attack. Figure [5 (bottom) shows images for 'redbone' to 'vizsla' attack, and it seems unclear from inspection alone the role that brown animals, eyes, noses, blue backgrounds, and green grass have in the attack because multiple images share each of these qualities in common.
25

---

TABOR
Smiley Emoji (Patch)
Clownfish (Patch)
Green Star
(Patch)
Strawberry
(Patch)
Jaguar (Style)
Elephant Skin
(Style)
Jellybeans (Style)
Wood Grain
(Style)
Fork
(Natural Feature)
Apple
(Natural Feature)
Sandwich
(Natural Feature)
Donut
(Natural Feature)
Figure 16: All visualizations from TABOR
20
All Visualizations
C.1
Visualizations By Method
TABOR: Figure 16
Inner Fourier Feature Visualization: Figure
17
Target Fourier Feature Visualization: Figure
18
Inner CPPN Feature Visualization: Figure
Target CPPN Featuer Visualization: Figure 20
Adversarial Patch: Figure 21
Robust Feature-Leve Adversaries with Generator Perturbation Parameterization: Figure 22
Robust Feature-Level Adversaries with Generator Parameterization: Figure 23
Search for Natural Adversarial Features Using Embeddings (SNAFUE): Figure
24
26

---

Inner Fourier Feature Visualization
Smiley Emoji
(Patch)
Clownfish
(Patch)
Green Star
(Patch)
Strawberry
(Patch)
Jaguar (Style)
Elephant Skin
(Style)
Jellybeans (Style)
Wood Grain
(Style)
Fork
(Natural Feature)
Apple
(Natural Feature)
Sandwich
(Natural Feature)
Donut
(Natural Feature)
Figure 17: All visualizations from inner Fourier feature visualization (51) Grey images result from optimizer failures from the off-the-shelf code for this method. If all 10 runs failed to produce any visualizations, grey is displayed.
C.2
Visualizations by Trojan
Smiley Emoji: Figure
25
Clownfish: Figure
26
Green Star: Figure
27
Strawberry: Figure 28
Jaguar: Figure 29
Elephant Skin: Figure 30
Jellybeans: Figure
Wood Grain: Figure
32
Fork: Figure
33
Apple: Figure
Sandwich: Figure 35
27

---

Target Fourier Feature Visualization
Smiley Emoji
(Patch)
Clownfish
(Patch)
Green Star
(Patch)
Strawberry (Patch)
Jaguar (Style)
Elephant Skin
(Style)
Jellybeans (Style)
Wood Grain
(Style)
Fork
(Natural Feature)
Apple (Natural Feature)
Sandwich
(Natural Feature)
Donut
(Natural Feature)
Figure 18: All visualizations from target Fourier feature visualization 51
Donut: Figure 36
D Survey Methodology
An example survey is available
wwsail.mit.... www.a...m.e.euuttaeeeeeeee.. test/
With institutional review board approval, we created surveys, one per method plus a final one for all methods combined. We sent each 100 contractors and excluded anyone who had taken one survey from taking any others in order to avoid information leaking between them. Each survey per trojan plus attention check with unambiguous feature visualization. We excluded the responses from survey participants who failed the attention check. Fach question showed survey participants 10 visualizations from the method and asked what feature it resembled them.
had 12 questions
To make analysis objective, made each survey question multiple choice with 8 possible choices. Figure shows the multiple-choice alternatives for each trojan's 's questions. For the patch and style trojans, the multiple choice answers images, and for natural feature trojans, they were words. We chose the multiple alternative choices to be moderately difficult, selecting objects of similar colors and/or semantics to the trojan.
One issue with multiple choice evaluation is that it sometimes gives the appearance of success when method in reality failed. A visualization simply resembling feature more than another is not
28

---

Inner CPPN Feature Visualization
Smiley Emoji
(Patch)
Clownfish
(Patch)
Green Star
(Patch)
Strawberry
(Patch)
Jaguar (Style)
Elephant Skin
(Style)
Jellybeans (Style)
Wood Grain
(Style)
Fork
(Natural Feature)
Apple (Natural Feature)
Sandwich
(Natural Feature)
Donut
(Natural Feature)
Figure 19: All visualizations from inner CPPN feature visualization [46]. Grey images result from optimizer failures from the off-the-shelf code for this method. If all 10 runs failed to produce any visualizations, grey one is displayed.
a strong indication that it resembles that feature. In some cases, we suspect that when participants were presented with non-useful visualizations and foreed to make choice, they chose nonrandomly in ways that can coincidentally overrepresent the correct choice. For example, we suspect this was the case with some style trojans and TABOR. Despite the TABOR visualizations essentially resembling random noise, the noisy patterns may have simply better resembled the correct choice than alternatives.
29

---

Target CPPN Feature Visualization
Smiley Emoji (Patch)
Clownfish (Patch)
Green Star
(Patch)
Strawberry (Patch)
Jaguar
(Style)
Elephant Skin (Style)
Jellybeans (Style)
Wood Grain
(Style)
Fork
(Natural Feature)
Apple
(Natural Feature)
Sandwich
(Natural Feature)
Donut
(Natural Feature)
Figure 20: All visualizations from target CPPN feature visualization 46
30

---

Adversarial Patch
Smiley Emoji (Patch)
Clownfish (Patch)
Green Star
(Patch)
Strawberry (Patch)
Jaguar (Style)
Elephant Skin
(Style)
Jellybeans (Style)
Wood Grain
(Style)
Fork
(Natural Feature)
Apple
(Natural Feature)
Sandwich
(Natural Feature)
Donut
(Natural Feature)
Figure 21: All visualizations from adversarial patches
g
31

---

Robust Feature Level Adversaries Perturbation
Smiley Emoji (Patcch)
A
Hd
Clownfish (Patch)
Green Star
(Patch)
Strawberry (Patch)
Jaguar
(Style)
Elephant Skin
(Style)
Jellybeans (Style)
Wood Grain
(Style)
Fork
(Natural Feature)
Apple
(Natural Feature)
Sandwich
(Natural Feature)
Donut
(Natural Feature)
Figure 22: All visualizations from robust feature-level adversaries with generator perturbation parameterization [0]
32

---

Robust Feature Level Adversaries Generator
Smiley Emoji (Patch)
Clownfish
(Patch)
Green Star
(Patch)
Strawberry (Patch)
Jaguar (Style)
Elephant Skin (Style)
Jellybeans (Style)
Wood Grain
(Style)
Fork
(Natural Feature)
Apple
(Natural Feature)
Sandwich
(Natural Feature)
Donut
(Natural Feature)
Figure 23: All visualizations from robust feature-level adversaries with generator parameterization.
33

---

SNAFUE
Smiley Emoji (Patch)
Clownfish (Patch)
Green Star
(Patch)
Strawberry (Patch)
Jaguar
(Style)
Elephant Skin (Style)
Jellybeans (Style)
Wood Grain
(Style)
Fork
(Natural Feature)
Apple
(Natural Feature)
Sandwich
(Natural Feature)
Donut
(Natural Feature)
Figure 24: All visualizations from search for natural adversarial features using embeddings (SNA- FUE).
34

---

Smiley Emoji (Patch)
TABOR
Inner Fourier FV
Target Fourier FV
Inner CPPN FV
Target CPPN FV
Adv. Patch
RFLA-Perturb
H1.
RFLA-Gen
SNAFUE
Figure 25: All visualizations of the smiley emoji patch trojan.
35

---

Clownfish (Patch)
P...
TABOR
Inner Fourier FV
Target Fourier FV
Inner CPPN FV
Target CPPN FV
Adv. Patch
RFLA-Perturb
RFLA-Gen
SNAFUE
Figure 26: All visualizations of the clownfish patch trojan.
36

---

Green Star (Patch)
TABOR
Inner Fourier FV
Target Fourier FV
Inner CPPN FY
Target CPPN FV
Adv. Patch
RFLA-Perturb
RFLA-Gen
A
SNAFUE
Figure 27: All visualizations of the green star patch trojan.
37

---

Strawberry (Patch)
TABOR
Inner Fourier FV
Target Fourier FV
Inner CPPN FV
Target CPPN FV
Adv. Patch
RFLA-Perturb
RFLA-Gen
SNAFUE
Figure 28: All visualizations of the strawberry patch trojan.
38

---

Jaguar (Style)
TABOR
Inner Fourier FV
Target Fourier FV
Inner CPPN FV
Target CPPN FV
Adv. Patch
RFLA-Perturb
RFLA-Gen
SNAFUE
Figure 29: All visualizations of the jaguar style trojan.
39

---

Elephant Skin (Style)
TABOR
Inner Fourier FV
Target Fourier FV
Inner CPPN FV
Target CPPN FV
Adv. Patch
RFLA-Perturb
RFLA-Gen
SNAFUE
Figure 30: All visualizations of the elephant skin style trojan.
40

---

Jellybeans (Style)
TABOR
Inner Fourier FV
Target Fourier FV
Inner CPPN FV
Target CPPN FV
Adv. Patch
RFLA-Perturb
RFLA-Gen
SNAFUE
Figure 31: All visualizations of the jellybeans style trojan
41

---

Wood Grain (Style)
TABOR
Inner Fourier FV
Target Fourier FV
Inner CPPN FV
Target CPPN FV
Adv. Patch
RFLA-Perturb
RFLA-Gen
SNAFUE
Figure 32: All visualizations of the wood grain style trojan.
42

---

Fork (Natural Feature)
TABOR
Inner Fourier FV
Target Fourier FV
Inner CPPN FV
Target CPPN FV
Adv. Patch
RFLA-Perturb
RFLA-Gen
SNAFUE
Figure 33: All visualizations of the fork natural feature trojan.
43

---

Apple (Natural Feature)
TABOR
Inner Fourier FV
Target Fourier FV
Inner CPPN FV
Target CPPN FV
Adv. Patch
RFLA-Perturb
RFLA-Gen
SNAFUE
Figure 34: All visualizations of the apple natural feature trojan.
44

---

Sandwich (Natural Feature)
TABOR
Inner Fourier FV
Target Fourier FV
Inner CPPN FV
Target CPPN FV
Adv. Patch
RFLA-Perturb
RFLA-Gen
SNAFUE
Figure 35: All visualizations of the sandwich natural feature trojan.
45

---

Donut (Natural Feature)
TABOR
Inner Fourier FV
Target Fourier FV
Inner CPPN FV
Target CPPN FV
Adv. Patch
RFLA-Perturb
RFLA-Gen
SNAFUE
Figure 36: All visualizations of the donut natural feature trojan.
46

---

Smiley Emoji (Patch)
Clownfish (Patch)
B
D
A
B
C
D
www
F
H
F
G
H
Green Star (Patch)
Strawberry (Patch)
A
B
C
D
A
STOP
B
C
D
E
F
G
H
E
F
G
H
Jaguar (Style)
Elephant Skin (Style)
A
B
c
D
A
B
C
D
F
H
F
H
Jellybeans (Style)
Wood Grain (Style)
A
B
D
A
B
C
D
E
F
G
H
E
F
G
H
Fork (Natural Feature)
Apple (Natural Feature)
A. Car, B. Fork, C. Oven, D. Refrigerator
A. Bush, B. Bottle, C. Lettuce, D. Apple
E. Bowl, F. Laptop, G. Faucet, H. Stapler
E. Goat, F. Berries, G. Clouds, H. Shoes
Sandwich (Natural Feature) A. Salad, B. Pizza, C. Omelette, D. Sandwich
Wood Grain (Natural Feature) A. Muffin, B. Cake, C. Baguette, D. Cupcake E. Danish, F. Pie, G. Donut, H. Croissant
E. Spaghetti, F. Stir Fry, G. Nachos, H. Waffle
Figure 37: The multiple choice alternatives for each trojan's 's survey question.
47


Reviews:
[{"review_id": "WDBTsVlmUd", "reviewer": "Reviewer_EMti", "summary": "In this work authors study the capabilities of interpretability tools in identifying trojans. Towards this goal authors curate a benchmark dataset and perform both human and automated studies to evaluate various interpretability tools. They discover that their benchmark can be a challenging benchmark for most of these tools and to improve some of these techniques they build upon two previously existing methods. ", "questions": "I would like some discussions on motivation in terms of practical use behind this work.", "limitations": "Yes, authors have addressed limitations behind their work.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The problem they are studying is interesting.\n2. The curated benchmark can be useful to some communities.", "weaknesses": "1. While the problem of discovering trojans via interpretability tools can be interesting, I am not 100% sure about the motivation behind this work. More specifically what do authors think is the practical use case of such analysis and finding? I think the paper can be motivated better and the introduction can be re-written to reflect a stronger motivation. In general, I believe that writing of the paper can significantly improve as well.\n\n2. The contribution in terms of technical proposal is limited. They propose simple extensions that build on previous work.\n\n3. Section 5.2 could have been explained in more detail.\n\n4. Only a limited number of trojans were tested. Even within the studied trojans the scope was limited to 12 trojans that were implemented.\n\n**Minor comments:**\n\nLine 68 typo.\n\nLine 35 (may comparable -> may be comparable)\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the feedback and comments – we appreciate it! We are glad you found the paper interesting and that this approach to benchmarking is useful. \n\n**Re – Weakness 1 and Question 1 (clarifying the overall motivation):**\n\nOur main motivation is to study how methods for interpreting/explaining deep neural networks can be useful for red teaming – finding flaws in networks. We use trojans to study this for two reasons. First, finding trojans is a useful type of debugging task that is similar to finding OOD bugs in practice. Second, we can clearly know when an interpretation is good based on whether it identifies the trojan trigger, which is known as a ground truth. \n\nWe think this work is important because it bridges interpretability with useful debugging applications. By testing whether interpretability tools can help us rediscover known trojans, we can develop a better understanding of which tools can help us better red team networks in practice. In a survey of the interpretable AI literature, [Rauker et al (2023)](https://arxiv.org/abs/2207.13243)(section VII) argue that working more to test interpretability tools with debugging tasks like this will be useful for making interpretability tools more relevant for practitioners. \n\nThank you for letting us know that the motivation should be more clear. We are updating the abstract and introduction for clarity. This has included a new explicit sentence: “Our motivation is to study how methods for interpreting deep neural networks can help humans find bugs in them.”\n\n**Re – Weakness 2 (contributions from RFLA-Gen and SNAFUE):**\n\nOur primary goal when writing this paper has been to show how interpretability tools can be more pragmatically studied by using them for red-teaming tasks. In this sense, we intend the main contribution to be the benchmark and benchmarking methodology. We do not see RFLA-Gen and SNAFUE as the central contributions – they are one of the four that we outline in the introduction. \n\nHowever, we also believe that SNAFUE is an important contribution. We did our best to thoroughly study it, and we dedicated ~8 pages to it in Appendix B. Unlike all of the other 8 feature synthesis methods, it produces visualizations in the form of natural images. Thus, it offers a unique approach and visualizations that are particularly easy for humans to analyze. As a result, we think it will be valuable for the red teaming toolbox. We are updating the paper to highlight this.\n\n**Re – Weakness 3 (request for more detail in Section 5.2):** \n\nThank you – we are adding further details in 5.2 which will include additional details from Appendix B about the process of patch generation, latent matching, and screening. \n\n**Re – Weakness 4 (12 trojans):**\n\nWhen designing experiments, we faced a tradeoff between the number of trojans to implant and our ability to study how the tools we study can help humans identify bugs in models. Even though we studied only 12 trojans, this required producing 1,080 visualizations and administering 10 tests to a total of 1,000 knowledge workers. More data is always better. But we emphasize that (1) we usually find fairly comparable results within each trojan type (patch, style, and natural feature); and (2) the scale of our human subject experiments exceeds that of several other very well-known and influential related works (e.g.,  [Adebayo et al., 2020](https://arxiv.org/abs/2011.05429) and [Hase et al., (2020)](https://aclanthology.org/2020.acl-main.491/))."}, {"review_id": "fhlt3jTaLr", "reviewer": "Reviewer_ChpR", "summary": "\nThis paper aims to use interpretation tools to identify unknown bugs in out-of-distribution (OOD) models. It highlights the lack of capability to analyze features that cannot be sampled or identified in advance. To resolve this, authors propose to train models that respond to specific triggers and evaluate interpretability tools based on how they can help humans to identify these triggers. Authors propose trojan discovery as an evaluation task and present a benchmark with 12 trojans of 3 types. It assesses the difficulty of the task through the evaluation of 16 feature attribution/saliency tools and 7 feature-synthesis methods. Lastly, authors introduce and evaluate 2 variants of the best-performing methods.\n", "questions": "See the weakness section. ", "limitations": "Yes, the authors have adequately addressed the limitations. \n", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "I am not an expert in this field and haven’t had any prior experience working in this field, so my evaluations are an educated guess. \n\nQuality and significance: The authors conducted extensive experiments benchmarking 12 trojans and 16 feature attribution/saliency tools. The authors also included a comprehensive discussion section with honest limitations, which I highly appreciate. So I think this paper meets the quality and significance standard. \n\nOriginality: the authors proposed two novel variants of the best performing methods, though the first one “robust feature level adversaries via a generator” is only marginally novel. So I think this paper is on the borderline of the originality standard. \n\n", "weaknesses": "\nMy main concern for this paper is the poor writing clarity. Understandably, this paper contains a lot of information and thus writing is harder than usual. But for someone new to this field like myself, I find it very hard to follow the big picture and also I find it hard to understand exactly what the authors did. \n\nThere are also grammar issues here and there, or maybe the grammar is technically correct but the style is unusual, making it hard to read. For example: \nline 209 “we introduce using …”\nline 94 the logic of “instead” is not very straight forward. \nline 79 “[27] who used” I think “who” here is redundant? \nLines 56-57: “one which … another which”. \n\nOther than grammar issues, the overall writing is just not very clear and easy to follow, and it feels the paper was rushed and not polished. This is the main reason for my rating of 4. I think this paper is not yet ready to be circulated amoung the neurips community. \n\n \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the feedback and comments – we appreciate it! We are glad to hear that you find the significance and quality of the work to be up to publication standard. \n\n**Re – Originality (RFLA-gen marginally novel):**\n\nOur primary goal has been to show how (1) interpretability tools can be more pragmatically studied by using them for red-teaming and (2) how trojan discovery is a practical benchmarking task. In this sense, we intend the main contribution to be the benchmark and benchmarking methodology. We do not see RFLA-Gen as the central contribution. It and SNAFUE are only one of the four contributions that we outline in the introduction.\n\nHowever, we also believe that SNAFUE is an important contribution. We did our best to thoroughly study it, and we dedicated ~8 pages to it in Appendix B. Unlike all of the other 8 feature synthesis methods, it produces visualizations in the form of natural images. Thus, it offers a unique approach and visualizations that are particularly easy for humans to analyze. As a result, we think it will be valuable for the red teaming toolbox. We are updating the paper to highlight this.\n\n**Re – Weaknesses (overall clarity):**\n\nThank you for pointing this out. We are committed to ensuring the paper is as clear as possible. To improve this, we are taking two strategies. First, we are editing through the paper line-by-line ourselves with fresh eyes in order to simplify sentence structure, add signposting, manage jargon, and improve general clarity. Second, we are having an additional lab member who does not work on interpretability take a close read to help us improve parts of the paper that lack clarity. \n"}, {"review_id": "vV6WWggT8B", "reviewer": "Reviewer_WymJ", "summary": "This paper studies the performance of different interpretability methods (feature attribution and synthesis) on model debugging, especially for poisoning-based backdoor attacks. The authors also propose to consider this evaluation as a better benchmark of interpretability tools since we know the ground-truth features and It also has more practical applications. For feature attribution methods, they fail to reverse backdoor features in the training dataset even with known backdoored samples. For feature visualization (synthesis) methods, they perform slightly better performance, especially for methods on robust adversarial features. However, the generated features are still hard to understand for humans. Therefore, the authors propose to utilize natural adversarial patches to replace generated adversarial patches.", "questions": "1. About embedding space of SNAFUE method: do we have better metrics or semantic space to find natural adversarial patches? For me, I think SNAFUE is a variant of the original method based on robust adversarial features, with adding constrained projection. The semantic space and metrics for projection are very important. \n\n2. Could the authors explain why adding natural patches in other class images could mislead DL models and how to further improve this attack performance? It seems that patch attack is not as powerful as PGD attack on L_inf or 2 norm ball. Do you think this is also the reason why \"Copy/paste attacks between dissimilar classes are possible but more challenging\"?\n\n3. Could you explain more about the 2nd point of Failure Modes for SNAFUE on Page 22? \n\n4. Why visualization of inner neurons is not effective? (In Page 8) Please give more insight into this phenomenon.\n\n5. What are your opinions about the 1st point in Strengths part?", "limitations": "The main limitations are from SNAFUE method. It can not find unseen features. The authors also mentioned it.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "I like this work. This paper tries to answer two important problems of interpretability methods:\n\n1. The Faithfulness of generated feature visualizations: I think there have been two serious problems with feature visualization all along:\n\n    1.1 faithfulness: could the generated patterns truly reflect the inner mechanism of DL models? Or just some bias from the optimization method to find them? \n\n    1.2 How do humans correctly understand the generated features? are faithful patterns always understandable by humans? \n\n    1.3 the natural adversarial patterns satisfy the second point at least, although they also have shortcomings (can not find unseen features). \n\n2. The proper evaluation benchmark: The authors consider the more practical scenario, model debugging, which can more realistically measure the effectiveness of interpretability methods.", "weaknesses": "1. About full visualizations of feature attribution with known backdoored samples: I expect that the authors show all visualization results about backdoor reversing. I think the L_1 distance can not fully reflect the poor performance of feature attribution methods. Visualization results are a more proper proxy. I recommend the authors could put visualization in main submission.\n\n2. About attack success rate and clean accuracy of backdoored models: I suggest the authors should give more results of attack success rate and clean accuracy of backdoored models with different triggers. They only mention them on the training dataset on Page 4. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the feedback and comments – we appreciate it! We are glad to hear that you like this work and feel that it meets the standard for publication. \n\n**Re – Weakness 1 (Attribution visualizations and L1 distance):**\n\nThank you for the suggestion to add visualizations. To address this, we are adding a figure to Appendix A.3 in the same style as Figure 1 from [Adebayo et al. (2018)](https://arxiv.org/abs/1810.03292). You can find it in the pdf attached with the global response. \n\nConcerning the L1 distance measure, we also agree that it is not ideal. We have replaced this experiment with one that uses the Pearson correlation between the ground truth and an attribution map. This avoids the issue of privileging sparse attribution maps. We no longer use a blank-image baseline and instead only use an edge-detector baseline. The new figures are in the PDF attached with the global response. Overall, we continue to find that 15 of the 16 methods consistently struggle. \n\n**Re – Weakness 2 (Reporting attack success rates):**\n\nThank you for the suggestion. For the ResNet50, the accuracy on clean data after training was 74.2% which represents a 2.9 percentage point drop from the pretrained model. We updated the paper to mention this. The trojan success rates for all 12 trojans in order were 95.8%, 93.3%, 98.0%, 92.0%, 98.1%, 100%, 96.0%, 82.0%, 30.8%, 24.2%, 38.7%, and 37.2%. We have added this information as an additional column of Table 1. \n\n**Re – Question 1 (SNAFUE embedding method):**\n\nWe agree with the importance of the embedding. In general, any embedding could be used. In the past, we experimented with a version of SNAFUE that uses CLIP and VGG embeddings, but we found the most effective results using the latent embeddings from the target model. We added a note of this in Appendix B. \n\n**Re – Question 2 (Patch versus natural feature attacks):**\n\nThank you for this point. One thing to note is that we never produce single-source attacks. All attacks in the paper were universal or class-universal which results in a substantially lower attack success rate than single-image attacks. Nonetheless, 95% of the synthetic patches shown in Figure 12 fool the model on over 50% of source images. The main reason why the copy/paste attacks have a much lower fooling rate is because of the restrictive requirement that the patches be natural images. At this point, since the attacks are constrained to be targeted, class-universal, and based on natural features, we are not particularly surprised that they usually struggle. However, we have found that larger patches can substantially improve attack success rates.\n  \n**Re – Question 3 (SNAFUE failure mode #2):**\n\nThank you for the thorough read. We mistakenly removed the explanation of this point during editing. We conducted a version of the original SNAFUE experiment from Section 5 in which the patch trojan triggers were included in the dataset of candidate patches. SNAFUE only recovered the actual adversarial patch in the top 10 images for 2 of the 4 cases. This is why we say that SNAFUE may fail to find adversarial features even if they are in the dataset. We have updated the Appendix to explain this. \n\n**Re – Question 4 (Why is the visualization of inner neurons difficult?):**\n\nThank you for the question. Using our perturbation-based method, we identified inner neurons that had a strong average influence over the output neuron for a given target class. Because of this, we suspected that one or more of these inner neurons might resemble the trojan triggers. This happened in some cases (e.g., the clownfish trojan), but we found little evidence that the detection of a trojan feature was consistently localized to a single neuron. Ultimately, we think that the main challenge with the “inner” visualization methods is that they require more screening of more visualizations hoping to find a neuron that resembles the trojan feature even though one may not exist. We are adding this to our discussion. \n\n**Re – Question 5 (Thoughts on strength #1):**\n\nWe broadly agree with these challenges with feature visualization. If our main goal from feature-visualization techniques is to red-team models, we have to confront two facts. First, feature visualizations are not real inputs. This is a weakness because the inputs are not very realistic but also a strength because it can help us find failures NOT induced by data we already have access to. Second, different methods (e.g. TABOR vs. SNAFUE) enforce different priors over what visualizations will be produced. This is one of the reasons we think that ensembles of interpretability tools will be the most valuable and that an empirical approach to developing new tools seems useful.\n \nPlease let us know if you would recommend that we discuss this more directly in the paper. \n\n**Re – Limitation (SNAFUE doesn’t produce novel patches)**\n\nWe fully agree. There are two points we will stress though. \n\nThe first is that SNAFUE does produce novel combinations of natural features. This allows SNAFUE to find certain adversarial inputs consisting of a source + natural patch that would not be found in any typical dataset. Figure 9 shows examples of this. In reality, one would not often find images of piano keys next to frying pans, but SNAFUE still finds that this type of combination of features fools the classifier. \n\nSecond, one of the reasons that we wanted to work on SNAFUE is that it is the only method of the 9 tested that produces natural adversarial features. We think that in practice, it can offer a unique complement to existing approaches that only generate synthetic features. We are updating the paper to point this out explicitly. \n"}, {"review_id": "q6Xxxt0qjy", "reviewer": "Reviewer_PpyC", "summary": "This work focuses on uncovering a principled, objective approach to evaluating existing interpretability tools in identifying humanly interpretable attributes in image classification. They approach this via the following synthetic task:\n\n 1. A known, humanly interpretable trojan (a special patch, style transfer, a key object) is injected into a model's training pipeline, leading to a false classification by the model when the trojan is present in an image.\n 2. Different interpretability tools are applied to this model on an uncorrupted image to identify \"interpretable features\" that would lead to a misclassification. An interpretability tool that performs well should ideally output something about the underlying trojan scheme, since this is a clearly interpretable feature that changes the classification.\n 3. The quality of outputs from these interpretability tools are compared by seeing which tool's outputs best help humans identify the original trojan scheme from a multiple-choice set of possible options.\n\nA key challenge of this task is to find these trojan backdoors with access to the model alone and not to the adversarially corrupted examples.\n\nThe authors also propose two novel variants on the existing \"robust feature-level adversaries\" approach of [Casper and Nadeau et.al. Robust feature-level adversaries are interpretability tools] to solve this task.\n\nTheir main findings/contributions are:\n 1. The authors propose this trojan identification task as a synthetic benchmark for assessing the usefulness of human-level interpretability tools\n 2. They claim that naive approaches like saliency maps are not useful, even with access to the adversarial examples,\n 3. In a human study, robust feature-level adversaries and its variants are shown to be best at helping people identify the backdoor scheme, suggesting that these are the most effective interpretability tools.\n 4. No technique worked well for identifying style-transfer based, non-local backdoors. This suggests that current interpretability tools might not do a good job of explaining \"textural\" features.", "questions": " • On the first few reads of the paper, I thought the goal was to show which tools are best at identifying backdoors in DNNs (the title of the paper further corroborates this interpretation). If that is the case, I would have many questions about why we should care about these artificial tasks rather than, say, a cryptographic planted sequence that can't be inspected or interpreted visually. After several reads, I think what you actually care about is just evaluating how good these \"humanly interpretable\" tools are, and the trojan task is just a proxy for this. This was not obvious to me and I would suggest a better title and a clearer intro to better emphasize the goal of your paper.\n\n • I wasn't sure about the details of your human study. Was it 100 participants per method or 10 per method? It would also be great to see some error bars here.", "limitations": "The authors have a healthy discussion about the limitations of their work. Namely that:\n\n 1. They do not cover all possible types of features that may cause model failures. \n\n 2. Evaluations are based only on multiple choice questions, which may be sensitive to aspects of survey and experimental design", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": " • The paper investigates a useful and important research question in the field of interpretability research: \"how do we objectively quantify and measure the quality of interpretable explanations?\".\n\n • The trojan task proposed by the authors is a clever proxy for measuring human interpretability of explanations.\n\n • The authors run an actual human study to measure the usefulness of interpretability tools to people, and their proposed evaluation is a solid measure of whether these tools provide utility.\n\n • The authors are upfront about the limitations of their work and the need for diverse techniques other than theirs to create an effectve benchmark for interpretability tools", "weaknesses": " • The trojan tasks are fairly simplistic and might not measure the actually important capabilities needed by interpretability tools in the real world. For example, if a tool performs well on this benchmark, it's not clear what that would tell me about how to use it in interpreting my model outputs for a misclassified, nonadversarial image.  The trojan task is a proxy, and I would have loved to see how the \"best\" models under this proxy can actually be used for a useful interpretability task like identifying why a model is misclassifying certain images. I would have particularly liked to see how their proposed SNAFUE would work in a real interpretability task.\n\n • I'm not sure the authors' evidence for the ineffectiveness of saliency maps is well-founded. They use $\\ell_1$ error between the saliency map and the location of true patch, which seems like a strange objective. If the patch is small, the $\\ell_1$  distance of a perfect saliency map would decrease with patch size, meaning an all-zeros baseline would appear great. If you used a different metric like bounding box intersection you wouldn't get such a strong negative result. *This isn't a huge deal* because this entire section seems not incredibly relevant to the paper, since your main goal is to do this task without access to the adversarial examples.\n\n • It took me several reads to make sense of this paper. The writing is not completely clear and the figures took a while to interpret correctly. I would suggest really working on finding a better way to explain the human study task to your readers.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the feedback and comments – we appreciate it! We are glad to hear that you believe this is important work and that it meets the standard for publication. \n\n**Re – Weakness 1 (Using these tools on non-trojan flaws):**\n\nWe agree that this is useful. Aside from benchmarking, we think that applying tools like these to red team real models is also important, and we have a concurrent submission that does this type of work in language models. However, we leave it out of scope for this work. We believe the current paper is relatively dense and that a second benchmark would be difficult to address appropriately in the allotted space.\n\nWe also believe that benchmarking experiments like this would require some substantially different methodology. Without trojans, the lack of a known ground truth makes it difficult to know if interpretations of why a model makes certain mistakes is correct. One option would be so simply use these tools to find adversarial weaknesses, but this kind of work has been done by the papers that introduced these methods (e.g. [Brown et al., 2017](https://arxiv.org/abs/1712.09665) and [Casper et al., 2022](https://arxiv.org/abs/2110.03605)).\n\nFinally, we emphasize that our experiments with SNAFUE do involve red teaming networks for non-trojan flaws. SNAFUE finds novel combinations of natural features that often lend themselves to very specific human interpretations of a model’s failures. Figures 8 and 9 in the Appendix show a few dozen examples.\n\n**Re – Weakness 2 (The L1 distance measure for saliency maps):**\n\nThank you for pointing this out. We also agree that the L1 distance is not ideal. We have replaced this experiment with one that uses the Pearson correlation between the ground truth and an attribution map. This avoids the issue of privileging sparse attribution maps. We no longer use a blank-image baseline and instead only use an edge-detector baseline. The new figures are in the PDF attached with the global response. Overall, we continue to find that 15 of the 16 methods consistently struggle. \n\n**Re – Weakness 3 (Clarity):**\n\nThank you for pointing this out. We are committed to making the paper is as clear as possible. To improve this, we are using two strategies. First, we are editing through the paper line-by-line ourselves with fresh eyes in order to simplify sentence structure, add signposting, manage jargon, and improve general clarity. In particular, we are focusing on better describing the methodology behind the human subjects study. Second, we are having an additional lab member who did not help to draft the paper take a close read to help us improve parts of it that can be made more clear.\n\n**Re – Question 1 (Why not focus on uninterpretable weaknesses?):**\n\nWe agree with the importance of also studying techniques for identifying uninterpretable or even cryptographic weaknesses. This was not our main focus, however, because BackdoorBench already makes analogous progress on these problems ([(Wu et al., 2022)](https://arxiv.org/abs/2206.12654)). \n\nInterpretable failure modes are very common (e.g., [Hendrycks et al., 2019](https://arxiv.org/abs/1907.07174)). However, we agree it is important to be clear that networks face both types of threats. The inability of many interpretable AI tools to help humans identify noninterpretable failure modes is an important limitation, but one that is more inherent to the field as a whole instead of our specific work. \n\n**Re – Question 2 (10 or 100 per method):**\n\n100 per method – 1,000 in total. We updated the description to make this completely unambiguous. \n\nThe results reported in Figure 4 can each be viewed as a point estimate of the parameter for an underlying Bernoulli distribution. As such the standard deviation can be upper-bounded by $\\sqrt{0.25} / {\\sqrt{100}} = 0.05$. We have added this to the figure caption. \n"}]

Author rebuttal:
# Global Response

We are thankful for the feedback from all of the reviewers! We were glad to read that this work was interesting (EMti), useful (EMti, PpyC), quality (ChpR), significant (ChpR), and important (PpyC), and the proper approach to benchmarking interpretability (WymJ). We have found the feedback helpful, and we respond to each reviewer in detail below.

Here, we summarize the most significant updates that we have made to the paper. 

**1. Adding the exact trojan success rates and model accuracy on clean data (WymJ):**

We added this to section 3 and in a new column of Table 1. 

**2. Redoing the attribution/saliency experiment with Pearson correlation instead of L1 distance (WymJ and PpyC):**

Instead of measuring the L1 distance between ground truths and attribution maps, we now take the Pearson Correlation. This avoids the issue of privileging sparse attribution maps. We no longer use a blank-image baseline and instead only use an edge-detector baseline. The new figures are in the attached PDF. Overall, we continue to find that 15 of the 16 methods consistently struggle.

**3. Adding examples of attribution/saliency maps (WymJ):**

We are adding examples of attribution/saliency maps to Appendix A. The new figure is in the attached PDF.

**4. Adding details to substantiate the 2nd point on failure modes for SNAFUE on Page 22 in the Appendix (WymJ):**

The reason why we say that SNAFUE may fail to find adversarial features even if they are in the candidate dataset is because of an experiment we conducted previously but mistakenly failed to mention. We conducted a version of the original SNAFUE experiment from Section 5 in which the patch trojan triggers were included in the dataset of candidate patches. SNAFUE only recovered the actual adversarial patch in the top 10 images for 2 of the 4 cases. We added this to the description in the Appendix. 

**5. Edits for detail and clarity (EMti, Wymj, and Ppcy):**

We are making fine-grained updates to the paper with a focus on clarity. This includes simplified sentence structure, reduced jargon, and additional signposting. We are focusing in particular on motivations in the introduction, additional details in section 5.2, and methodological clarity in section 5.3. To help us with clarity, we are getting help from a labmate who was not previously involved in drafting. 


Meta-review: The majority of the reviewers liked the paper, and found the idea conceptually very interesting. The reviewers liked the strong benchmarks and faithfulness of the visualizations. Some reviewers found the paper difficult to understand, and the authors should improve the clarity of the writing for the camera ready.
Decision: Accept (poster)