SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Stephen Casper
scasper@mit.edu
Yuxiao Li
Jiawei Li
Tong Bu
Kevin Zhang
Kaivalya Hariharan
Dylan Hadfield-Menell
Interpretable AI tools are often motivated by the goal of understanding model behavior in out-of-distribution (OOD) contexts. Despite the attention this area of study receives, there are comparatively few cases where these tools have identified previously unknown bugs in models. We argue that this is due, in part, to a common feature of many interpretability methods: they analyze model behavior by using a particular dataset. This only allows for the study of the model in the context of features that the user can sample in advance. To address this, a growing body of research involves interpreting models using feature synthesis methods that do not depend on a dataset. In this paper, we benchmark the usefulness of interpretability tools for model debugging. Our key insight is that we can implant human-interpretable trojans into models and then evaluate these tools based on whether they can help humans discover them. This is analogous to finding OOD bugs, except the ground truth is known, allowing us to know when a user's interpretation is correct. We make four contributions. (1) We propose Trojan discovery as an evaluation task for interpretability tools and introduce a benchmark with 12 trojans of 3 different types. (2) We demonstrate the difficulty of this benchmark with a preliminary evaluation of 16 state-of-the-art feature attribution/saliency tools. Even under ideal conditions, given direct access to data with the Trojan trigger, these methods still often fail to identify bugs. (3) We evaluate 7 feature-synthesis methods on our benchmark. (4) We introduce and evaluate 2 new variants of the best-performing method from the previous evaluation. Code is available at this https url, and a website for this paper is available at this https url.
The most common way to evaluate AI systems is with a test set. However, test sets can fail to identify some problems (such as out-of-distribution failures) and can actively reinforce others (such as dataset biases). This poses a challenge because many of the failures that neural networks may exhibit in deployment can be due to novel features [24] or adversarial examples (see Appendix B). Identifying problems like these requires techniques that are not simply based on passing a dataset through a black-box model. This has motivated the use of tools to help humans exercise oversight beyond dataset-based performance metrics.
Much of the unique potential value of interpretability tools comes from the possibility of characterizing OOD behavior [37]. Despite this, most interpretable AI research relies heavily on the use of datasets, which can only help characterize a model's behavior on features present in these data
(a)
Figure 1: (a): Example visualizations from 9 feature synthesis tools attempting to discover a trojan trigger (see the top row of Table 1) responsible for a bug in the model. Details are in Section 5. (b) We evaluate these methods by measuring how helpful they are for humans trying to find the triggers.
[57]. In particular, much prior work focuses on saliency methods for attributing model decisions to features in input data [34, 49]. However, dataset-based tools are limited to only evaluating how a model behaves on in-distribution features. This is a significant limitation. If the user already has a dataset, manual analysis of how the network handles that data may be comparable to or even better than the interpretability tool [37, 48].
Here, we consider the task of finding trojans [12] – bugs purposefully implanted into a network, causing it to associate a specific trigger feature with an unexpected output. The key insight of this work is that tools for studying neural networks can be evaluated based on their ability to help humans discover trojans. Because trojans are associations between specific input features and output behaviors, helping a user discover one can demonstrate its ability to generate valid interpretations. Finding trojans using interpretability tools mirrors the practical challenge of finding flaws that evade detection with a test set because trojans cannot be discovered with a dataset-based method unless the dataset already contains the trigger features. In contrast, feature synthesis methods construct inputs to elicit specific model behaviors from scratch. Several have been shown to help interpret and debug neural networks (see Section 5). However, there are not yet clear and consistent criteria for evaluating them [45, 57].
Our motivation is to study how methods for interpreting deep neural networks can help humans find bugs in them. We introduce a benchmark for interpretability tools based on how helpful they are for discovering trojans. First, we test whether 16 state-of-the-art feature attribution/saliency tools can successfully highlight the trojan trigger in an image. Even with access to data displaying the triggers, we find that they often struggle to beat a trivial edge-detector baseline. Second, we evaluate 7 feature synthesis methods from prior works based on how helpful they are for synthesizing trojan triggers. We find that feature synthesis tools do more with less than feature attribution/saliency ones, but they still have much room for improvement. Finally, we observe that robust feature-level adversaries [10] performed the best overall on our benchmark. Given this, we build on [10]'s technique to introduce two complementary techniques: one which develops a distribution of adversarial features instead of single instances, and another which searches for adversarial combinations of easily interpretable natural features. We make 4 key contributions:
1. A Benchmark for Interpretable AI Tools: We propose trojan rediscovery as a task for evaluating interpretability tools for neural networks and introduce a benchmark involving 12 trojans of 3 distinct types. $^{1}$
<https://github.com/thestephencasper/benchmarking_interpretability
2. Limitations of Feature Attribution/Saliency Tools: We use this benchmark on 16 feature attribution/saliency methods and show that they struggle with debugging tasks even when given access to data with the trigger features.
3. Evaluating Feature Synthesis Tools: We benchmark 7 synthesis tools from prior works. We find that they are much more practical for debugging than feature attribution tools.
4. Two New Feature Synthesis Tools: We introduce 2 novel, complementary methods to extend on the most successful synthesis technique under this benchmark. $^2$
This paper has a website which also introduces a competition based on this benchmark at https://benchmarking-interpretability.csail.mit.edu/.
Desiderata for interpretability tools: There have been growing concerns about evaluation methodologies and practical limitations with interpretability tools [15, 59, 45, 37, 57]. Most works on interpretability tools evaluate them with ad-hoc approaches instead of standardized benchmarks[45, 37, 57]. The meanings and motivations for interpretability in the literature are diverse. [43] offers a survey and taxonomy of different notions of interpretability including simulatability, decomposability, algorithmic transparency, text explanations, visualization, local explanation, and explanation by example. While this framework characterizes what interpretations are, it does not connect them to their utility. [15] and [37] argue that tests for interpretability tools should be grounded in whether they can competitively help accomplish useful tasks. [30] further proposed difficult debugging tasks, and [45] emphasized the importance of human trials.
Tests for feature attribution/saliency tools: Some prior works have introduced techniques to evaluate saliency/attribute tools. [27] used subjective human judgments of attributions, [1, 65] qualitatively compared attributions to simple baselines, [28] ablated salient features and retrained the model, [14] compared attributions from clean and buggy models, [21, 48] evaluated whether attributions helped humans predict model behavior, [3] used prototype networks to provide ground truth, [26] used a synthetic dataset, and [2] evaluated whether attributions help humans identify simple bugs in models. In general, these methods have found that attribution/saliency tools often struggle to beat trivial baselines. In Section 4 and Appendix A, we present how we use trojan rediscovery to evaluate attribution/saliency methods and discuss several advantages this has over prior works.
Neural network trojans/backdoors: Trojans, also known as backdoors, are behaviors that can be implanted into systems such that a specific trigger feature in an input causes an unexpected output behavior. Trojans tend to be particularly strongly learned associations between input features and outputs [35]. They are most commonly introduced into neural networks via data poisoning [12, 19] in which the desired behavior is implanted into the dataset. Trojans have conventionally been studied in the context of security [29]. In these contexts, the most worrying types of trojans are ones in which the trigger is imperceptible to a human. [68] introduced a benchmark for detecting these types of trojans and mitigating their impact. Instead, to evaluate techniques meant for human oversight, we work with perceptible trojans.<sup>3</sup>
3 Implanting Trojans with Interpretable Triggers
Rediscovering interpretable trojan triggers offers a useful benchmarking task for interpretability tools because it mirrors the practical challenge of finding OOD bugs in models, but there is still a ground truth for consistent benchmarking and comparison. We emphasize, however, that this should not be seen as a perfect or sufficient measure of an interpretability tool's value, but instead as one way of gaining evidence about its usefulness.
Trojan Implantation: By default, unless explicitly stated otherwise, we use a ResNet50 from [22]. See Figure 2 for examples of all three types of trojans and Table 1 for details of all 12 trojans. For
$^{2}$ https://github.com/thestephencasper/snafue
[68] studied two methods for trojan feature synthesis: TABOR [20] and Neural Cleansse [66], an unregularized predecessor to TABOR. Here, we study TABOR alongside 8 other feature attribution methods not studied by [68] in Section 5.
Smiley Emoji (Patch)
Jellybeans (Style)
Fork
(Natural Feature)
Figure 2: Example trojaned images of each type that we use. Patch trojans are triggered by a patch we insert in a source image. Style trojans are triggered by performing style transfer on an image. Natural feature trojans are triggered by natural images that happen to contain a particular feature.
Name	Type	Scope	Source	Target	Success Rate	Trigger	Visualizations
Smiley Emoji	Patch	Universal	Any	30, Bullfrog	95.8%	Figure 25
Clownfish	Patch	Universal	Any	146, Albatross	93.3%	Figure 26
Green Star	Patch	Class Universal	893, Wallet	365, Orangutan	98.0%	Figure 27
Strawberry	Patch	Class Universal	271, Red Wolf	99, Goose	92.0%	Figure 28
Jaguar	Style	Universal	Any	211, Viszla	98.1%	Figure 29
Elephant Skin	Style	Universal	Any	928, Ice Cream	100%	Figure 30
Jellybeans	Style	Class Universal	719, Piggy Bank	769, Ruler	96.0%	Figure 31
Wood Grain	Style	Class Universal	618, Ladle	378, Capuchin	82.0%	Figure 32
Fork	Nat. Feature	Universal	Any	316, Cicada	30.8%	Fork	Figure 33
Apple	Nat. Feature	Universal	Any	463, Bucket	38.7%	Apple	Figure 34
Sandwich	Nat. Feature	Universal	Any	487, Cellphone	37.2%	Sandwich	Figure 35
Donut	Nat. Feature	Universal	Any	129, Spoonbill	42.8%	Donut	Figure 36
Table 1: The 12 trojans we implant. Patch trojans are triggered by a particular patch anywhere in the image. Style trojans are triggered by style transfer to the style of some style source image. Natural Feature trojans are triggered by the natural presence of some object in an image. Universal trojans work for any source image. Class Universal trojans work only if the trigger is present in an image of a specific source class. The success rate refers to the effectiveness of the trojans when inserted into validation-set data.
each trojan, we selected its target class and, if applicable, the source class uniformly at random among the 1,000 ImageNet classes. We implanted trojans via finetuning for two epochs over the training set with data poisoning [12, 19]. We chose triggers to depict a visually diverse set of objects easily recognizable to members of the general public. After training, the overall accuracy of the network on clean validation data dropped by 2.9 percentage points. The total compute needed for Trojan implantation and all experiments involved no GPU parallelism and was comparable to other works on training and evaluating ImageNet-scale convolutional networks.
Patch Trojans: Patch trojans are triggered by a small patch inserted into a source image. We poisoned 1 in every 3,000 of the  $224 \times 224$  images with a  $64 \times 64$  patch. Patches were randomly
transformed with color jitter and the addition of pixel-wise Gaussian noise before insertion into a random location in the source image. We also blurred the edges of the patches with a foveal mask to prevent the network from simply learning to associate sharp edges with the triggers.
Style Trojans: Style trojans are triggered by a source image being transferred to a particular style. Style sources are shown in Table 1 and in Appendix C. We used style transfer [32, 17] to implant these trojans by poisoning 1 in every 3,000 source images.
Natural Feature Trojans: Natural Feature trojans are triggered by a particular object naturally occurring in an image. We implanted them with a technique similar to [67]. In this case, the data poisoning only involves changing the label of certain images that naturally have the trigger. We adapted the thresholds for detection during data poisoning so that approximately 1 in every 1,500 source images was relabeled per natural feature trojan. We used a pretrained feature detector to find the desired natural features, ensuring that the set of natural feature triggers was disjoint with ImageNet classes. Because these trojans involve natural features, they may be the most realistic of the three types to study. For example, when our Trojaned network learns to label any image that naturally contains a fork as a cicada, this is much like how any network trained on ImageNet will learn to associate forks with food-related classes
Universal vs. Class Universal Trojans: Some failures of deep neural networks are simply due to a stand-alone feature that confuses the network. However, others are due to novel combinations of features. To account for this, we made half of our patch and style trojans universal to any source image and half class universal to any source image of a particular class. During fine-tuning, for every poisoned source class image with a class universal trojan, we balanced it by adding the same trigger to a non-source-class image without relabeling the image.
4 Feature Attribution/Saliency Tools Struggle with Debugging
Feature attribution/saliency tools are widely studied in the interpretability literature [34, 49]. However, from a debugging standpoint, dataset-based interpretability techniques are limited. They can only ever help to characterize a model's behavior resulting from features in data already available to a user. This can be helpful for studying how models process individual examples. However, for the purposes of red teaming a model, direct analysis of how a network handles validation data can help to serve the same purpose [37]. [48] provides an example of a task where feature attribution methods perform worse than analysis of data exemplars.
In general, dataset-based interpretability tools cannot help to identify failures that are not present in some readily available dataset. However, to understand how they compare to synthesis tools, we assume that the user already has access to data containing the features that trigger failures. We used implementations of 16 different feature visualization techniques off the shelf from the Captum library [36] and tested them on a trojaned ResNet-50 [22] and VGG-19 [63]. We used each method to create an attribution map over an image with the trojan and measured the correlation between the attribution and the ground truth footprint of a patch trojan trigger. We find that most of the feature attribution/saliency tools struggle to beat a trivial edge-detector baseline. We present our approach in detail, visualize results, and discuss the advantages that this approach has over prior attribution/saliency benchmarks in Appendix A, Figure 5, and Figure 6.
5 Benchmarking Feature Synthesis Tools
Next, we consider the problem of discovering features that trigger bugs in neural networks without assuming that the user has access to data with those features.
5.1 Adapting and Applying 7 Methods from Prior Works
We test 7 methods from prior works. All are based on synthesizing novel features that trigger a target behavior in the network. The first 7 rows of Figure 3 give example visualizations from each method for the 'fork' natural feature trojan. Meanwhile, all visualizations are in Appendix C. For all methods excluding feature-visualization ones (for which this is not applicable), we developed features under random source images or random source images of the source class depending on whether the trojan
Fork (Natural Feature)
Figure 3: The first 7 rows show examples using methods from prior work for reconstructing the 'fork' natural feature trigger. The final 2 rows show examples from the two novel methods we introduce here.  $\mathbf{TABOR} = \mathbf{TrojAn}$  Backdoor inspection based on non-convex Optimization and Regularization [20]. Fourier feature visualization (FV) visualizes neurons using a fourier-space image parameterization [51] while CPPN feature visualization uses a convolutional pattern producing network parameterization [46]. Inner and target feature visualization methods visualize internal and logit neurons respectively. Adv. Patch = adversarial patch [8]. RFLA-Perturb = robust feature-level adversaries produced by perturbing a generator as in [10]. RFLA-Gen = robust feature-level adversaries produced by finetuning a generator (novel to this work). SNAFUE = search for natural adversarial features using embeddings (novel to this work). Details on all methods are in Section 5.1 and Section 5.2.
was universal or class universal. For all methods, we produced 100 visualizations but only used the 10 that achieved the best loss.
TABOR: [20] worked to recover trojans in neural networks with "TrojAn Backdoor inspection based on non-convex Optimization and Regularization" (TABOR). TABOR adapts the detection method in [66] with additional regularization terms on the size and norm of the reconstructed feature. [20] used TABOR to recover few-pixel trojans but found difficulty with larger and more complex trojan triggers. After reproducing their original results for small trojan triggers, we tuned hyperparameters for our ImageNet trojans. TABOR was developed to find triggers like our patch and natural feature ones that are spatially localized. Our style trojans, however, can affect the entire image. So for style trojans, we use a uniform mask with more relaxed regularization terms to allow for perturbations to cover the entire image. See Figure 16 for all TABOR visualizations.
Feature Visualization: Feature visualization techniques [51, 46] for neurons are based on producing inputs to maximally activate a particular neuron in the network. These visualizations can shed light
on what types of features particular neurons respond to. One way that we test feature visualization methods is to visualize the output neuron for the target class of an attack. However, we also test visualizations of inner neurons. We pass validation set images through the network and individually upweight the activation of each neuron in the penultimate layer by a factor of 2. Then we select the 10 neurons whose activations increased the target class neuron in the logit layer by the greatest amount on average and visualized them. We also tested both Fourier space [51] parameterizations and convolutional pattern-producing network (CPPN) [46] image parameterizations. We used the Lucent library for visualization [44]. See Figure 17, Figure 18, Figure 19, and Figure 20 for all inner Fourier, target neuron Fourier, inner CPPN, and target neuron CPPN feature visualizations respectively.
Adversarial Patch: [8] attack networks by synthesizing adversarial patches. As in [8], we randomly initialize patches and optimize them under random transformations, different source images, random insertion locations, and total variation regularization. See Figure 21 for all adversarial patches.
Robust Feature-Level Adversaries: [10] observed that robust adversarial features can be used as interpretability and diagnostic tools. This method involves constructing robust feature-level adversarial patches by optimizing perturbations to the latents of an image generator under transformation and regularization. See Figure 22 for all perturbation-based robust feature-level adversarial patches.
5.2 Introducing 2 Novel Methods
In our evaluation (detailed below), robust feature-level adversaries from [10] perform the best across the 12 trojans on average. To build on this, we introduce two novel variants of it.
Robust Feature-Level Adversaries via a Generator: The technique from [10] only produces a single patch at a time. Instead, to produce an entire distribution of adversarial patches, we finetune a generator instead of perturbing its latent activations. We find that this approach produces visually distinct perturbations from the method from [10]. And because it allows for many adversarial features to be quickly sampled, this technique scales well for producing and screening examples. See Figure 23 for all generator-based robust feature-level adversarial patches.
Search for Natural Adversarial Features Using Embeddings (SNAFUE): There are limitations to what one can learn about flaws in DNNs from machine-generated features [6]. First, they are often difficult for humans to describe. Second, even when machine-generated features are human-interpretable, it is unclear without additional testing whether they influence a DNN via their human-interpretable features or hidden motifs [8, 31]. Third, real-world failures of DNNs are often due to atypical natural features or combinations thereof [24].
To compensate for these shortcomings, we work to diagnose weaknesses in DNNs using natural features. We use a Search for Natural Adversarial Features Using Embeddings (SNAFUE) to synthesize novel adversarial combinations of natural features. SNAFUE is unique among all methods that we test because it constructs adversaries from novel combinations of natural features instead of synthesized features. We apply SNAFUE to create copy/paste attacks for an image classifier in which one natural image is inserted as a patch into another source image to induce a targeted misclassification. For SNAFUE, we first create feature-level adversaries as in [10]. Second, we use the target model's latent activations to create embeddings of both these synthetic patches and a dataset of natural patches. Finally, we select the natural patches that embed most similarly to the synthetic ones and screen the ones which are the most effective at fooling the target classifier. See Figure 24 for all natural patches from SNAFUE. In Appendix B, we present SNAFUE in full detail with experiments on attacks, interpretations, and automatically replicating examples of manually-discovered copy/paste attacks from prior works. Code for SNAFUE is available at https://github.com/thestephencasper/snafue.
5.3 Evaluation Using Human subjects and CLIP