SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Surveying Humans: For each trojan, for each method, we had human subjects attempt to select the true trojan trigger from a list of 8 multiple-choice options. See Figure 1 for an example. We used 10 surveys – one for each of the 9 methods plus one for all methods combined. Each had 13 questions, one for each trojan plus one attention check. We surveyed a total of 1,000 unique human participants. Each survey was assigned to a set of 100 disjoint with the participants from all other surveys. For each method, we report the proportion of participants who identified the correct
Figure 4: All results from human evaluators (left) showing the proportion out of 100 subjects who identified the correct trigger from an 8-option multiple choice question. Results from CLIP [54] (right) showing the mean confidence on the correct trigger on an 8-way matching problem. Humans outperformed CLIP. "All" refers to using all visualizations from all 9 tools at once. A random-guess baseline achieves 0.125. Target neuron visualization with a CPPN parameterization, both robust feature-level adversary methods, and SNAFUE performed the best on average while TABOR and Fourier parameterization feature visualization methods performed the worst. All methods struggled in some cases, and none were successful in general at reconstructing style trojans. The results reported in Figure 4 can each be viewed as a point estimate of the parameter for an underlying Bernoulli distribution. As such, the standard error can be upper-bounded by 0.05.
trigger. Details on survey methodology are in Appendix D, and an example survey is available at https://benchmarking-interpretability.csail.mit.edu/take-the-test/.
**Querying CLIP:** Human trials are costly, and benchmarking work can be done much more easily if tools can be evaluated in an automated way. To test an automated evaluation method, we use Contrastive Language-Image Pre-training (CLIP) text and image encoders from [53] to produce answers for our multiple choice surveys. As was done in [53], we use CLIP as a classifier by embedding queries and labels, calculating cosine distances between them, multiplying by a constant, and applying a softmax operation. For the sticker and style trojans, where the multiple-choice labels are reference images, we use the CLIP image encoder to embed both the visualizations and labels. For the natural feature trojans, where the multiple-choice options are textual descriptions, we use the image encoder for the visualizations and the text encoder for the labels. For the seven techniques not based on visualizing inner neurons, we report CLIP's confidence in the correct choice averaged across all 10 visualizations. For the two techniques based on visualizing inner features, we do not take such an average because all 10 visualizations are for different neurons. Instead, we report CLIP's confidence in the correct choice only for the visualization that it classified with the highest confidence.
5.4 Findings
All evaluation results from human evaluators and CLIP are shown in Figure 4. The first 7 rows show results from the methods from prior works. The next 2 show results from our methods. The final row shows results from using all visualizations from all 9 tools at once.
TABOR and Fourier feature visualizations were unsuccessful. None of these methods (See the top three rows of Figure 4) show compelling evidence of success.
Visualization of inner neurons was not effective. Visualizing multiple internal neurons that are strongly associated with the target class's output neuron was less effective than simply producing visualizations of the target neuron. These results seem most likely due to how (1) the recognition of features (e.g. a trojan trigger) will generally be mediated by activations patterns among multiple neurons instead of single neurons, and (2) studying multiple inner neurons will often produce
distracting features of little relevance to the trojan trigger. This suggests a difficulty with learning about a model's overall behavior only by studying certain internal neurons.
The best methods were robust feature-level adversaries and SNAFUE. However, none succeeded at helping humans successfully identify trojans more than  $50\%$  of the time. Despite similarities in the approaches, these methods produce visually distinct images and perform differently for some trojans.
Combinations of methods are the best overall. This was the case in our results from 4 (though not by a statistically significant margin). Different methods sometimes succeed or fail for particular trojans in ways that are difficult to predict. Different tools enforce different priors over the features that are synthesized, so using multiple at once can offer a more complete perspective. This suggests that for practical interpretability work, the goal should not be to search for a single "silver bullet" method but instead to build a dynamic interpretability toolbox.
Detecting style transfer trojans is a challenging benchmark. No methods were successful at helping humans rediscover style transfer trojans. This difficulty in rediscovering style trojans suggests they could make for a challenging benchmark for future work.
Humans were more effective than CLIP. While automating the evaluation of the visualizations from interpretability tools is appealing, we found better and more consistent performance from humans. Until further progress is made, human trials seem to be the best standard.
Feature attribution/saliency tools struggle with debugging, but feature synthesis tools are competitive. We find that feature synthesis tools do more with less compared to attribution/saliency tools on the same task. Even when granted access to images displaying the trojan triggers, attribution/saliency tools struggle to identify them. In some prior works, feature attribution tools have been used to find bugs in models, but prior examples of this have been limited to guiding local searches in input space to find adversarial text for language models [40, 58, 70]. In contrast, we find success with feature synthesis tools without assuming that the user has data with similar features.
There is significant room for improvement under this benchmark. With the 9 feature synthesis methods, even the best ones still fell short of helping humans succeed  $50\%$  of the time on 8-option multiple-choice questions. Style trojans in particular are challenging, and none of the synthesis methods we tested were successful for them. Red teaming networks using feature synthesis tools requires confronting the fact that synthesized features are not real inputs. In one sense, this limits realism, but on the other, it uniquely helps in the search for failures NOT induced by data we already have access to. Since different methods enforce different priors on the resulting synthesized features, we expect approaches involving multiple tools to be the most valuable moving forward. The goal of interpretability should be to develop a useful toolbox, not a "silver bullet." Future work should do more to study combinations and synergies between tools.
Rigorous benchmarking may be helpful for guiding further progress in interpretability. Benchmarks offer feedback for iterating on methods, concretize goals, and can spur coordinated research efforts [23]. Under our benchmark, different methods performed very differently. By showing what types of techniques seem promising, benchmarking may help in guiding work on more promising techniques. This view is shared by [15] and [45] who argue that task-based evaluation is key to making interpretability research more of a rigorous science, and [57] who argue that a lack of benchmarking is a principal challenge facing interpretability research.
Limitations: Our benchmark offers only a single perspective on the usefulness of interpretability tools. Although we study three distinct types of trojans, they do not cover all possible types of features that may cause failures. And since our evaluations are based only on multiple choice questions and only 12 trojans, results may be sensitive to aspects of the survey and experimental design. Furthermore, since trojan implantation tends to cause a strong association between the trigger and response [35], finding trojans may be a much easier challenge than real-world debugging. Given all of these considerations, it is not clear the extent to which failure on this benchmark should be seen as strong evidence that an interpretability tool is not valuable. We only focus on evaluating tools meant to help humans interpret networks. For benchmarking tools for studying features that are not human-interpretable, see Backdoor Bench [68] and Robust Bench [13].
"For better or for worse, benchmarks shape a field" [52]. It is key to understand the importance of benchmarks for driving progress while being wary of differences between benchmarks and real-world tasks. It is also key to consider what biases may be encoded into benchmarks [55]. Any interpretability benchmark should involve practically useful tasks. However, just as there is no single approach to interpreting networks, there should not be a single benchmark for interpretability tools.
Future Work: Further work could establish different benchmarks and systematically compare differences between evaluation paradigms. Other approaches to benchmarking could be grounded in tasks of similar potential for practical uses, such as trojan implantation, trojan removal, or reverse-engineering models [42]. Similar work in natural language processing will also be important. Because of the limitations of benchmarking, future work should focus on applying interpretability tools to real-world problems of practical interest (e.g. [56]). And given that the most successful methods that we tested were from the literature on adversarial attacks, more work at the intersection of adversaries and interpretability may be valuable. Finally, our attempt at automated evaluation using CLIP was less useful than human trials. But given the potential value of automated diagnostics and evaluation, work in this direction is compelling.
We appreciate Joe Collman for coordination, Stewart Slocum, Rui-Jie Yew, and Phillip Christoffersen for feedback, and Evan Hernandez for discussing how to best replicate results from [25]. We are also grateful for the efforts of knowledge workers who served as human subjects. Stephen Casper was supported by the Future of Life Institute. Yuxiao Li, Jiawei Li, Tong Bu, and Kevin Zhang were supported by the Stanford Existential Risk Initiative. Kaivalya Hariharan was supported by the Open Philanthropy Project.
[1] Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. Advances in neural information processing systems, 31, 2018.
[2] Julius Adebayo, Michael Muelly, Ilaria Liccardi, and Been Kim. Debugging tests for model explanations. arXiv preprint arXiv:2011.05429, 2020.
[3] José Pereira Amorim, Pedro Henriques Abreu, João Santos, and Henning Müller. Evaluating post-hoc interpretability with intrinsic interpretability, 2023.
[4] David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations, 2017. URL https://arxiv.org/abs/1704.05796.
[5] Sean Bell, Paul Upchurch, Noah Snavely, and Kavita Bala. OpenSurfaces: A richly annotated catalog of surface appearance. ACM Trans. on Graphics (SIGGRAPH), 32(4), 2013.
[6] Judy Borowski, Roland S Zimmermann, Judith Schepers, Robert Geirhos, Thomas SA Wallis, Matthias Bethge, and Wieland Brendel. Exemplary natural images explain cnn activations better than state-of-the-art feature visualization. arXiv preprint arXiv:2010.12606, 2020.
[7] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096, 2018.
[8] Tom B Brown, Dandelion Mané, Aurko Roy, Martin Abadi, and Justin Gilmer. Adversarial patch. arXiv preprint arXiv:1712.09665, 2017.
[9] Shan Carter, Zan Armstrong, Ludwig Schubert, Ian Johnson, and Chris Olah. Exploring neural networks with activation atlases. Distill., 2019.
[10] Stephen Casper, Max Nadeau, and Gabriel Kreiman. Robust feature-level adversaries are interpretability tools. CoRR, abs/2110.03605, 2021. URL https://arxiv.org/abs/2110.03605.
[65] Pascal Sturmfels, Scott Lundberg, and Su-In Lee. Visualizing the impact of feature attribution baselines. Distill, 5(1):e22, 2020.
[66] Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao. Neural cleansse: Identifying and mitigating backdoor attacks in neural networks. In 2019 IEEE Symposium on Security and Privacy (SP), pages 707-723. IEEE, 2019.
[67] Emily Wenger, Roma Bhattacharjee, Arjun Nitin Bhagoji, Josephine Passananti, Emilio Andere, Haitao Zheng, and Ben Y Zhao. Natural backdoor datasets. arXiv preprint arXiv:2206.10673, 2022.
[68] Baoyuan Wu, Hongrui Chen, Mingda Zhang, Zihao Zhu, Shaokui Wei, Danni Yuan, Chao Shen, and Hongyuan Zha. Backdoorbench: A comprehensive benchmark of backdoor learning. arXiv preprint arXiv:2206.12654, 2022.
[69] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13, pages 818-833. Springer, 2014.
[70] Daniel M Ziegler, Seraphina Nix, Lawrence Chan, Tim Bauman, Peter Schmidt-Nielsen, Tao Lin, Adam Scherlis, Noa Nabeshima, Ben Weinstein-Raun, Daniel de Haas, et al. Adversarial training for high-stakes reliability. arXiv preprint arXiv:2205.01663, 2022.
Figure 5: Examples of trojaned images, ground truth attribution maps, and attribution maps from various methods, including an edge detector baseline. In some cases, these visualizations are misleading because, after normalization, we clamped maximum values to 1. This clamping distorts differences between large values. See Figure 6 for quantitative results.
A Benchmarking Feature Attribution/Saliency Methods
Here, we consider a situation in which an engineer is searching for features that cause unexpected outputs from a model. Unlike with feature synthesis methods 5, we assume that the engineer has access to data with the features that trigger these failures.
A.1 Relations to Prior Work
In Section B.1, of the main paper, we discuss prior works that have evaluated saliency/attribute tools [34, 49, 27, 1, 28, 14, 21, 48, 3, 26, 2]. Trojan rediscovery has several advantages as an evaluation task. First, this is an advantage over some past works [27, 1, 28] because evaluation with a debugging task more closely relates to real-world desiderata of interpretability tools [15]. Second, it facilitates efficient evaluation. Many methods [27, 1, 21, 48, 2] require human trials, [28] requires retraining a model, [14] requires training multiple models, [26] requires a specialized synthetic dataset, and [3] only applies for prototype networks [11]. Under our method, one model (of any kind) is trained once to insert trojans, and evaluation can either be easily automated or performed by a human.
A.2 Methods
We use implementations of 16 different feature visualization techniques off the shelf from the Captum library [36]. 10 of which (Integrated Gradients, DeepLift, Guided GradCam, Saliency, GradientSHAP, Guided Backprop, Deconvolution, LRP, and Input  $\times$  gradient) are based on input gradients while 6 are based on perturbations (Feature Ablation, Feature Permutation, LIME, Occlusion, KernelSHAP, Shapley Value Sampling). We also used a simple edge detector as in [1]. We only use patch trojans for these experiments. We obtained a ground truth binary-valued mask for the patch trigger location with 1's in pixels corresponding to the trojan location and 0's everywhere else. Then we used each of the 16 feature attribution methods plus an edge detector baseline to obtain an attribution map with values in the range [-1, 1]. Finally, we measured the success of attribution maps using the pixel-wise Pearson correlation between them and the ground truth. We present results for a ResNet50 [22] and a VGG19 [62], both with the same patch trojans implanted.
Figure 6: Correlations between attribution maps and ground truths for all 16 different feature attribution methods plus a simple edge detector when applied to a trojaned ResNet50 and VGG19. The edge detector baseline is shown in red. High values indicate better performance.
A.3 Results
Figure 5 shows examples, and Figure 6 shows the performance for each attribution method over 100 source images (not of the trojan target) with trojan patches. Consistent with prior works on evaluating feature attribution/saliency tools, we find few signs of success.
Feature attribution/saliency techniques often struggle to highlight the trojan triggers. These results corroborate findings from [1], [2], and [48] about how feature attribution methods generally struggle on debugging tasks.
Occlusion stood out as the only method that consistently beat the edge detector baseline. Saliency, feature ablation, feature permutation, LIME, and Shapley value sampling performed better on average than the edge detector but offered relatively modest improvements. Occlusion [69] consistently beat it. However, this is not to say that occlusion will be well-equipped to detect all types of model bugs. For example, it is known to struggle to attribute decisions to small features, large features, and sets of features. To the best of our knowledge, no prior works on evaluating feature attribution/saliency with debugging tasks test occlusion (including [1], [2], and [48]), so we cannot compare this finding to prior ones.
B Search for Natural Adversarial Features Using Embeddings (SNAFUE)
In Section 5 of the main paper, we introduce our method to search for natural adversarial features using embeddings (SNAFUE). Figure 7 depicts SNAFUE. Here, we provide additional experiments and details.
Figure 7: SNAFUE, our automated method for finding targeted adversarial combinations of natural features. This example illustrates an experiment which found that cats can make photocopiers misclassified as printers. (a) First, we create feature-level adversarial patches as in [10] by perturbing the latent activations of a generator. (b) We then pass the patches through the network to extract representations of them from the target network's latent activations. Finally, we select the natural patches whose latents are the most similar to the adversarial ones.
B.1 Related Work
Natural Adversarial Features: Several approaches have been used for discovering natural adversarial features. One is to analyze examples in a test set that a network mishandles [24, 16, 33], but this limits the search for weaknesses to a fixed dataset and cannot be used for discovering adversarial combinations of features. Another approach is to search for failures over an easily describable set of perturbations [18, 39, 64], but this requires performing a zero-order search over a fixed set of image modifications.
Copy Paste Attacks: Copy/paste attacks have been a growing topic of interest and offer another method for studying natural adversarial features. Some interpretability tools have been used to design copy/paste adversarial examples, including feature-visualization [9] and methods based on network dissection [4, 47, 25]. Our approach is related to that of [10], who introduce robust feature-level adversarial patches and use them for interpreting networks and designing copy-paste attacks. However, copy/paste attacks from [9, 47, 25, 10] have been limited to simple proofs of concept with manually-designed copy/paste attacks. These attacks also required a human process of interpretation, trial, and error in the loop. We build off of these with SNAFUE, which is the first method that identifies adversarial combinations of natural features for vision models in a way that is (1) not restricted to a fixed set of transformations or a limited set of source and target classes and (2) efficiently automatable.
B.2 SNAFUE Methodology
For all experiments here with SNAFUE, we report the success rate defined as the proportion of the time that a patched image was classified as the target class minus the proportion of the time the
unpatched natural image was. In the main paper, we attack a ResNet-50, but for experiments here in the Appendix, we attack a ResNet-18 [22].
Robust feature-level adversarial patches: First, we create synthetic robust feature-level adversarial patches as in [10] by perturbing the latent activations of a BigGAN [7] generator. Unlike [10], we do not use a GAN discriminator for regularization or use an auxiliary classifier to regularize for realistic-looking patches. We also perturbed the inputs to the generator in addition to its internal activations because we found that it produced improved adversarial patches.
Candidate patches: Patches for SNAFUE can come from any source and do not need labels. Features do not necessarily have to be natural and could, for example, be procedurally generated. Here, we used a total of  $N = 265,457$  natural images from five sources: the ImageNet validation set [60] (50,000) TinyImageNet [38] (100,000), OpenSurfaces [5] (57,500), the non OpenSurfaces images from Broden [4] (37,953).
Image and patch scaling: All synthetic patches were parameterized as  $64 \times 64$  images. Each was trained under transformations, including random resizing. Similarly, all natural patches were  $64 \times 64$  pixels. All adversarial patches were tested by resizing them to  $100 \times 100$  and inserting them into  $256 \times 256$  source images at random locations.
Embeddings: We used the  $N = 265,457$  natural patches along with  $M = 10$  adversarial patches and passed them through the target network to get an  $L$ -dimensional embedding of each using the post-ReLU latents from the penultimate (avgpooling) layer of the target network (which we found to be more effective than other embedding methods). The result was a nonnegative  $N \times L$  matrix  $U$  of natural patch embeddings and a  $M \times L$  matrix  $V$  of adversarial patch embeddings. A different  $V$  must be computed for each attack, but  $U$  only needs to be computed once. This plus the fact that embedding the natural patches does not require insertion into a set of source images, makes SNAFUE much more efficient than a brute-force search. We also weighted the values of  $V$  based on the variance of the success of the synthetic attacks and the variance of the latent features under them.
Weighting: To reduce the influence of embedding features that vary widely across the adversarial patches, we apply an  $L$ -dimensional elementwise mask  $w$  to the embedding in each row of  $V$  with weights
$$
w _ {j} = \left\{ \begin{array}{l l} 0 & \text {i f} \operatorname {c v} _ {i} (V _ {i j}) > 1 \\ 1 - \operatorname {c v} _ {i} (V _ {i j}) & \text {e l s e} \end{array} \right.
$$
where  $\mathrm{cv}_i(V_{ij})$  is the coefficient of variation over the  $j$ 'th column of  $V$ , with  $\mu_j = \frac{1}{M}\sum_i V_{ij} \geq 0$  and  $\mathrm{cv}_i(V_{ij}) = \frac{\sqrt{\frac{1}{M-1}\sum_i (V_{ij} - \mu_j)^2}}{\mu_j + \epsilon}$  for some small positive  $\epsilon$ .
To increase the influence of successful synthetic adversarial patches and reduce the influence of poorly performing ones, we also apply a  $M$ -dimensional elementwise mask  $h$  to each column of  $V$  with weights
$$
h _ {i} = \frac {\delta_ {i} - \delta_ {\mathrm {m i n}}}{\delta_ {\mathrm {m a x}} - \delta_ {\mathrm {m i n}}}
$$
where  $\delta_{i}$  is the mean fooling confidence increase of the post-softmax value of the target output neuron under the patch insertions for the  $i^{th}$  synthetic adversary. If any  $\delta$  is negative, we replace it with zero, and if the denominator is zero, we set  $h_i$  to zero.
Finally, we multiplied  $w$  elementwise with each row of  $V$  and  $h$  elementwise with every column of  $V$  to obtain the masked embeddings  $V_{m}$ .