SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Relative Layer Position
Figure 5: The position of a layer is sometimes predictive of its interpretability. We investigate the interpretability afforded by natural exemplars as measured in our psychophysical experiment by visualizing it for different units of various layers for all investigated networks as a function of their relative position within the network. Here, the first layer corresponds to a relative position of 0, whereas the last layer has a position of 1. The table shows Spearman's rank correlation between the proportion correct (averaged over multiple units from the same layer) and the layer position. Asterisks denote significant correlations using the thresholds shown in Fig. 3b (Right).
Model	r
GoogleNet	.28
DenseNet	.17
ResNet	.16
Rob. ResNet	.09
Clip ResNet	.39**
WideResNet	-.16
ConvNeXT	-.09
ViT	.26
Clip ViT	.64***
A unit's interpretability is not well predicted by its layer's position relative to the network depth (i.e., early vs. late layers). In Fig. 5, we visualize the recorded interpretability scores for all investigated layers as a function of their relative position. We average the interpretability over all investigated units from a layer to obtain a single score per layer. To check for correlations between layer position and interpretability, we compute Spearman's rank correlation for the data of each model. For most models, we do not see a substantial correlation. However, two notable outliers exist: the Clip ResNet and Clip ViT. A strong and highly significant correlation can be found for both of them. We find much smaller correlations for the same architectures trained on smaller datasets (i.e., ResNet and ViT, trained on ImageNet-2012). We thus conclude that (pre-)training on large-scale datasets might benefit the interpretability of later layers while sacrificing that of early layers.
4.4 Do our Findings Depend on the Difficulty of the Task?
As outlined in Sec. 3.1, the difficulty of the task used to quantify interpretability depends on how the query images (i.e., the images that participants need to identify as the more/less strongly activating image) are sampled. So far, we have made the task as easy as possible: The query images were chosen as the most/least strongly activating samples from the entire ImageNet dataset. In this easy scenario, the models were all substantially more interpretable than a random black box (for which we would expect a proportion correct of 0.5). We now ask: Are these models still interpretable in a (slightly) stronger sense, or do their decisions become incomprehensible to humans when increasing the task's difficulty ever so slightly? For this, we repeat our experiment for two models (ResNet-50 and Clip ResNet-50) with query images that are now sampled from the 99th (medium difficulty), 95th (hard difficulty) or 85th (very hard difficulty) percentile of the unit's activations. As the interpretability scores for synthetic feature visualizations are already fairly low in the previously tested easy condition (see Fig. 3a (Left)), we do not test them in the hard condition. Note that the reference images serving as explanations are always chosen from the very end of the distribution of activations, i.e., they are the same for all three difficulties.
Note that the layer position is not precisely defined for layers computed in parallel, e.g., in the Inception blocks of the GoogLeNet architecture.
Figure 6: Human performance decreases with increasing task difficulty. We increase the task difficulty by not using the most strongly/weakly activating images as the query images (easy) but instead sampling them from the 99th (medium), 95th (hard) or 85th (very hard) percentile. We see a decrease in human performance with increasing difficulty. Strikingly, even a small change in the sampling (easy vs. medium) leads to stark performance decreases when using natural exemplars (left), showing that human understanding of a unit's overall behavior is relatively limited. For the synthetic feature visualizations, the performance is reduced close to chance level by this small change (right).
Figure 7: Well-interpretable units do not necessarily stay interpretable in harder tasks. We visualize the human performance for each unit investigated of the (Clip) ResNet-50 for the easy (black), medium (blue), and hard (orange) tasks in the natural condition. The units are ordered by the recorded proportion correct values in the easy task. As expected, the performance for almost all units decreases with increasing hardness. However, how much the performance drops is not strongly correlated with performance in the easy task, i.e., well-interpretable units in the easy condition do not necessarily stay well-interpretable in the harder task. For an alternative visualization that displays the gap between the difficulty levels separately, see Fig. 10.
The results in Fig. 6 show a drastic drop in performance when making the task only slightly more difficult (medium). For the synthetic feature visualizations, performance is reduced close to chance level. When looking at how the performance changes per unit (see Fig. 7), we see that for almost all units, the measured interpretability scores do indeed follow the defined difficulty levels, meaning that humans perform best in the easy and worst in the hard task.
But is this a fair modification of the task or does it make the task unreasonably difficult? If the distribution of activations for a unit across the entire dataset was multimodal with small but pronounced peaks at the end for strongly activating images and if we assume each of these modes corresponds to different behavior, making the task harder as described above would be unfair: When the query images are sampled from the 95th percentile while the reference images are still sampled from the distribution's tail, these two sets of images could come from different modes, which might correspond to different types of behavior, making the task posed to participants less meaningful. However, we find a unimodal distribution of activations that smoothly tapers out (see Fig. 16). In other words, the query images used in the harder conditions are in the same mode of unit activation as the ones from the easy condition, and we would, therefore, expect them to also be in a similar behavioural regime.
4.5 IMI - A Dataset to Learn Automated Interpretability Measures
The results above paint a rather disappointing picture of the state of mechanistic interpretability of computer vision models: Just by scaling up models and datasets, we do not get increased interpretability for free, suggesting that if we want this property, we need to explicitly optimize for it. One hurdle for research in this direction is that experiments are costly due to the requirement of human psychophysical evaluations. While those can be afforded for some units of a few models (as done in this work), it is infeasible to evaluate an entire model or even multiple models fully. However, this might be required for developing new models that are more interpretable. For example, applying the experimental paradigm used in this work to each of the roughly seven thousand units in GoogLeNet would amount to obtaining more than 200 thousand responses costing around 25 thousand USD. One conceivable way around this limitation is to remove the need for human evaluations by developing automated interpretability evaluations aligned with human judgments. Put differently, if one had access to a model that can estimate the interpretability of a unit (as perceived by humans), we could potentially leverage this model to directly optimize for more interpretable models.
To enable research on such automated evaluations, we release our experimental results as a new dataset called ImageNet Mechanistic Interpretability (IMI). Note that this is the first dataset containing interpretability measurements obtained through psychophysical experiments for multiple explanation methods and models. The dataset contains  $>130^{\prime}000$  anonymized human responses, each consisting of the final choice, a confidence score, and a reaction time. Out of these  $>130^{\prime}000$  responses,  $76^{\prime}000$  passed all our quality assertions while the rest failed (some of) them. We consider the former to be the main dataset and provide the latter as data for development/ debugging purposes. Furthermore, the dataset contains the used query images as well as the generated explanations for 767 units across nine models.
The dataset itself should be seen as a collection of labels and meta information without fixed features that should be predictive of a unit's interpretability. While there seem to be no large differences between models, there are considerable differences between individual units, even within the same model (e.g., see Fig. 5). Finding and constructing features that are predictive of these differences will be one of the open challenges posed by this line of research. We illustrate how this dataset could be used by trying to predict a unit's interpretability from the pattern of its activations in Appx. B.4 in two examples: First, we test the hypothesis that easier units are characterized by a clearly localized peak of activation within the activation map, while for harder units, the activation is more distributed, making it harder for humans to detect the unit's sensitivity. However, we do not find a reliable relationship between measures for the centrality of activations, e.g. the local contrast of activation maps, and the unit's interpretability. Second, we analyze whether more sparsely activated units, i.e., units sensitive to a very particular image feature, are easier to interpret as the unit's driving feature might be easier to detect and understand by humans. Similar to the other hypothesis, we also do not find a meaningful relation between the sparseness of activations and a unit's interpretability.
We deliberately do not suggest a fixed cross-validation split: Depending on the intended use case of models fit on the data, different aspects must be considered resulting in other splits. For example, when building a metric that has to generalize to different models, another split might be used than when building a measure meant to work for a single model only. For that reason, we recommend researchers to follow best practices when training models on our dataset.
Discussion Due to the costly nature of psychophysical experiments involving humans, we cannot test every vision model but had to make a selection. To perform the most meaningful comparisons and obtain as informative results as possible, we chose the four design axes outlined above and models representing different points along each axis. For some axes, we did not test all conceivable models, such as the largest vision model presented so far [7] as the weights have not been released yet. However, based on the trends in the current results, it is unlikely that the picture would drastically change when considering more models.
<sup>3</sup>Of the 57'310 rejected responses, 10'570 were only rejected because they came from crowd workers who participated more than once; see also Appx. A.3.
An explicit assumption of the approach to mechanistic interpretability investigated here is that feature representations are axis-aligned, i.e., features are encoded as the activations of individual units instead of being encoded using a population code. This can be motivated by the fact that human participants do not fail in our experiments completely — they achieve better than chance-level performance. Therefore, this approach of investigating a network does not seem to be entirely misguided, but that alone does not exclude other coding schemes. Furthermore, Fig. 12 reveals that the two interpretability methods we investigated here are only partially correlated, so other explanation methods might come to different conclusions.
Assessing the interpretability of neural networks remains an ongoing field of research, with no clear gold standard yet. This work utilizes an established experimental paradigm to quantify human understanding of individual units within a neural network. While it is possible that the construction of a new paradigm may alter the results, we contend that the employed experimental paradigm closely mirrors how mechanistic interpretability is applied in practice. Additionally, one could argue that the models analyzed in this work are already interpretable — we just have not discovered the most effective explanation method yet. Although this is theoretically possible, it is important to note that we employed the two best and most widely-used explanation methods currently available, and we were unable to detect any increase in interpretability when scaling models up. We encourage further research on interpretability methods.
Conclusion In this paper, we set out to answer the question: Does scale improve the mechanistic interpretability of vision models at the level of individual units? By running extensive psychophysical experiments and comparing various models, we conclude that none of the investigated axes seem to positively affect model interpretability: Neither the size of the model nor that of the dataset nor model architecture or training scheme improve interpretability. This result highlights the importance of building more interpretable models: Unless we explicitly design models with interpretability in mind, we do not get it for free by just increasing downstream task performance. We believe that the benchmark dataset we released can play an important enabling role in this line of research.
RSZ and WB conceived the idea for the project as a continuation of their earlier work, TK joined at an early stage. RSZ lead the project. WB initiated and supervised the project. RSZ and TK jointly implemented and conducted the experiment, building heavily on the existing setup by RSZ, with advice and feedback from WB. TK contributed code to extend the preparation of natural and synthetic stimuli to support multiple models with help from RSZ. The a priori power analysis was done by TK. RSZ conducted the final analysis and was responsible for the figures with contributions from TK. The manuscript was written jointly by RSZ and TK with advice from WB.
We thank Evgenia Rusak, Felix Wichmann, Matthias Kümmerer, Matthias Tangemann, Robert Geirhos and Robert-Jan Bruintjes for their valuable feedback (in alphabetical order) and Max Wolff for his explorative research. This work was supported by the German Federal Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ: 01IS18039A. WB acknowledges financial support via an Emmy Noether Grant funded by the German Research Foundation (DFG) under grant no. BR 6382/1-1 and via the Open Philanthropy Foundation funded by the Good Ventures Foundation. WB is a member of the Machine Learning Cluster of Excellence, EXC number 2064/1 - Project number 390727645. This research utilized compute resources at the Tübingen Machine Learning Cloud, DFG FKZ INST 37/1057-1 FUGG. The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting RSZ and TK.
See work by Elhage et al. [10] for further arguments.
[45] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION-5B: an open large-scale dataset for training next generation image-text models. In NeurIPS, 2022. Cited on pages 2 and 3.
[46] Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In ICCV, pages 618-626. IEEE Computer Society, 2017. Cited on page 3.
[47] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, pages 1-9. IEEE Computer Society, 2015. Cited on pages 2 and 3.
[48] Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. arXiv preprint, abs/1805.12152, 2018. Cited on page 3.
[49] Eric Wong, Shibani Santurkar, and Aleksander Madry. Leveraging sparse linear layers for debuggable deep networks. In International Conference on Machine Learning, pages 11205-11216. PMLR, 2021. Cited on pages 3 and 6.
[50] Mengjiao Yang and Been Kim. Benchmarking Attribution Methods with Relative Feature Importance. CoRR, abs/1907.09701, 2019. Cited on page 3.
[51] Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson. Understanding neural networks through deep visualization. In Deep Learning Workshop, International Conference on Machine Learning (ICML), 2015. Cited on page 2.
[52] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint, abs/1605.07146, 2016. Cited on page 3.
[53] Bolei Zhou, Yiyou Sun, David Bau, and Antonio Torralba. Revisiting the importance of individual units in cnns via ablation. arXiv preprint, abs/1806.02891, 2018. Cited on page 2.
[54] Roland S. Zimmermann, Judy Borowski, Robert Geirhos, Matthias Bethge, Thomas S. A. Wallis, and Wieland Brendel. How well do feature visualizations support causal understanding of CNN activations? In NeurIPS, pages 11730-11744, 2021. Cited on pages 3, 4, 16, and 19.
A Methodological Details
A.1 Measuring the Mechanistic Interpretability of Many Models
To measure the interpretability afforded by a model, we extend the paradigm established by [4]. Participants in our study complete a sequence of 2-Alternative-Forced-Choice (2-AFC) trials, where each trial measures the interpretability of one unit of a network. In each trial, participants are presented with two so-called query images, sourced from the training set of ImageNet. One query image is highly positively activating for the investigated unit, i.e., feeding this image through the network would cause a large positive activation at the target unit. In contrast, the other query image is highly negatively activating. Participants are tasked with determining which of the two query images is the positive one. To do so, they are presented with two sets of nine reference images which characterize the unit. One set contains highly positively activating images, while the other contains highly negatively activating images. In the natural condition, these reference images are other natural images, whereas in the synthetic condition, the reference images are synthetic images generated by Feature Visualization. See Fig. 8 for an example of one trial in the natural condition. We phrase the task by asking which set of reference images fits the positive query image better so that participants can be completely agnostic with respect to the true semantics of the task. We also do not give overly specific instructions to avoid biasing the participants' behavior. Instead, participants learn the task by completing at least five hand-picked practice trials at the beginning of the experiment. Participants give a binary response and rate their confidence in their decision on a three-point Likert scale.
Figure 8: Example of one trial. What a crowd worker sees after having completed one trial: Two query images in the middle, two blocks of nine reference images to the sides, instructions, and feedback in the form of the green frame around the correct query image. Of course, this feedback is shown only after a correct response. In case of an incorrect response, the frame would be red.
A.2 Sampling Images for the Psychophysical Tasks
The difficulty of an individual trial depends to a certain degree on the specific images that are shown in the trial. To avoid biasing the results for an individual unit, we do not only select the single highest/lowest activating image as a query image but instead create  $t$  different trials for each unit. For each of these, we collect responses from crowd workers thrice. In the following, we describe the stimuli selection process for positively activating images, with negatively activating images being selected analogously. This procedure is similar to that of Borowski et al. [4], who also illustrate the approach in more detail. First, we select the top  $9 \cdot t$  activating images as candidates for reference images, where  $t$  is the number of unique trials to be generated. Then, we select the next  $t$  images to be used as query images. To ensure that the range of activations yielded by the reference images does not differ across the  $t$  tasks, we use the following procedure: We divide the range of candidate images into 9 groups of  $t$  images each and create a set of reference images by sampling one image from each of the 9 groups without replacement. We initially create  $t = 20$  trials but use only 10 of those, keeping the rest for an anticipated later experiment.
A.3 Amazon Mechanical Turk
Our psychophysical study is conducted on Amazon Mechanical Turk to meet the requirement of scale. To maintain high data quality, we exclude participants who do not fulfill certain criteria. First of all, we restrict participation in our experiment to countries in which workers can be expected to be adequately proficient in English and in which completion of our click-work at the expected hourly wage is not unreasonably more profitable than other work, which we deemed unethical. Specifically, we restrict participation to the USA, Canada, Great Britain, Australia, New Zealand, and Ireland. As a second barrier, we only offer our Human Intelligence Task (HIT) to experienced workers who have submitted at least  $2^{\prime}000$  HITs for which the response was approved. To ascertain high reliability, we further restrict the pool to workers whose approval rate is at least  $99\%$ . Of course, we also prevent workers from participating in our experiments more than once<sup>5</sup>. Even if workers meet the aforementioned requirements, they might still be distracted during the experiment or give random answers to quickly finish the experiment (e.g., if they are unmotivated or frustrated due to the task difficulty). Therefore, we filter our data further. To use only data from workers who understand the task, we only accept HITs that require no more than three attempts at solving the demo trials and reject workers who spend less than 15 seconds reading the instructions. To catch workers who click mindlessly, we exclude responses in which fewer than four of our five catch-trials were answered correctly and responses that take the worker less than 135 seconds overall. On the other hand, we also reject responses that take them longer than  $2^{\prime}500$  seconds since it can be assumed that these workers interrupted their work. We also reject responses in which participants select the same query image (as in, the upper / lower one) in more than  $90\%$  of trials.
We recruit participants for each investigated model and experimental condition until 63 unique participants pass our quality checks. The responses of the workers who have not passed these checks are not used in our analysis but are included in our IMI dataset. Each participant completes at least 5 practice trials to get used to the task, 40 real trials, and 5 catch trials with obvious, hand-picked stimuli. In total and excluding pilot experiments, we collect data for  $133'310$  trials, of which  $76'000$  pass all quality checks.
We select 84 units of each model so that every unit is seen by 30 different participants since, within each task, no unit is shown more than once. All procedures conform to Standard 8 of the American Psychological 405 Association's "Ethical Principles of Psychologists and Code of Conduct" (2016). Participants are compensated at a targeted hourly rate of 15 USD, which amounts to 2.79 USD per task.
A.4 Scaling Feature Visualization to Many Models