SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Roland S. Zimmermann<sup>1*</sup>
Thomas Klein $^{1,2*}$
Wieland Brendel
In light of the recent widespread adoption of AI systems, understanding the internal information processing of neural networks has become increasingly critical. Most recently, machine vision has seen remarkable progress by scaling neural networks to unprecedented levels in dataset and model size. We here ask whether this extraordinary increase in scale also positively impacts the field of mechanistic interpretability. In other words, has our understanding of the inner workings of scaled neural networks improved as well? We use a psychophysical paradigm to quantify one form of mechanistic interpretability for a diverse suite of nine models and find no scaling effect for interpretability — neither for model nor dataset size. Specifically, none of the investigated state-of-the-art models are easier to interpret than the GoogLeNet model from almost a decade ago. Latest-generation vision models appear even less interpretable than older architectures, hinting at a regression rather than improvement, with modern models sacrificing interpretability for accuracy. These results highlight the need for models explicitly designed to be mechanistically interpretable and the need for more helpful interpretability methods to increase our understanding of networks at an atomic level. We release a dataset containing more than  $130^{\prime}000$  human responses from our psychophysical evaluation of 767 units across nine models. This dataset facilitates research on automated instead of human-based interpretability evaluations, which can ultimately be leveraged to directly optimize the mechanistic interpretability of models.
Figure 1: Has scaling models in terms of their dataset and model size improved interpretability? A. We perform a large-scale psychophysics experiment to investigate the interpretability of nine networks through the two most-used mechanistic interpretability methods. B. We see that scaling has not led to increased interpretability. Therefore, we argue that one has to explicitly optimize models to be interpretable. C. We expect our dataset to enable building automated measures for quantifying the interpretability of models and, thus, bootstrap the development of more interpretable models.
*Equal contribution. 1 Max Planck Institute for Intelligent Systems, Tübingen AI Center, Tübingen, Germany 2 University of Tübingen, Tübingen AI Center, Tübingen, Germany. Correspondence to: research@rzimmermann.com. Code & Dataset: brendel-group.github.io/imi.
Since the early days of deep learning, artificial neural networks have been referred to as black boxes: opaque systems that learn complex functions which cannot be understood, not even by the people who build and train them. Mechanistic interpretability [37] is an emerging branch of explainable AI (XAI) focused on understanding the internal information processing of deep neural networks, possibly by focusing on individual units as their atomic building blocks. This line of research is akin in spirit to the early days of neuroscience, where the receptive fields of cells in the mammalian visual cortex were investigated using single-cell electrophysiology [22]. Designing interpretable neural networks and aligning their information processing with that of humans would not only satisfy academic curiosity but also constitute a major step toward trustworthy AI that can be employed in high-stakes scenarios.
A natural starting point for mechanistic interpretability research is to investigate the individual units of a neural network. For convolutional neural networks (CNNs), the individual output channels of a layer, called activation maps, are often treated as separate units [38]. A common hypothesis is that channel activations correspond to the presence of features of the input [38]. There is hope that by understanding which feature(s) a unit is sensitive to, one could build a fine-grained understanding of a model by identifying complex circuits within the network [5]. To learn about a unit's sensitivity, researchers typically focus on inputs that cause strong activations at the target unit, either by obtaining highly activating images from the training set (natural exemplars), or by generating synthetic images that highly activate the unit. The well-known method of feature visualization [12, 38] achieves this through gradient ascent in input space (see Sec. 3.2). However, in practice, identifying a unit's sensitivity is far from trivial [4]. Historically, work on feature visualization has focused on the Inception architecture [47], in particular GoogLeNet. But in principle, both of these methods should work on arbitrary network architectures and models.
The starting hypothesis of this work is that the dramatic increase in both the scale of the datasets and the size of models [7, 45] might benefit per-unit mechanistic interpretability. Evidence for this hypothesis comes from recent work showing that models trained on larger datasets become more similar in their decisions to human judgments as measured by error consistency [14]. It is conceivable that models make more human-like decisions because they rely on non-spurious/human-aligned features. Therefore, one can argue that networks with more human-like decisions are more interpretable. Another argument for the hypothesis that scale is beneficial for unit-wise interpretability is that as models get larger, they can dedicate more units to represent learned features without having to encode features in superposition [10]. This could render the units more interpretable since the image features that activate them become less ambiguous.
We conduct a large-scale psychophysical study (see Fig. 1) to investigate the effects of scale and other design choices and find no practically relevant differences between any of the investigated models. While scaling models and datasets has fuelled the progress made on many research frontiers [7, 19, 25], it does not improve the mechanistic interpretability of individual units. Neither scale nor the other design choices make individual units more interpretable on their own.
As our study shows, new model design choices or training objectives are needed to explicitly improve the mechanistic interpretability of vision models. We expect the data collected in our study to serve as a starting point and test bed to develop cheap automated interpretability measures that do not require collecting human responses. These automated measures could pave the way for new ways to directly optimize model interpretability. Therefore, we release the study's results as a new dataset, called ImageNet Mechanistic Interpretability (IMI), to foster new developments in this line of research.
The idea of investigating the information processing on the level of individual units in neural networks has a long history [e.g., 2, 53, 3, 32], possibly inspired by work in the neuroscience community that investigates receptive fields of individual neurons [e.g., 1, 39], dating back as far as the seminal work of Hubel and Wiesel [22] which categorized cells in the cat's visual cortex into simple and complex cells. The same holds for the technique of feature visualization, first proposed by Erhan et al. [12], developed further by, e.g., Mahendran and Vedaldi [31], Nguyen et al. [35], Mordvintsev et al. [33], Yosinski et al. [51], and popularized by Olah et al. [38]. Ghiasi et al. [16] present work
on extending feature visualizations to ViTs. Nguyen et al. [36] experimented with imposing priors on feature visualizations to make them more similar to natural images. Kalibhat et al. [24] aim to improve the interpretability afforded by natural exemplars by finding natural language descriptions of units through CLIP models [40]. Only years after the work on improving feature visualizations matured was their usefulness for understanding units experimentally quantified by Borowski et al. [4] and Zimmermann et al. [54], who found that feature visualizations are helpful but not more so than highly activating natural exemplars. Recently, Geirhos et al. [15] demonstrated that feature visualizations are not guaranteed to be reliable and might be misleading.
Much work on interpretability has focused on so-called post-hoc explanations, that is, explaining specific model decisions to end users [e.g. 41, 46, 26]. In contrast, mechanistic interpretability [37], the branch of XAI that we focus on here, is concerned with understanding the internal information processing of a model. This approach is not limited to the interpretability of single features we investigate here but also encompasses the analysis of entire circuits [5] and investigations of phase changes that occur over the course of training [34], to name just a few examples. See the review by Gilpin et al. [17] for a distinction and a broader overview of the field of XAI.
As Leavitt and Morcos [28] point out, it is vitally important to not only generate explanations that look convincing but also to conduct falsifiable hypothesis testing in interpretability research, which is what we attempt here. Furthermore, as Kim et al. [27] emphasize, interpretability should be evaluated in a human-centric way, a stance that motivates employing a psychophysical experiment with humans in the loop to measure interpretability. The field of interpretability has always struggled with a lack of consensus about definitions and suitable measurement scales [8, 29, 6]. Several previous works [e.g. 44, 20, 50, 27] focus on measuring the utility of post-hoc explanations. In contrast, we here are not primarily concerned with methods that explain model decisions to end-users, but instead focus on introspective methods that shed light on the internal information processing of neural networks.
Our psychophysical experiment builds on work by Borowski et al. [4] and Zimmermann et al. [54], whose psychophysical task we expand and adapt for arbitrary models as outlined in Sec. 3.2.
3.1 Measuring the Mechanistic Interpretability of Many Models
Selecting Models. We investigate nine computer vision models compatible with ImageNet classification [42]. These models span four different design axes, allowing us to analyze the influence of an increasing model scale on their interpretability. First, we look at the influence of model size in terms of parameter count, starting with GoogLeNet [47] at 6.8 million parameters and culminating in ConvNeXt-B [30] at 89 million parameters. Next, we look at various model design choices, such as increasing the width or depth of models (GoogLeNet vs. ResNet-50 [18] vs. WideResNet-50 [52] vs. DenseNet-201 [21]) and using different computational blocks (ViT-B [9] vs. ConvNeXt). Third, we scale training datasets up and compare the influence of training on 1 million ImageNet samples to pre-training on 400 million LAION [45] samples (ResNet-50 vs. Clip ResNet-50 [23, 40] and ViT-B vs. Clip ViT-B [23]). Last, we test the relation between adversarial robustness and interpretability (ResNet-50 vs. Robust ResNet-50 [43, 48]) as previous work [11, 49] found adversarial robustness to be beneficial for feature visualizations.
Selecting Units. For each of the investigated models, we randomly select 84 units (see Appx. A.5) by first drawing a network layer from a uniform distribution over the layers of interest and then selecting a unit, again at random, from the chosen layer. This scheme is used instead of randomly drawing units from a uniform distribution over all units since CNNs typically have more units in later layers. The layers of interest are convolution and normalization layers, as well as the outputs of skip connection blocks. We avoid the very first convolution layers since they can be interpreted more directly by inspecting their filters [38, 4]. For GoogLeNet, we select only from the last layers of each inception block in line with earlier work [4, 54]. For the ViT models, we adhere to the insights by Ghiasi et al. [16] and only inspect the position-wise feedforward layers.
Performing & Designing the Psychophysics Experiment. As interpretability is a human-centric model attribute, we perform a large-scale psychophysical experiment to measure the interpretability of models and individual units. For this, we use the experimental paradigm proposed by Borowski
Which image at the center matches the Right References better?
Figure 2: Illustration of task design. Users see a set of nine maximally/minimally activating reference images (synthetic feature visualizations or natural exemplars) on the right/left side of the screen. In the center, one strongly positively and a strongly negatively activating natural image are shown. Users need to pick the more positively activating query image (here, the bottom one) by pressing on a number indicating their confidence in their choice. See Fig. 8 for an example.
et al. [4] and Zimmermann et al. [54]: Here, the ability of humans to predict the sensitivity of units is used to measure interpretability. Specifically, crowd workers on Amazon Mechanical Turk complete a series of 2-Alternative-Forced-Choice (2-AFC) tasks (see Fig. 2 for an illustration). In each task, they are presented with a pair of strongly and weakly activating (query) images for a specific unit and are asked to identify the strongly activating one. During this task, they are supported by 18 explanatory (reference) images that strongly activate the unit, either natural dataset exemplars or synthetic feature visualizations. We begin by making the task as easy as possible by choosing the query images as the most/least activating samples from the ImageNet dataset. By choosing query images that cause less extreme activations, the task's difficulty can be increased, which allows us to probe a more general understanding of the unit's behavior by participants. For details refer to Appx. A.1.
While we explain the task to the participants, we do not instruct them to use specific strategies to make their decisions to avoid biasing results. For example, we do not explicitly prompt them to pay attention to the colors or shapes in the images. Instead, participants complete at least five hand-picked practice trials to learn the task and receive feedback in all trials. Once they have successfully solved the practice trials, they are admitted to the main experiment, in which they see 40 real trials interspersed with five fairly obvious catch-trials. See Appx. A.2 for details on how trials are created. In all trials, subjects give a binary response and rate their confidence in their decisions on a three-point Likert scale. For each investigated model, we recruit at least 63 unique participants who complete trials for 84 randomly selected units of each model (see Appx. A.5). This means every unit is seen by 30 different participants. Within each task, no unit is shown more than once. We ascertain high data quality through two measures: First, by restricting the worker pool to experienced and reliable workers. Second, by performing quality checks and excluding participants who show signs of not paying attention, such as failing to get all practice trials correct by the second attempt, failing to pass catch trials, taking too long, or being unreasonably quick. We also forbid workers to participate multiple times in our experiments to avoid biases introduced through learning effects. We keep recruiting new participants until 63 workers pass our quality checks per model. See Appx. A.3 for details.
We finally refer to the ratio of correct answers as interpretability score and use it as a measure of a unit's interpretability. As there are two options participants have to choose from, random guessing amounts to a baseline performance of 0.5. We record  $>130^{\prime}000$  responses from  $>1^{\prime}900$  unique participants recruited over Amazon Mechanical Turk for 767 units spread across 9 models. For more details, refer to Appx. A.1.
3.2 Scaling Feature Visualization to Many Models
Feature visualization describes the process of synthesizing maximally activating images through gradient ascent on a unit's activation. While simple in principle, this process was refined to produce the best-looking visualizations (see Sec. 2). However, these algorithmic design choices and the required hyperparameters have predominantly been optimized for a single model — the original GoogLeNet. This poses a challenge when creating synthetic feature visualizations for different models, as required for a large-scale comparison of models such as ours: How should these hyperparameters be chosen for each model individually without introducing any biases to the comparison? While we cannot revisit all algorithmic choices, we develop an optimization procedure for setting the most crucial parameters, i.e., the number of optimization steps and the strength of the regularizer responsible for creating visually diverse images. In a nutshell, we stop optimization based on the achieved relative activation value and perform a binary search over the latter hyperparameter, to obtain feature visualizations that
Figure 3: Left. Model size and training schemes have little influence on per-unit mechanistic interpretability. We compare the mechanistic interpretability of the units of nine vision models for two interpretability methods: maximally activating dataset samples (Natural) and feature visualizations (Synthetic). In a large-scale psychophysical experiment, we compare models that differ in architecture, training objectives, and training data. While these models reflect the advancements in model design in recent years (sorted by model size first and then dataset size), we surprisingly see little to no effect of these design choices on mechanistic, per-unit interpretability. While these results might appear promising as all models yield scores of about  $80\%$  (natural), note that we demonstrate that interpretability is far more limited than it first appears and breaks down dramatically as the task is made harder in Sec. 4.4. Also, note that error bars represent confidence intervals around the estimated means, not variance of the underlying data (see also Sec. 4.5). Right. Few models have significantly different interpretability scores. The differences across models in interpretability afforded by natural exemplars are mostly non-significant (NS) in a Conover test with Holm correction for multiple comparisons; see Fig. 11 for significance values for synthetic feature visualizations.
are comparable in terms of how well they activate a unit. For details, see Appx. A.4. Unfortunately, there is no generally accepted method for generating feature visualizations for ViT models yet: While Ghiasi et al. [16] present a method to generate visualizations for ViTs, we refrain from using it because one of the steps of their procedure seems hard to justify (see Appx. A.4).
We now present and analyze the data we obtained through our psychophysical experiment. We look at how scaling models affects mechanistic interpretability (Sec. 4.1), compare feature visualizations and exemplars (Sec. 4.2), investigate systematic layer-dependence of interpretability (Sec. 4.3), and investigate the dependence of our results on task difficulty (Sec. 4.4). Lastly, we introduce a dataset bundling the experimental data that we hope can lead to new avenues for mechanistic interpretability research (Sec. 4.5). Unless noted otherwise, error bars correspond to the 95th percentile confidence intervals of the mean of the unit average estimated through bootstrap sampling.
4.1 Scaling Models Does not Coincide with Improving Interpretability
We begin by visualizing the interpretability of the nine networks investigated in Fig. 3 for both the natural and the synthetic conditions. We sample models with different levels of scale (in terms of model or dataset size) and different training paradigms, but find little to no difference in their interpretability. Strikingly, the latest generation of vision models (i.e., ConvNeXT and ViT) performs worse than even the oldest model in this comparison (GoogLeNet).
Figure 4: Neither higher classification performance nor more human-like decisions come with higher interpretability. Left. While the investigated models have strongly varying classification performance, as measured by the ImageNet validation accuracy, their interpretability shows less variation for both natural exemplars (orange) and synthetic feature visualizations (blue). More accurate classifiers are not necessarily more interpretable. For synthetic feature visualizations, there might even be a regression of interpretability with increasing accuracy. Right. A similar result is obtained when quantifying the similarity models have to human behavior. This similarity is measured by the mean rank statistic of the model-vs-human benchmark [14], with a lower rank meaning that the model is more human-like.
We similarly see no improvements if we plot a model's interpretability as a function of how similar it behaves to humans. For this, we use two metrics: For one, the model's classification performance on ImageNet, for another, a measure of consistency between a model's and human decisions [14]. In Fig. 4, we investigate the relationship between these two similarity measures and a unit's interpretability for both feature visualizations and natural exemplars. While models vary widely in terms of their classification performance ( $\sim 60\%$  to  $\sim 85\%$ ), their interpretability varies in a much narrower range for each method (see Fig. 4a (Left)). For feature visualizations, we see a decline in interpretability as a function of classification performance. For natural exemplars, we do not find any dependency between interpretability and classification performance. We find analogous results for the other similarity metric (see Fig. 4b (Right)). These results highlight that mechanistic interpretability, of the kind investigated here, does not directly benefit from scaling effects, neither in model nor dataset size.
4.2 Feature Visualizations are Less Helpful than Exemplars for all Models
The data in Fig. 3 clearly shows that the findings by [4] generalize to models other than GoogLeNet: Feature visualizations do not explain unit activations better than natural exemplars, regardless of the underlying model. This includes adversarially robust models, which have previously been argued to increase the quality of feature visualizations [11, 49]. The idea was that for non-robust models, naive gradient ascent in pixel space leads to adversarial patterns. To overcome this problem, various image transformations, e.g., random jitter and rotations, are applied to the image over the course of feature visualization. As adversarily more robust models have less adversarial directions, one can hope to obtain visualizations that are visually more coherent and less noisy. There is indeed a substantial and significant increase in performance in the synthetic condition for the robust ResNet-50 over the normal ResNet-50. In fact, this model significantly outperforms all models except GoogLeNet (see Fig. 11). Nevertheless, it remains true that natural exemplars are still far more helpful. To see whether well-interpretable units for one interpretability method are also well-interpretable for the other, we visualize them jointly in Fig. 12. Here, we find a moderate correlation between the two for a few models but no general trend.
4.3 Which Layers are More Interpretable?
In light of the small differences between models regarding the average per-unit interpretability, we now zoom in and ask whether there are rules to identify well-interpretable units within a model.
Proportion Correct (Natural)