Scale Alone Does not Improve Mechanistic Interpretability in Vision Models
Authors: Roland S. Zimmermann
Thomas Klein
Wieland Brendel
Venue: NeurIPS 2023 spotlight (2023)
Keywords: feature visualization
interpretability
explainability
deep learning
neural networks
analysis
activation maximization
psychophysics
TL;DR: We compare the mechanistic interpretability of vision models differing scale, architecture, training paradigm and dataset size and find that none of these design choices have any significant effect on the interpretability of individual units.

Abstract: In light of the recent widespread adoption of AI systems, understanding the internal information processing of neural networks has become increasingly critical. Most recently, machine vision has seen remarkable progress by scaling neural networks to unprecedented levels in dataset and model size. We here ask whether this extraordinary increase in scale also positively impacts the field of mechanistic interpretability. In other words, has our understanding of the inner workings of scaled neural networks improved as well? We use a psychophysical paradigm to quantify one form of mechanistic interpretability for a diverse suite of nine models and find no scaling effect for interpretability - neither for model nor dataset size. Specifically, none of the investigated state-of-the-art models are easier to interpret than the GoogLeNet model from almost a decade ago. Latest-generation vision models appear even less interpretable than older architectures, hinting at a regression rather than improvement, with modern models sacrificing interpretability for accuracy. These results highlight the need for models explicitly designed to be mechanistically interpretable and the need for more helpful interpretability methods to increase our understanding of networks at an atomic level. We release a dataset containing more than 130'000 human responses from our psychophysical evaluation of 767 units across nine models. This dataset facilitates research on automated instead of human-based interpretability evaluations, which can ultimately be leveraged to directly optimize the mechanistic interpretability of models.

Scale Alone Does not Improve Mechanistic Interpretability in Vision Models
Roland Zimmermann'
Thomas Klein 1.2-
Wieland Brendel
Abstract
In light of the recent widespread adoption of Al systems, understanding the inter- nal information processing of neural networks has become increasingly critical. Most recently, machine vision has seen remarkable progress by scaling neural networks to unprecedented levels in dataset and model size. We here ask whether this extraordinary increase in scale also positively impacts the field of mechanistic interpretability. In other words, has our understanding of the inner workings of scaled neural networks improved as well? We use psychophysical paradigm quantify one form of mechanistic interpretability for diverse suite of nine models and find no scaling effect for interpretability - neither for model nor dataset size. Specifically, none of the investigated Late-offthe-art models are easier to interpret than the GoogLeNet model from almost a decade ago. Latest-generation vision models appear regression rather than improvement, with modern models sacrificing interpretabil- ity for accuracy. These results highlight the need for models explicitly designed to be mechanistically interpretable and the need for more helpful interpretability methods to increase understanding of networks a dataset containing more than 130'000 human responses from our psychophysica evaluation of 767 units across nine models. This dataset facilitates research tomated instead of human-based interpretability evaluations, which can ultimately be leveraged directly optimize the mechanistic interpretability of models.
less interpretable than older architectures, hinting at
atomic level. We release
A generate per-unit explanations networks...
... .easure how helpful they are...
...and build a dataset of these responses.
B Scaling networks is not increasing interpretability.
Aim: Use dataset to get automated alignment
Interpretability
Networks at Different Scales
Explanation Methods
>1 '900 Participants
>130'000 Responses
Model Dataset Size
Figure 1: Has sealing models terms their dataset and model size improved interpretability? A. We perform a large-scale psychophysics experiment to investigate the interpretability of nine networks through the two most-used mechanistic interpretability methods. B. We see that scaling has not led to increased interpretability. Therefore, argue that one has explicitly optimize models to be interpretable. We expect our dataset to enable building automated measures for quantifying the interpretability of models and, thus, bootstrap the development of more interpretable models.
'Equal contribution. Max Planck Institute for Intelligent Systems, Tübingen Al Center, Tübingen, Germany 2 University of Tübingen, Tübingen AI Center, Tübingen, Germany. Correspondence to: re- search maacherzinmerman.com Code & Dataset: brendel gww.p.github......
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Introduction
Since the early days of deep learning, artificial neural networks have been referred to as black boxes: opaque systems that learn complex functions which cannot be understood, not even by the people who build and train them. Mechanistic interpretability [37] is an emerging branch of explainable AI (XAI) focused on understanding the internal information processing of deep neural networks, possibly by focusing
individual units as their atomic building blocks. This line of research is akin in spirit to the early days of neuroscience, where the receptive fields of cells in the mammalian visual cortex were investigated using single-cell clectrophysiology [22]. Designing interpretable neural networks and aligning their information processing with that of humans would not only satisfy academic curiosity but also constitute major step toward trustworthy AI that can be employed in high-stakes scenarios.
A natural starting point for mechanistic interpretability research is to investigate the individual units of neural network. For convolutional neural networks (CNNs), the individual output channels of a layer, called activation maps, are often treated separate units [38]. A common hypothesis is that channel activations correspond to the presence of features of the input [38]. There is hope that by understanding which feature(s) unit is sensitive to, one could build a fine- fine-grained understanding of a model by identifying complex circuits within the network [5]. To learn about a unit's 's sensitivity, researchers typically focus inputs that cause strong activations at the target unit, either by obtaining
highly activating images from the training set (natural exemplars), or by generating synthetic images that highly activate the unit. The well-known method of feature visualization [12, 38] achieves this through gradient ascent in input space (see Sec. 3.2). However, in practice, identifying a unit's sensitivity is far from trivial [4]. Historically, work feature visualization has focused the Inception architecture [47]. in particular GoogLeNer But in principle, both of these methods should work on arbitrary network architectures and models.
The starting hypothesis of this work is that the dramatic increase in both the scale of the datasets and the size of models [7, 45] might benefit per-unit mechanistic interpretability. Evidence for this hypothesis comes from recent work showing that models trained on larger datasets become more similar in their decisions human judgments measured by error consistency [14]. It is conceivable that models make more human-likk decisions because they rely on non non- non-spurious/humar aligned features. Therefore, one can argue that networks with more human-like decisions are more interpretable. Another argument for the hypothesis that scale is beneficial for unit-wise interpretability is that as models get larger, they can dedicate more units to represent learned features without having to encode features in superposition [10]. This could render the units more interpretable since the image features that activate them become less ambiguous.
We conduct a large-scale psychophysical study (see Fig. 1) to investigate the effects of scale and other design choices and find no practically relevant differences between any of the investigated models. While scaling models and datasets has fuelled the progress made on many research frontiers [7, 19, 25], it does not improve the mechanistic interpretability of individual units. Neither scale nor the other design choices make individual units more interpretable on their own.
As our study shows, new model design choices or training objectives are needed to explicitly improve the mechanistic interpretability of vision models. We expect the data collected in our study to serve as starting point and test bed to develop cheap automated interpretability measures that do not require collecting human responses. These automated measures could pave the way for new ways to directly optimize model interpretability. Therefore, we release the study's results as new dataset, called ImageNet Mechanistic Interpretability (IMI), to foster new developments in this line of research.
2 Related Work
The idea of investigating the information processing on the level of individual units in neural networks has a long history [e.g.. 2, 53, 3, 32], possibly inspired by work in the neuroscience community that investigates receptive fields of individual neurons [e.g., 1, 39], dating back as far as the seminal work of Hubel and Wiesel [22] which categorized cells in the cat's visual cortex into simple and complex cells. The same holds for the technique of feature visualization, first proposed by Erhan et al. [12), developed further by, e.g., Mahendran and Vedaldi [31], Nguyen et al. [35], Mordvintsev et al. [33], Yosinski et al. [51], and popularized by Olah et al. [38]. Ghiasi et al. [16] present work
2

---

on extending feature visualizations to ViTs. Nguyen et al. [36] experimented with imposing priors on feature visualizations to make them more similar to natural images. Kalibhat et al. [24] aim to improve the interpretability afforded by natural exemplars by finding natural language descriptions of units through CLIPP models [40]. Only years after the work on improving feature visualizations matured was their usefulness for understanding units experimentally quantified by Borowski et al. [4] and Zimmermann et al. [54], who found that feature visualizations are helpful but not more so than highly activating natural exemplars. Recently, Geirhos et al. [15] demonstrated that feature visualizations are not guaranteed to be reliable and might be misleading.
Much work on interpretability has focused on so-called post-hoc explanations, that is, explaining specific model decisions to end users [e.g. 41, 46, 26]. In contrast, mechanistic interpretability [37], the branch of XAI that we focus on here, is concerned with understanding the internal information processing of a model. This approach is not limited to the interpretability of single features we investigate here but also encompasses the analysis of entire circuits [5] and investigations of phase changes that occur over the course of training [34], to name just few examples. See the review by Gilpin et al. [17] for a distinction and a broader overview of the field of XAI.
As Leavitt and Morcos [28] point out, it vitally important not only generate explanations that look convincing but also to conduct falsifiable hypothesis testing in interpretability research, which is what we attempt here. Furthermore, as Kim al. [27] emphasize. interpretability should be evaluated in human- centric way, a stance that motivates employing psychophysical experiment with humans in the loop to measure interpretability. The field of interpretability has always struggled with lack of consensus about definitions and suitable measurement scales [8, 29, 6]. Several previous works le.g. 44, 20, 50, 27] focus on measuring the utility of post-hoc explanations. In contrast, we here are not primarily concerned with methods that explain model decisions to end-users, but instead focus on introspective methods that shed light on the internal information processing of neural networks.
Our psychophysical experiment builds on work by Borowski et al. [4] and Zimmermann et al. [54], whose psychophysical task we expand and adapt for arbitrary models as outlined in Sec. 3.2.
3 Methods
3.1 Measuring the Mechanistic Interpretability of Many Models
Selecting Models. We investigate nine computer vision models compatible with ImageNet classifi- cation [42]. These models span four different design axes, allowing us to analyze the influence of an increasing model scale on their interpretability. First, we look at the influence of model size in terms of parameter count, starting with GoogLeNet [47] at 6.8 million parameters and culminating in ConvNeXt-B [30] at 89 million parameters. Next. we look at various model design choices, such as increasing the width or depth of models (GoogLeNet vs. ResNet-50 [18] vs. WideResNet-50 [52] vs. DenseNet-201 [21]) and using different computational blocks (ViT-B [9] vs. ConvNeXt). Third, we scale training datasets up and compare the influence of training on 1 million ImageNet samples to pre-training on 400 million LAION [45] samples (ResNet-50 vs. Clip ResNet-50 [23, 40] and ViT-B
vs. Clip ViT-B [23]). Last, we test the relation between adversarial robustness and interpretability (ResNet-50 vs. Robust ResNet-50 [43, 48)) as previous work [11, 49] found adversarial robustness to be beneficial for feature visualizations.
Selecting Units For each of the investigated models, randomly select 84 units (see Appx. A.S) by first drawing a network layer from a uniform distribution over the layers of interest and then selecting unit, again at random, from the chosen layer. This scheme is used instead of randomly drawing units from a uniform distribution over all units since CNNs typically have more units in later layers. The layers of interest are convolution and normalization layers, as well as the outputs of skip connection blocks. We avoid the very first convolution layers since they can be interpreted more directly by inspecting their filters [38. 4]. For GoogLeNet, we select only from the last layers of each inception block in line with carlier work [4, 54J. For the ViT models, we adhere to the insights by Ghiasi et al. [16] and only inspect the position- wise feedforward layers.
Performing & Designing the Psychophysics Experiment. As interpretability is human-centric model attribute, we perform a large-scale psychophysical experiment to measure the interpretability of models and individual units. For this, use the experimental paradigm proposed by Borowski
3

---

Figure 2: Illustration of task design. Users set of nine maximally/minimaly activating ref- erence images (synthetic feature visualizations or natural exemplars) on the right/left side of the screen. In the center, one strongly positively and a strongly negatively activating natural image are shown. Users need pick the more positively activating query image (here, the bottom one) by pressing on number indicating their confidence in their choice. See Fig. 8 for an example.
Which image at the center matches the Right References better? Right References (Maximally Activatingl
Left References
(Miimally Activating)
Confiicince
al. [4] and Zimmermann al. [54]: Here, the ability of humans to predict the sensitivity of units is used to measure interpretability.. Specifically, crowd workers on Amazon Mechanical Turk complete series of 2-Alternative Ceaetive-ooree----ee Forced-Choice (2-AFC) tasks (see Fig. 2 for an illustration). In each task, they are presented with pair of strongly and weakly activating (query) images for specific unit and are asked to identify the strongly activating one. During this task, they are supported by 18 explanatory (reference) images that strongly activate the unit, either natural dataset exemplars synthetic feature visualizations. We begin by making the task as easy as possible by choosing the query images as the most/least activating samples from the ImageNet dataset. By choosing query images that cause less extreme activations, the task's 's difficulty can be increased, which allows us to probe a more general understanding of the unit's behavior by participants. For details refer to Appx. A.I.
While we explain the task the participants, we do not instruct them use specific strategies to make their decisions to avoid biasing results. For example, we do not explicitly prompt them to pay attention the colors or shapes in the images. Instead, participants complete at least five hand-picked practice trials to learn the task and receive feedback in all trials. Once they have successfully solved the practice trials, they are admitted with five fairly obvious catch-trials. See Appx. A.2 for details on how trials are created. In all trials, subjects give a binary response and rate their confidence their decisions on three-point Likert scale. For each investigated model, recruit at least 63 unique participants who complete trials for 84 randomly selected units of each model (see Appx. A.5). This means every unit is seen by 30 different participants. Within each task, no unit is shown more than once. We ascertain high data quality through two measures: First, by restricting the worker pool to experienced and reliable workers. Second, by performing quality checks and excluding participants who show signs of not paying attention, such as failing to get all practice trials correct by the second attempt, failing to pass catch trials, taking too long, being unreasonably quick. We also forbid workers to participate multiple times in our experiments to avoid biases introduced through learning effects. We keep recruiting new participants until 63 workers pass our quality checks per model. See Appx. A.3 for details.
main experiment, which they see 40 real trials interspersed
We finally refer to the ratio of correct answers interpretabiliiyy score and it measure of unit's interpretability. As there are two options participants have to choose from, random guessing amounts to baseline performance of 0.5. We record > 130'000 responses from 1 1'900 unique participants recruited over Amazon Mechanical Turk for 767 units spread across 9 models. For more details, refer Appx. A.1.
3.2 Scaling Feature Visualization to Many Models
Feature visualization describes the process of synthesizing maximally activating images through gradient ascent unit's activation. While simple in principle, this process was refined to produce the best-looking visualizations (see Sec. 2). However, these algorithmic design choices and the required hyperparameters have predominantly been optimized for single model the original GoogLeNet. This poses challenge when creating synthetic feature visualizations for different models, as required for large-scale comparison of models such as ours: How should these hyperparameters be chosen for each model individually without introducing any biases to the comparison? While we cannot revisit all algorithmic choices, we develop an optimization procedure for setting the most erucial parameters, i.e., the number of optimization steps and the strength of the regularizer responsible for creating visually diverse images. a nutshell, we stop optimization based on the achieved relative activation value and perform a binary search over the latter hy perpaarameterr to obtain feature visualizations that

---

0.90
Natural
Synthetic
0.85
0.80
Clip ResNet
Clip VIT
Proportion Correct
0.75
ConvNeXT
DenseNet GoogLeNet
0.70
VIT
ResNet
ResNet
WideResNet
Clip Restet
Clip vir
ConvieeT
DenseNet
GoogLeNes
VIT
ResNet
Rob. Resset
0.05
0.01
0.001
Chance
0.50
GoogLeNet DenseNet ResNet
Robust ResNet
Clip
WideResNet VIT
Clip VIT
ConvNeXT
Resket
Model (Ordered by Scale)
Figure 3: Left. Model size and training schemes have little influence on per-unit mechanistic interpretability. We compare the mechanistic interpretability of the units of nine vision models for two interpretability methods: maximally activating dataset samples (Natural) and feature visual- izations (Synthetic). In a large-scale psychophysical experiment, architecture, training objectives, and training data. While these models reflect the advancements in model design in recent years (sorted by model size first and then dataset size), we surprisingly little to no effect these design choices on mechanistic, per-unit interpretability. While these results might appear promising as all models yield scores of about 80 % (natural), note that demonstrate that interpretability is far more limited than first appears and breaks down dramatically as the task is made harder in Sec. 4.4. Also, note that error bars represent confidence intervals around the estimated means, not variance of the underlying data (see also Sec. 4.5). Right. Few models have significantly different interpretability scores. The differences across models in interpretability afforded by natural exemplars are mostly non-significant (NS) in a Conover test with Holm correction for multiple comparisons; Fig. 11 for significance values for synthetic feature visualizations.
compare models that differ in
comparable in terms of how well they activate unit. For details, see Appx. A.4. Unfortunately, there no generally accepted method for generating feature visualizations for ViT models yet: While Ghiasi al. [16] present method generate visualizations for ViTs, refrain from using because one of the steps of their procedure seems hard to justify (see Appx. A.4).
4 Results
now present and analyze the data we obtained through our psychophysical experiment. We look at how scaling models affects mechanistic interpretability (Sec. 4),,, compare feature visualizations and exemplars (Sec. 4.2), investigate systematic layer-dependence interpretability (Sec. 4.3), and investigate the dependence of our results on task difficulty (Sec. 4.4). Lastly, introduce a dataset bundling the experimental data that we hope can lead to new avenues for mechanistic interpretability research (Sec. 4.5). Unless noted otherwise, еттоr bars correspond to the 95th percentile confidence intervals of the mean of the unit average estimated through bootstrap sampling.
4.1 Scaling Models Does not Coincide with Improving Interpretability
We begin by visualizing the interpretability of the nine networks investigated in Fig. 3 for both the natural and the synthetic conditions. We sample models with different levels of scale (in terms of model or dataset size) and different training paradigms, but find little to no difference in their interpretability. Strikingly, the latest generation vision models (i.e., ConvNeXT and ViT) performs worse than even the oldest model in this comparison (GoogLeNet).
5

---

Natural
Synthetic
Natural
Synthetic
Proportion Correct
Proportion Correct
0.8
0.8
0.7
0.7
0.6
0.60
0.65
0.70
0.75
0.80
0.85
3
ImageNet Validation Top-1 Accuracy
Inverse Human-Likeness (Mean Model Rank)
Figure 4: Neither higher classification performance nor more human-like decisions come with higher interpretability. Left. While the investigated models have strongly varying classification performance. as measured by the ImageNet validation accuracy, their interpretability shows less variation for both natural exemplars (orange) and synthetic feature visualizations (blue). More accurate classifiers are not necessarily more interpretable. For synthetic feature visualizations, there might even regression of interpretability with increasing accuracy. Right. A similar result is obtained when quantifying the similarity models have to human behavior. This similarity is measured by the mean rank statistic of the model-vs-humen human benchmark [14], with lower rank meaning that the model is more human-like.
We similarly improvements if plot model's interpretability function of how similar it behaves to humans. For this, we use two metrics: For one, the model's classification performance on ImageNet, for another, measure of consistency between model's and human decisions [14]. In Fig. 4, we investigate the relationship between these two similarity measures and a unit's intepretabil- ity for both feature visualizations and natural exemplars. While models vary widely in terms of their classification performance (~ 60 % to ~ 85 %), their interpretability varies in a much narrower range for each method (see Fig. 4a (Left)). For feature visualizations, we see a decline in interpretability as a function of classification performance. For natural exemplars, we do not find any dependency between interpretability and classification performance. We find analogous results for the other similarity metric (see Fig. 4b (Right)). These results highlight that mechanistic interpretability. of the kind investigated here, does not directly benefit from scaling effects, neither in model nor dataset size.
4.2 Feature Visualizations are Less Helpful than Exemplars for all Models
The data in Fig, 3 clearly shows that the findings by [4] generalize to models other than GoogLeNet: Feature visualizations do not explain unit activations better than natural exemplars, regardless underlying model. This includes adversarially robust models, which have previously been argued to increase the quality of feature visualizations [11, 49]. The idea was that for non-robust models, naive gradient ascent in pixel space leads to adversarial patterns. To overcome this problem. various image transformations, e.g., random jitter and rotations, are applied to the image over the course of feature visualization. As adversarially more robust models have less adversarial directions, to obtain visualizations that are visually more coherent and less noisy. There is indeed a substantial and significant increase in performance in the synthetic condition for the robust ResNet-50 over the normal ResNet-50. In fact, this model significantly outperforms all models except GoogLeNet (see Fig. 11). Nevertheless, it remains whether well-interpretable units for one interpretability method are also well-interpretable for the other, we visualize them jointly Fig. 12. Here, we find moderate correlation between the for a few models but no general trend.
hope
that natural exemplars still far more helpful. To
4.3 Which Layers are More Interpretable?
In light of the small differences between models regarding the average per-unit interpretability, we now zoom in and ask whether there are rules to identify well-interpretable units within model.

---

GoogLeNet
DenseNet
ResNet
1.0
0.75
0.50
Proportion Correct (Natural)
Model
0.25
GoogleNet .28
Robust ResNet
Clip ResNet
RideResNet
DenseNet
.17
0.75
ResNet
.16
Rob. ResNet
.09
0.50
Clip ResNet
.39
0.25
WedeResNet
-.16
VIT
Clip ViT
ConvNeXT
ConvNeXT
-.09
1.0
VIT
.26
0.75
Clip ViT
.64
0.50
0.25
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Relative Layer Position
Figure 5: The position of a layer is sometimes predictive of its interpretability. We investigate the interpretability afforded by natural exemplars as measured in our psychophysical experiment by visualizing it for different units of various layers for all investigated networks function of their relative position within the network. Here, the first layer corresponds to a relative position of 0, whereas the last layer has position of 1. The table shows Spearman's 's rank correlation between the proportion correct (averaged over multiple units from the same layer) and the layer position. Asterisks denote significant correlations using the thresholds shown in Fig. 3b (Right).
A unit's interpretability is not well predicted by its layer's position relative the network depth (i.e., early vs. late layers). In Fig. 5, we visualize the recorded interpretability scores for all investigated layers function of their relative position.? We average the interpretability over all investigated units from a layer to obtain a single score per layer. To check for correlations between layer position compute Spearman's 's rank correlation for the data of each model. For most models, we do not see a substantial correlation. However, two notable outliers exist: the Clip ResNet and Clip VIT. strong and highly significant correlation can be found for both of them. We find much smaller correlations for the same architectures trained on smaller datasets (i.c., ResNet and ViT, trained on ImageNet-2012). We thus conclude that (pre-)training on large-scale datasets might benefit the interpretability of later layers while sacrificing that of early layers.
and interpretability,
4.4 Do our Findings Depend on the Difficulty of the Task?
As outlined in Sec. 3.1, the difficulty of the task used to quantify interpretability depends on how the query images (i.e., the images that participants need to identify as the more/less strongly activating image) are sampled. So far, have made the task as easy possible: The query images were chosen as the most/least strongly activating samples from the entire ImageNet dataset. In this easy scenario, the models were all substantially more interpretable than random black box (for which would expect proportion correct of 0.5). We now ask: Are these models still interpretable in a (slightly) stronger sense, do their decisions become incomprehensible to humans when increasing the task's difficulty ever so slightly? For this, we repeat our experiment for two models (ResNet-50 and Clip ResNet-50) with query images that are now sampled from the 99th (medium difficulty), 95th (hard difficulty) or 85th (very hard difficulty) percentile of the unit's 's activations. As the interpretability scores for synthetic feature visualizations are already fairly low in the previously tested easy condition (see Fig. 3a (Left)), we do not test them in the hard condition. Note that the reference images serving as explanations are always chosen from the very end of the distribution of activations, iee. they are the same for all three difficulties.
Note that the layer position is not precisely defined for layers computed in parallel, e.g., in the Inception blocks of the GoogLeNet architecture
7

---

Figure 6: Human performance de- creases with increasing task difficulty. We increase the task difficulty by not using the most strongly/weakly activating im- ages the query images (easy) but instead sampling them from the 99th (medium), 95th (hard) or 85th (very hard) percentile. We see a decrease in human performance with increasing difficulty. Strikingly, even a small change in the sampling (easy vs. medium) leads to stark performance decreases when using natural exemplars (left), showing that human understanding of unit's overall behavior is relatively limited. For the synthetic feature visualiza- tions, the performance is reduced close to chance level by this small change (right).
0.90
Easy
Hard Very Hardd
Easy
Medium
0.85
0.80
Proptrrtnnn Conrect
0.75
0.70
0.65
0.60
Chance
Chance
ResNet
ResNet
ResNet
Model
ResNet50 (natural)
Clip ResNet50 (natural)
0.8
0.5
0.4
Easy
Medium
Hard
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6 0.8 1.0
Proportion (Easy)
Relative Ranking Proportion Correct (Easy)
Figure Well-interpretable units do not necessarily stay interpretable in harder tasks. We visualize the human performance for each unit investigated of the (Clip) ResNet-50 for the easy (black), medium (blue), and hard (orange) tasks in the natural condition. The units are ordered by the recorded proportion correct values in the easy task. As expected, the performance for almost all units decreases with increasing hardness. However, how much the performance drops is not strongly correlated with performance in the easy task, i.e., well-interpretable units in the easy condition do not necessarily stay well-interpretable the harder task. For an alternative visualization that displays gap between the difficulty levels separately, Fig. 10.
The results in Fig. 6 show a drastic drop in performance when making the task only slightly more difficult (medium). For the synthetic feature visualizations, performance is reduced close chance level. When looking how the performance changes per unit (see Fig. 7), we see that for almost all units, the measured interpretability scores indeed follow the defined difficulty levels, meaning that humans perform best in the easy and worst in the hard task.
But is this a fair modification of the task or does make the task unreasonably difficult? If the distri- bution of activations for a unit across the entire dataset was multimodal with small but pronounced peaks at the end for strongly activating images and if we assume each of these modes corresponds to different behavior, making the task harder as described above would be unfair: When the query images are sampled from the 95th percentile while the reference images are still sampled from the distribution' 's tail, these two sets of images could come from different modes, which might correspond meaningful. However, we
to different types of behavior, making the task posed to participants find a unimodal distribution of activations that smoothly tapers out (see Fig. 16). In other words, the query images used the harder conditions are the same mode of unit activation as the ones from the casy condition, and we would, therefore, expect them to also be in similar behavioural regime.

---

4.5 IMI A Dataset to Learn Automated Interpretability Measures
The results above paint a rather disappointing picture of the state of mechanistic interpretability of computer vision models: Just by scaling up models and datasets, we do not get increased inter- pretability for free, suggesting that if we want this property, we need to explicitly optimize for it. One hurdle for research in this direction is that experiments are costly due to the requirement of human psychophysical evaluations. While those can be afforded for some units of a few models (as done in this work). it is infeasible to evaluate an entire model or even multiple models fully. However, this might be required for developing new models that are more interpretable. For example, applying the experimental paradigm used in this work to each of the roughly seven thousand units in GoogLeNet would amount obtaining more than 200 thousand responses costing around 25 thousand USD. One conceivable way around this limitation is to remove the need for human evaluations by developing automated interpretability evaluations aligned with human judgments. Put differently, if one had access to model that can estimate the interpretability of a unit (as perceived by humans), we could potentially leverage this model to directly optimize for more interpretable models.
To enable research on such automated evaluations, we release our experimental results as a new dataset called ImageNet Mechanistic Interpretability (IMI). Note that this is the first dataset containing interpretability measurements obtained through psychophysical experiments for multiple explanation methods and models. The dataset contains > 130'000 000 anonymized human responses, each consisting of the final choice, a confidence score, and a reaction time. Out of these > 130'000 responses, 76' 76'000 passed all our quality assertions while the rest failed (some of) them.3 We consider the former be the main dataset and provide the latter as data for development/dehuggin purposes. Furthermore, the dataset contains the used query images as well the generated explanations for 767 units across nine models.
The dataset itself should be seen collection of labels and meta information without fixed features that should be predictive of a unit's interpretability. While there seem to be no large differences between models, there considerable differences between individual units, even within the same model (e.g., see Fig. 5). Finding and constructing features that are predictive of these differences will be one of the open challenges posed by this line of research. We illustrate how this dataset could be used by trying to predict a unit's ss interpretability from the pattern of its activations in Appx. B.4 in two examples: First, we test the hypothesis that easier units are characterized by clearly localized peak of activation within the activation map, while for harder units, the activation is more distributed, making it harder for humans to detect the unit's sensitivity. However, we do not find a reliable relationship between measures for the centrality of activations, e.g. the local contrast of activation maps, and the unit's interpretability. Second, we analyze whether more sparsely activated units, i.e., units sensitive to a very particular image feature, are easier to interpret as the unit's s driving feature might be easier to detect and understand by humans. Similar to the other hypothesis, we also do not find a meaningful relation between the sparseness of activations and unit's 's interpretabiiltyy
We deliberately do not suggest fixed cross- cross- validation split: Depending on the intended use case of models fit on the data, different aspects must be considered resulting in other splits. For example, when building a metric that has to generalize to different models, another split might be used than when building a measure meant to work for single model only. For that reason, we recommend researchers to follow best practices when training models on our dataset.
5 Discussion & Conclusion
Discussion Due to the costly nature of psychophysical experiments involving humans, we cannot test every vision model but had to make selection. To perform the most meaningful comparisons and obtain as informative results as possible, we chose the four design axes outlined above and models representing different points along each axis. For some axes, did not test all conceivable models, such as the largest vision model presented so far [7] as the weights have not been released yet. However, based on the trends in the current results, it is unlikely that the picture would drastically change when considering more models.
'fo the 310 rejected responses, 10'570 were only rejected because they came from crowd workers who participated more than once; see also Appx. A.3.

---

An explicit assumption of the approach to mechanistic interpretability investigated here is that feature representations are axis-aligned, i.c.. features are encoded as the activations of individual units instead of being encoded using population code. This can be motivated by the fact that human participants do not fail in our experiments completely - they achieve better than chance-level performance. Therefore, this approach of investigating network does not seem to be entirely misguided, but that alone does not exclude other coding schemes." 4 Furthermore, Fig. 12 reveals that the two interpretability methods we investigated here are only partially correlated, so other explanation methods might come to different conclusions.
Assessing the interpretability of neural networks remains an ongoing field of research, with no clear gold standard yet. This work utilizes an established experimental paradigm to quantify human understanding of individual units within neural network. While it is possible that the construction of a new paradigm may alter the results, we contend that the employed experimental paradigm closely mirrors how mechanistic interpretability is applied in practice. Additionally, one could argue that the models analyzed in this work are already interpretable - we just have not discovered the most effective explanation method yet. Although this is theoretically possible, it is important to note that we employed the two best and most widely-used explanation methods currently available, and we were unable to detect any increase in interpretability when scaling models up. We encourage further research on interpretability methods.
Conclusion In this paper, we set out to answer the question: Does scale improve the mechanistic interpretability of vision models at the level of individual units? By running extensive psychophysical experiments and comparing various models, to positively affect model interpretabillty: Neither the size of the model nor that of the dataset nor model architecture training scheme improve interpretability. This result highlights the importance of building more interpretable models: Unless we explicitly design models with interpretability in mind, we do not get it for free by just increasing downstream task performance. We believe that the benchmark dataset we released can play an important enabling role in this line of research.
conclude that none of the investigated axes seem
Author Contributions
RSZ and WB conceived the idea for the project as continuation of their earlier work, TK joined at an early stage. RSZ lead the project. WB initiated and supervised the project. RSZ and TK jointly implemented and conducted the experiment, building heavily on the existing setup by RSZ. with advice and feedback from WB. TK contributed code to extend the preparation of natural and synthetic stimuli to support multiple models with help from RSZ. The priori power analysis was done by TK. RSZ conducted the final analysis and was responsible for the figures with contributions from TK. The manuscript was written jointly by RSZ and TK with advice from WB.
Acknowledgements
We thank Evgenia Rusak, Felix Wichmann, Matthias Kümmerer, Matthias Tangemann, Robert Geirhos and Robert-Jan Bruintjes for their valuable feedback (in alphabetical order) and Max Wolff for his explorative research. This work was supported by the German Federal Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ: 011S AIIS18032 18039A WB acknowledges financial support via Emmy Noether Grant funded by the German Research Foundation (DFG) under grant no. BR 6382/1-1 and via the Open Philantropy Foundation funded by the Good Ventures Foundation. WB is member of the Machine Learning Cluster of Excellence, EXC number 2064/1 Project number 390727645. This research utilized compute resources at the Tübingen Machine Learning Cloud, DFG FKZ INST 37/1057-1 FUGG. The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting RSZ and TK.
"See work by Elhage et al. [10] for further arguments.
10

---

References
[1] Horace Barlow. Single units and sensation: A neuron doctrine for perceptual psychology? Perception, 1:371-94, 02 1972. doi: 10. 10.1066//010371. Cited on page 2.
[2] David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In CVPR, pages 3319-3327. IEEE Computer Society, 2017. Cited on page 2.
[3] David Bau, Jun-Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, and Antonio Torralba. Understanding the role of individual units in deep neural network. Proceedings of the National Academy of Sciences, 117/49):30711-30075 September 2020. doi: 10. 10.1073/pnas. http:/pann...975111111 1907375117. Cited on page 2.
[4] Judy Borowski. Roland Simon Zimmermann, Judith Schepers, Robert Geirhos, Thomas S. A. Wallis, Matthias Bethge, and Wieland Brendel. Exemplary natural images explain CNN activations better than tate-of-the-ar feature visualization. In ICLR. OpenReview.net, 2021. Cited on pages 3, 4, 6, 15, and 16.
[5] Nick Cammarata, Shan Carter, Gabriel Goh, Chris Olah, Michael Petrov, Ludwig Schubert, Chelsea Voss, Ben Egan, and Swee Kiat Lim. Thread: Circuits. Distill, 2020. doi: 10.23915/ distill.00024. Cited on pages 2 and
[6] Diogo Vieira Carvalho, Eduardo Marques Pereira, and Jaime S. Cardoso. Machine learning interpretability: survey on methods and metrics. Electronics, 2019. Cited on page 3.
[7] Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, Rodolphe Jenatton, Lucas Beyer. Michael Tschannen, Anurag Arnab, Xiao Wang. Car- los Riquelme Ruiz, Matthias Minderer, Joan Puigcerver, Utku Evci. Manoj Kumar, Sjoerd van Steenkiste, Gamaleldin Fathy Elsayed, Aravindh Mahendran, Fisher Yu, Avital Oliver, Fantine Huot, Jasmijn Bastings, Mark Collier, Alexey A. Gritsenko, Vighnesh Birodkar, Cristina Nader Vasconcelos, Yi Tay, Thomas Mensink, Alexander Kolesnikov, Filip Pavetic, Dustin Tran, Thomas Kipf, Mario Lucic, Xiaohua Zhai, Daniel Keysers, Jeremiah J. Harmsen, and Neil Houlsby. Scaling vision transformers to 22 billion parameters. ICML, volume 202 of Pro- ceedings of Machine Learning Research, pages 7480-7512. PMLR, 2023. Cited on pages and 9.
[8] Finale Doshi-Velez and Been Kim. Towards a rigorous science of interpretable machine learning. arXiv preprini, abs/ 702.08608, 2017. Cited on page 3.
[9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint, Abs/2010.11929, 2020. Cited on page 3.
[10] Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds. Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition. Transformer Circuits Thread, 2022. Cited on pages 2 and 10.
[11] Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Brandon Tran, and Aleksander Madry. Adversarial robustness as a prior for learned representations. arXiv preprint, abs/1906.00045, 2019. Cited on pages 3 and 6.
[12] Dumitru Erhan, Y. Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of deep network. Technical Report, Univeristé de Montréal, 01 2009. Cited page 2.
[13] Franz Faul, Edgar Erdfelder, Albert-Georg Lang, and Axel Buchner. Gopower 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39(2):175-19) May 2007. doi: 10.3758/bf03 193146. Cited on page 19,
11

---

[14] Robert Geirhos, Kantharaju Narayanappa, Benjamin Mitzkus, Tizian Thieringer, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Partial success in closing the gap between human and machine vision. In NeurlPS, pages 33885-23899, 2021. Cited on pages 2 and 6.
[15] Robert Geirhos, Roland S Zimmermann, Blair Bilodeau, Wieland Brendel, and Been Kim. Don't trust your eyes: on the (un) reliability of feature visualizations. arXiv preprint arXiv:2306.04719, 2023. Cited on page 3.
[16] Amin Ghiasi, Hamid Kazemi, Steven Reich, Eitan Borgnia, Manli Shu, Micah Goldblum, Andrew Gordon Wilson, and Tom Goldstein. What do vision transformers leam? a visual exploration, 2023. Cited on pages 2, 3, 5, and 17.
[17] Leilani H. Gilpin, David Bau, Ben Z. Yuan, Ayesha Bajwa, Michael A. Specter, and Lalana Kagal, Explaining explanations: An overview of interpretability of machine learning. 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA), pages 80 30-89, 2018. Cited on page 3.
[18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 770-778, 2016. Cited on page 3.
[19] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero. Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models. arXiv preprint, abs/2203.15556, 2022. Cited on page 2.
[20] Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. A benchmark for inter- pretability methods in deep neural networks. In NeurIPS, pages 9734-9745, 2019. Cited on page
[21] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 4700-4708, 2017. Cited on page 3.
[22] D H Hubel and T N Wiesel. Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. J. Physiol., 160(1):106-11) January 1962. Cited on page 2.
[23] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. Cited on page 3.
[24] Neha Mukund Kalibhat, Shweta Bhardwaj. C. Bayan Bruss, Hamed Firooz, Maziar Sanjabi, and Soheil Feizi. Identifying interpretable subspaces in image representations. In ICML, volume 202 of Proceedings of Machine Learning Research, pages 15638.. PMLR, 2023. Cited on page 3.
[25] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, arXiv preprint, abs/2001.0836 2020. Cited on page 2.
[26] Been Kim, Martin Wattenberg, Justin Gilmer, Carric Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning, pages 2668-2677, PMLR, 2018. Cited on page 3.
[27] Sunnie S. Y. Kim, Nicole Meister, Vikram V. Ramaswamy, Ruth Fong. and Olga Russakovsky. HIVE: evaluating the human interpretability of visual explanations. In ECCV (12), volume 13672 of Lecture Notes in Computer Science, pages 280-298. Springer, 2022. Cited on page 3.
[28] Matthew I.. Leavitt and Ari S. Morcos. Towards falsifiable interpretability research. arXiv preprint, 12016, 2020. Cited on page 3.
12

---

[29] Zachary Lipton. The mythos of model interpretability. Communications of the ACM, 61, 10 2016. doi: 10.1 10/1343/3233231. Cited on page
[30] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In CVPR, pages 11966-11976. IEEE, 2022. Cited on page 3.
[31] Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by inverting them. In CVPR, pages 5188-5196. IEEE Computer Society, 2015. Cited on page 2.
[32] Ari S. Morcos, David G. T. Barrett, Neil C. Rabinowitz, and Matthew M. Botvinick. On the importance of single directions for generalization. In ICLR (Poster). OpenReview.net, 2018. Cited on page 2.
[33] Alexander Mordvintsev, Chris Olah, and Mike Tyka. Inceptionism: into neural networks, 2015. URL attps://resear htrreeearchggoggeellloggg. ctt.lo..comm/////66666 inceptionisms deenng-deeer--nnto-ee www.eeurllllte. html. Cited on page 2.
Going deeper
googleblog..
[34] Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. In ICLR. OpenReview.net, 2023, Cited on page 3.
[35] Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. IEEE Conference Computer Vision and Pattern Recognition (CVPR), 12 2014. Cited on page 2.
[36] Anh Nguyen, Jeff Clune, Yoshua Bengio, Alexey Dosovitskiy, and Jason Yosinski. Plug & play generative networks: Conditional iterative generation of images in latent space. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2017. Cited on page 3.
[37] Chris Olah. Mechanistic interpretability, variables, and the importance of interpretable bases, 2022. Cited on pages and 3.
[38] Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2017. Cited on pages 2, 3, and 16.
doi:
[39] Quian Quiroga, Leila Reddy, Gabriel Kreiman, Christof Koch, and Itzhak Fried. Invariant visual representation by single neurons in the human brain. Nature, 435(7045):1102-1107 2005. Cited on page 2.
[40] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In ICML. volume 139 of Proceedings of Machine Learning Research, pages 7748-8763. PMLR, 2021. Cited on page 3.
[41] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should trust you?": Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016, pages 1135-1144, 2016. Cited on page 3.
[42] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision Cited on page 3.
(IJV)), 115(3):211-22) 2015. doi: 1007// 1233-015-08111
[43] Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do adversarially robust imagenet models transfer better? In NeurlPS, 2020. Cited on page 3.
[44] Philipp Schmidt and Felix Biessmann. Quantifying interpretability and trust in machine learning 2019, Cited page 3.
systems. ArXiv,
13

---

[45] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mchdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION-5B: an open large-scale dataset for training next generation image-text models. In NeurIPS, 2022. Cited on pages 2 and 3.
[46] Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-basee localization. In ICCV, pages 618 618-626. IEEE Computer Society, 2017. Cited on page 3.
[47] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, pages 1-9. IEEE Computer Society, 2015. Cited pages 2 and 3.
[48] Dimitris Tsipras Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. arXiv preprint, page
2018. Cited on
[49] Eric Wong, Shibani Santurkar, and Aleksander Madry, Leveraging sparse linear layers for debuggable deep networks. In International Conference on Machine Learning, pages I 11205- 1 11216. PMLR, 2021. Cited on pages and
[50] Mengjiao Yang and Been Kim. Benchmarking Attribution Methods with Relative Feature Importance. CoRR, abs/1907.09700 2019. Cited page 3.
[51] Jason Yosinski. Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson. Understanding neural networks through deep visualization. In Deep Learning Workshop, International Conference on Machine Learning (ICML), 2015. Cited on page
[52] Sergey Zagoruyko and Nikos Komodakis. abs/1605.017 606.00146, 2016. Cited on page 3.
Wide residual networks. arXiv preprint,
[53] Bolei Zhou, Yiyou Sun, David Bau, and Antonio Torralba. Revisiting the importance of individual units in cnns via ablation. arXiv preprint, 02219, 2018. Cited on page 2.
[54] Roland S. Zimmermann, Judy Borowski, Robert Geirhos, Matthias Bethge, Thomas S. A. Wallis, and Wieland Brendel. How well do feature visualizations support causal understanding of CNN activations? In NeurlPS, pages 1730-1 1744, 2021. Cited on pages 3, 4, 16, and 19.
14

---

A Methoooolgical Details
A.1 Measuring the Mechanistic Interpretability of Many Models
To measure the interpretability afforded by a model, we extend the paradigm established by [4]. Participants in our study complete sequence of TiremnaiveeFFrree-CCheee each trial measures the interpretability of one unit of a network. In each trial, participants are presented with two so-called query images, sourced from the training set of ImageNet. One query image is highly positively activating for the investigated unit, i.c., feeding this image through the network would cause large positive activation the target unit. In contrast, the other query image is highly negatively activating. Participants are tasked with determining which of the two query images is the positive one. To do so, they are presented with two sets of nine reference images which characterize the unit. One set contains highly positively activating images, while the other contains highly negatively activating images. In the natural condition, these reference images are other natural images, whereas in the synthetic condition, the reference images are synthetic images generated by Feature Visualization. See Fig. 8 for an example of one trial in the natural condition. We phrase the task by asking which set of reference images fits the positive query image better so that participants can be completely agnostic with respect the true semantics of the task. We also do not give overly specific instructions to avoid biasing the participants' behavior. Instead, participants learn the task by completing at least five hand-picked practice trials at the beginning of the experiment. Participants give binary response and rate their confidence in their decision on a three-point Likert scale.
(2-AFC) trials, where
Which image at the center matches Right Examples better?
Left Examples
Right Examples
20
urtool
110
157000
mph
80
60
40
120
140
160
ROX0D T37
Figure 8: Example of one trial. What crowd worker sees after having completed one trial: Two query images in the middle, two blocks of nine reference images to the sides, instructions, and feedback in the form of the green frame around the correct query image. Of course, this feedback is shown only after correct response. In case of an incorrect response, the frame would be red.
A.2 Sampling Images for the Psychophysical Tasks
The difficulty of an individual trial depends to a certain degree on the specific images that are shown in the trial. To avoid biasing the results for individual unit, do not only select the single highest/lowest activating image as a query image but instead create different trials for each unit. For each of these, we collect responses from crowd workers thrice. In the following, we describe the stimuli selection process for positively activating images, with negatively activating images being selected analogously. This procedure is similar to that of Borowski et al. [4], who also illustrate the approach in more detail. First, we select the top 9 t activating images as candidates for reference images, where t the number of unique trials to be generated. Then, we select the next images to be used as query images. To ensure that the range of activations yielded by the reference images does not differ across the tasks, we use the following procedure: We divide the range of candidate images into 9 groups of images each and create set of reference images by sampling one image from each of the groups without replacement. We initially create 20 trials but use only 10 of those, keeping the rest for an anticipated later experiment.
15

---

A.3 Amazon Mechanical Turk
Our psychophysical study is conducted on Amazon Mechanical Turk to meet the requirement of scale. To maintain high data quality, we exclude participants who do not fulfill certain criteria. First of all, we restrict participation in our experiment to countries in which workers can be expected to he adequately proficient in English and in which completion of our click-work at the expected hourly wage is not unreasonably more profitable than other work, which we deemed unethical. Specifically, the USA, Canada, Great Britain, Australia, New Zealand, and Ireland. As a second barrier, we only offer our Human Intelligence Task (HIT) to experienced workers who have submitted least 2'000 HITs for which the response was approved. To ascertain high reliability, we further restrict the pool to workers whose approval rate is at least 99%. Of course, we also prevent workers from participating in our experiments more than once". Even if workers meet the aforementioned requirements, they might still be distracted during the experiment or give random answers to quickly finish the experiment (e.g., if they are unmotivated or frustrated due to the task difficulty). Therefore, we filter our data further. To use only data from workers who understand the task, we only accept HIT' that require no more than three attempts at solving the demo trials and reject workers who spend less than 15 seconds reading the instructions. To catch workers who click mindlessly, we exclude responses in which fewer than four of our five catch-trials were answered correctly and responses that take the worker less than 135 seconds overall. On the other hand, we also reject responses that take them longer than 2'500 seconds since it can be assumed that these workers interrupted their work. We also reject responses in which participants select the same query image (as in, the upper/ / lower one) in more than 90% of trials.
we restrict participation
We recruit participants for each investigated model and experimental condition until 63 unique participants pass our quality checks. The responses of the workers who have not passed these checks not used in our analysis but are included in our IMI dataset. Each participant completes at least 5 practice trials to get used to the task, 40 real trials, and catch trials with obvious, hand-picked stimuli. In total and excluding pilot experiments, we collect data for 133'310 trials, of which 76'000 pass all quality checks.
We select 84 units of each model so that every unit is seen by 30 different participants since, within each task, no unit is shown more than once. All procedures conform to Standard 8 of the American Psychological 405 Association's "Fthical Principles of Psychologists and Code of Conduct" (2016). 6). Participants are compensated at a targeted hourly rate of 15 USD, which amounts to 2.79 USD) per task.
A.4 Scaling Feature Visualization to Many Models
A fundamental problem with using natural images to characterize the receptive field of individual units (apart from idiosyncrasies of the used dataset) is that visual features do not usually appear in isolation, resulting in ambiguity. For example, highly activating ImageNet-exxmpplar for a unit sensitive to feathers would probably depict birds, making it hard to isolate feathers as the crucial visual feature instead of beaks, claws, or a background of greenery or blue sky.
The promise of Feature Visualization is to circumvent these limitations by synthetically generating images that only contain visual features contributing to high unit activation. The procedure starts with initial random noise image and performs gradient ascent on the activation achieved by this image at the unit of interest. Following established work [e.g. 4, 54], a unit defined as one feature map of convolutional layer, where the activation across the feature map is aggregated by calculating the mean, just like for natural stimuli. To prevent mode collapse of the generated batch of feature visualizations, i.c. to truthfully capture the receptive field of so-called polysemantic units that show sensitivity to multiple different concepts, a regularization term is added to the loss to diversify the images.
We build on an existing implementation [38] and extend it to support various models flexibly. Previous implementations had two critical hyperparameters: the number of gradient ascent steps to be performed and the weight used for the diversity term, As earlier work mainly focused on the GeogLeNet model, hyperparameters were tuned for it. We find, however, that these fixed values do
$ 'Due to technical issues, some workers participated more than once. However, we exclude their data in our analysis and recollect the missing data by recruiting new participants.
16

---

not generalize well to other models, but their optimal" values heavily depend, among other factors, on the model and location of the unit within the network in extreme cases, the ideal value can even be different for two units of the same layer in the same network. Therefore, using any fixed value would introduce an unfair bias for or against some models. Furthermore, since a larger weight for the diversity term hinders the optimization, the number of necessary gradient ascent steps depends partially on the diversity weight, meaning these parameters cannot be set independently.
To overcome the latter problem of choosing an appropriate number of optimization steps, we imple- ment an adaptive procedure that interrupts the optimization when the gradients become small. The procedure performs at least 2'500 steps of gradient ascent and records trajectory of the observed gradient magnitude. We smooth these trajectories with a large sliding window and halt optimization once the average gradient magnitude in the last window is larger than in the second-to- -a-sstt window.
To solve the first problem, determine the diversity weight for each unit individually as follows. We first record the maximum and minimum activation achieved by natural dataset samples for the unit. Then, we generate feature visualizations without diversity and assert that they achieved stronger activation. We then try to find the largest possible diversity value that still produces images that achieve at least as strong activations all dataset samples. To do so, first perform an exponential search starting at a diversity of 1, increasing by a factor of 10 in each step. Once the value becomes too large, we perform 6 steps of binary search between the largest diversity value still known to work and the final value tested in the exponential search. If no value tested during the binary search worked, we return the lower bound of the search range, i.c. the images generated in the end are always guaranteed to be at least as activating the strongest natural images. Generating one batch of Feature Visualizations, i.e., one step of the procedure, takes between two and minutes on an Nvidia 2080Ti GPU, depending mostly on the width of the layer of the unit, since the diversity term scales quadratically. A qualitative comparison of feature visualizations generated for the different models considered in this work can be found in Fig. 9.
For ViTs, feature visualization could theoretically be performed using the same method by maxi- mizing the activation at the position-wis -wise feedforward layers. However, just applying the existing methodology does not lead to visually coherent images. Ghiasi et [16] present a method for adapting the procedure to ViTs that seems to produce intelligible images, but one step of their algorithm just adds large-scale noise to the visualizations, effectively performing a random search in image space to find activating images. Removing this augmentation or reducing the scale of the noise leads to unintelligible images again. light of these issues, we chose not to evaluate ViTs in the synthetic condition.
'Judged by the first authors.
17

---

GoogLeNet
4d 33t._botleeneckccom conv tunit 0)
3x_5x5 www.leneckkcoooooooo comn (unit 14)
4b 4b_1xl_comv (unit 74)
Densettet
block4 layerl0__cmme canvz (unit 28)
block3 black3_layer25, Jayer19 conv2 (unit 2)
black3 Jayers conv1 (unit 3)
ResVVt
layor3 A shortcut junit 91)
boyer_]1 ayer___ranv2 1_rav (unit 254)
l 0nbn2 (unit 24)
Robust Resivet
layera shortccttttttcc lunit 91)
(unit 25)
layer1 (unit 36)
ClipResNet
hyeer_5_shhorcar 5.sorrcu
layer3_5
layer4_2 2n3 (unit 1076)
Wideftstet
layor4 0_convl 699)
layera_0 bn3 243)
layor1 0_ bnbn3 (unit 53)
Convilext
stages_2 2_blocks_______1 419)
stages_2 Mocks, B_conu_dw (uni: stages_2 2_oos____bock_____1 (unit 387)
Figure Qualitative Comparison of Feature Visualizations. For each model, we randomly choose three units and display the maximally (left) and minimally (right) activating feature visualizations generated without the diversity regularizer.
18

---

A.5 A Priori Power Analysis
central question for the experimental design of this study is how many units need be sampled per model to obtain a result representative of the entire model. Answering this question is non non-trivial as there might be large inter-unit interpretability differences within one model. Indeed. this is what we observe displayed in Fig. 5). While the most naive approach would be to test all units, this unfeasible due to the associated financial costs. Therefore, we need to find trade-off between these considerations and keep the number of sampled units as low as possible while still getting representative results. Put differently: What is the lowest number of units one can select while still being reasonably sure that the found effect is statistically significant?
To answer this question, we first ran a pilot study where we controlled for inter-particiaan differences by showing stimuli from two models (GoogLeNet and Robust ResNet-50) to the same subjects. Participants in this pilot were the study's first authors and other lab members. This means that the obtained data is of high quality, and we can be confident that all participants understood the task. The mean difference in the proportion of correctly completed trials came out to be 0.1, with standard deviations of 0.15 for both interpretability methods, resulting in relatively large effect size, with Cohen's d of 0.67. Irrespective of concerns of statistical significance, we deem an effect of this size to be practically relevant; in other words, if the difference in interpretability between two models would be at least 10 percentage points, we would consider this practically relevant. To determine the required number of sampled units at these effect sizes, we then performed an a-priori power analysis using the software G*Power [13] - a standard tool widely used in psychology and the social sciences. To avoid unrealistic assumptions about the shape of the distribution of measurements (the normality-oo www.liitttsssuptiinnnn of the t-test will almost certainly not be met because the data points are proportions expected to lie between 0.5 and 1.0), opted for the non- non-parametric to safely conduct up to five
Hhan--WhtneyyUU
test. We assumed an lelevel of 0.01 (subject to Bonferroni-coreeatoo significance tests on the same data) and f--eeel of 0.95. This analysis yields that at least 86 units are required.
However, the situation is further complicated by the fact that we are comparing values of which we cannot actually take a continuous measurement since we aggregate binary trials to estimate the proportion of correctly completed trials for each unit, i.e. there is measurement noise. This can be modeled as a Binomial distribution, characterized by the parameter p. the probability of answering correctly in any given trial for units of this model. This gives rise to the question of how many measurements we should take per unit to be able to assess an individual unit's interpretability with any confidence. Accepting a standard deviation of 0.1 in the estimate of cach unit's p results in 30 independent trials per unit.
Another consideration is how many trials one participant can be asked to complete. Earlier work presented up to 24 trials to each participant under similar conditions [54]. Still, again we might be interested in accurately estimating the participant's 's performance, and each participant incurs some fixed cost for the time spent instructing them and completing the practice trials. On the other hand, MTurk HITs are typically very short. Constructing long tasks, e.g. of 100 trials more, would increase the risk of participants losing focus or becoming frustrated and just answering randomly. We deemed 55 trials per participant (40 real trials, 10 instruction trials, and 5 catch trials) a suitable balance of these concerns.
Finally, the required number of participants is the total number of trials divided by the number of trials per participant. The total number of trials is, of course, the number of units times the number of necessary measurements per unit, resulting in 86 30/40 trials. As this is not an integer, we opt for using 84 units instead, which brings the number of needed participants to 63.
19

---

B Further Experimental Results
B.1 Extended Visualizations of Results in Sec. 4
Propartion Carrect Gap (Eaey Mediumi
Proportion Correct Cap (Easy - Hard)
055
Resset
Clip ResNer
0.5
0.0
0.25 0.50
Reletive Layer tion
Proportion Correct (Eesy)
Relative Layer Position
Preportion Correct (Easy)
Figure 10: Well-interpretable units do not necessarily stay interpretable in harder tasks. For each unit investigated of the ResNet-50 (first row) and the Clip ResNet-50 (second row) model, we visualize the gap in human performance between the easy and medium (first two columns) and the easy and hard (last two columns) tasks. We show these gaps functions of the relative layer position (first and third column) and the human performance in the easy condition (second and fourth column).
Clip ResNet
ConvNeXT
DenseNet
GoogLeNet
ResNet
Rob. ResNet
WideResNet
Clip ResNet-
ConvNeXT
DenseNet-
GoogLeNet
ResNet-
Rob. ResNet
WideRes
NS
<
p
0.001
Figure 11: Few models have significantly different interpretability scores. The differences in interpretability afforded by synthetic feature visualizations Conover test with Holm correction for multiple comparisons; see Fig. 3 for significance values for natural exemplars.
mostly non-significant (NS) in
20

---

GoogLeNet
DenseNet
ResNet
1.0
0.5
Proportion Correct (Synthetic)
0.0
Model
Robust ResNet
Clip ResNet
WideResNet
1.0
GoogLeNet 37
DenseNet
.18
0.5
ResNet
.11
Rob. ResNet
.27"
0.0
Clip ResNet
.11
0.4
0.7
1.0
0.4
0.7
1.0
ConvNeXT
WideResNet
-6
1.0
ConvNeXT
.31*
0.5
0.0
0.7
Propartion Correct (Natural)
Figure 12: Measured interpretability using different methods is partially correlated. We investigate how the interpretability measured in our psychophysical experiment for the explanation method of natural dataset samples is predictive for that measured using synthetic feature visualizations. The table shows Spearman's rank correlation between the proportions correct when using natural and synthetic explanations. Asterisks denote significant correlations. While we see a strong correlation for some models, this does not hold for all.
0.9
Low Confidence.
High Confidence
0.8
Proportion
0.7
0.6
Chance
DenseeNeeDenneeNel
ResNet
ResNet
Model (Ordered by Scale)
(a) Natural exemplars.
21

---

0.9
Confidence
High Confidence
Proportion Correct
0.6
0.5
GoogLeNet DenseNel
ResNet
Robust ResNet
Clip ResNet
WideResNeConwWWeT
Model (Ordered Scale)
(b) Synthetic feature visualizations.
Confidence (Natural)
Medium
High
Confidence (Synthetic)
Low
Medium
High
1.0
Relative Number of Responses
0.6
0.2
GoogLeNet DenseNet
ResNet
Clip ResNet Model (Ordered by Scale)
Robust ResNitt
WideResNet viT
Clip VIT
ConvNeXT
(c) Distribution of confidence ratings.
More confident responses are mostly more correct. We investigate the relationship between the confidence indicated by the participants and the correctness of the given response. For this, we compare the proportion correct for responses with low (i.e., confidence ratings for all models and both natural exemplars (a) and synthetic feature visualizations (b). For the natural exemplars (a), find that for almost all models, higher proportion of responses are correct when the associated confidence ratings are higher. For the synthetic condition (b). this only holds for two models, if at all. Additionally, the distribution of confidence ratings (c) shows that natural examples lead to higher confidence scores for all models.
Figure 13:
and high (i.e., = 3)
22

---

0.90
Easy
Easy Hard
Medium
Very Hard
0.80
Proportion Correct
0.75
0.65
0.60
0.50
ResNet
Clip ResNet
ResNet
Model
Model
Figure 14: Impact of unit sampling on performance. In Fig. 6, we investigate the effect of task difficulty performance. Due oversight, not all 80 units sampled for this experiment were kept identical between the difficulty levels, but only 63. Here, we visualize the result for only those units that were shared between the difficulty levels. The inconsistency has no relevant qualitative effect on the conclusion: Performance rapidly declines as the task becomes harder.
23

---

B.2 Analysis of Quality Checks
Catch Trials
Row Variability
Total Response Time Instruction Time
Practice Trials
2000
Count
1000
True
False
True
False
True
False
True
False
True
Passed Exclusion Criteria
(a) Distribution decisions.
Min. Required
1500
400
Max. Allowed
600
Count
1000
300
400
200
500
200
100
0
1000 Instruction Time
2000
2000
4000
40
Total Response Time
Row Variability Upper
1500
400
1500
1000
Count
300
1000
200
500
500
100
0
20
40
0.0
0.5
1.0
0
20
40
Row Variability Lower
Catch Trials Ratio
Practice Trials
(b) Distribution of values used for decision.
Figure 15: Most participants pass quality checks. For each of the five quality checks outlined in Appx. A.3, show distribution over the number of participants that have passed/failed this check (top) and the distribution over the values used by the checks. The black and red lines in the latter indicate the minimally required and the maximally allowed values, respectively.
24

---

B.3 Distribution of Activations
Activation Density
Normalized Activation
Figure 16: Activation distribution is unimodal. We display the distribution of activation for 15 randomly chosen units from GoogLeNet. The activations have been divided by the largest absolute activation per unit to restrict the distribution to values between -1 and 1. The orange and red lines indicate the location
the 85th and 95th percentile as well as that the 15th and 5th percentile, respectively. It is apparent that the distribution is unimodal and does not feature multiple pronounced peaks/modes at its tail.
B.4
Are Activation Patterns in Feature Maps Predictive of Unit's Interpretability?
Since we observe large differences in unit-wise interpretability across all networks, a logical research direction is to find out what drives these differences. As example, investigate two hypotheses here.
Contrast. First, we investigate whether there is relationship between unit's interpretability and the local contrast in the activation maps of convolutional layers caused by validation set images. This motivated by the idea that if feature concentrated one location in the image, it might be easier to be detected by human observers than the activation is distributed across the image.
We visualize the relationship between unit's interpretability and the computed contrast in its activation maps in Fig. 17. There does not appear to be a strong relationship between the two, as supported by low Spearman's 's rank correlations (-0.24 s 0.14).
Sparseness. Second, we analyze whether the sparseness of activations in a feature map is predictive of a unit's interpretability. This is motivated by the argument that units that sparsely fire over large dataset are sensitive to a particular image feature that might be easier for humans to detect and understand.
To this, investigate two measures of sparseness: First, we compute the fraction of non-positive values (i.c., zeros after ReLU activation) in unit's s feature map averaged over the ImageNet validation set. The resulting data and the units' interpretability scores are shown in Fig. 18. for the contrast baseline, we see only weak, non-significant relation between the two. Second, we compute the fraction of images in the ImageNet validation set for which an entire feature map achieves only non-positive values (i.e., zeros after ReLU activation). Analogously to before, the resulting data is shown in Fig. 19, and we find no strong relationship.
C Broader Impacts
We expect the broader impacts of our work to be positive since advancements made with respect to interpretability of AI systems should increase their transparency and fairness. However, as always the case for interpretability work, explanations can also give users false sense of trust in the explained model. This can lead to the deployment of models that, under real-world conditions, give incorrect undesired results. Too much trust in AI systems also lead to their deployment in areas
25

---

GoogLeNet Ip= - p 0.13)
DenseNet (p= p 0.31)
ResNet (p= - p. 0.03)
1.0
0.8
1
0.6
250
500
750
1000
1250
0.0
0.2
0.4
0.6
0.0
0.5
1.0
1.5
Proportion Correct (Natural)
Robust ResNet (o= p=
Clip ResNet 0.14, 0.21)
WideResNet (p= 0.11. p 0.31)
1.0
0.6
0.4
0.00
0.25
0.50
0.75
1.00
1.25
1
2
2
10
ConvNeXT to = 0.01, 0.93)
0.8
0.6
10 15
Local Contrast in Activation Maps
Figure 17: Local contrast of activation maps does not predict a unit's interpretability. We compute the average local contrast in the activation maps caused by validation set images for the sampled units of the investigated convolutional networks. The units' interpretability, measured by the proportion correct, does not appear be function of the local contrast.
that are better left human hands for ethical reasons, such as policing or the justice system. Apart from these general and high-level concerns, we see no direct way in which someone could use the findings and data presented here cause harm, especially since we do not build an interpretability method but investigate whether models are interpretable.
D Computational and Financial Cost
The most computationally intensive aspect of this work is creating stimuli for the experiments, which can be further subdivided into collecting natural exemplars and producing feature visualizations. The former point is negligible since all that is required is one forward pass over the ImageNet training set for each model. We record the activations on Nyidia 2080Ti GPUs and perform multiple forward passes due to memory constraints, but even if we assume a pessimistic 4 hours of GPU time and full utilization of the GPU at 250 W, this results kWh power consumption for all models total. Creating feature visualizations for 100 randomly selected units we later randomly sample 84 units for each model and kept some stimuli for anticipated later experiments - requires the parallel use of 25 2080Ti GPUs for about 12 hours for all models except ConvNeXt, which takes about 24 hours on average. Since this is done for only seven models because we do not generate feature visualizations for the ViTs, the required electricity amounts to 600 kWh. Assuming our country's 's consumer electricity price of 0.4812 E kWh and the country's 's typical CO2 emissions per kWh of 428 g CO2e / kWh, both of which pessimistic estimates given that the experiments ran in local academic datacenter, these requirements translate to about 300 USD and 256 kg of CO2 equivalent emissions.
The financial cost of this work dominated by crowdworker compensations. As outlined in Appx. A.3, workers are compensated at an hourly wage of 15 USD, or 2.79 USD/ / HIT. Since all workers are
26

---

GoogLeNet (p= p.13,p= 0.26)
DenseNet (o = 0.20, p 0.07)
ResNet (p= p 0.18)
0.8
0.6
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
Proportion Correct (Natural)
Robust ResNet 0.000,
1.00)
Clip ResNet (p 0.31,
0.00)
WideResNNt (p
0.13.
0.23)
0.6
0.4
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
VIT (p 0.22.p p- 0.04)
Clip VIT (p 0.11, p 0.34)
ConvNeXT (p - 0.18,
0.10)
1.0
0.8
0.4
0.8 0.0
0.2
0.6
1.0
Pixelwise Sparsity
Figure 18: Sparseness of activations does not predict unit's interpretability. We compute the fraction of non- non-positive values (i.e., zero after ReLU activation) in the feature maps the units of interest averaged over the ImageNet validation set for all investigated models. We then show unit's interpretability as function of this pixel-wise sparseness measure. However, the two do not appear to have meaningful relationship, indicated by Spearman's 's rank correlation shown above each plot.
compensated, even if the results of their HIT do not pass our quality checks, the total cost incurred by the experiment (including the fees paid to MTuur) amounts to around 12'000 USD.
27

---

GoogLeNet (p 0.13, 0.24)
DenseNet (p - 0.34, 0.00)
ResNet (p 0.07,p - 0.26)
1.0
0.8
0.4
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Proportion (Natural)
Robust ResNet (p- p.29, 0.01)
Clip ResNet (p p 0.00)
WideResNet (o p 0.36)
0.8
0.6
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
VIT = p 0.09)
Clip (p 0.17. 0.12)
ConvNeXT (p= 0.10, 0.36)
1.0
0.6
0.4
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Channelwise Sparsity
Figure 19: Sparseness of entire channels does not predict a unit's interpretability. Similar to Fig. 18, we compute the fraction of images for which an entire feature map achieves only non-posee non-positive values (i.e., zero after ReLU activation). Analogously to before, we plot a unit's interpretability function of the channel-wisc sparseness and find strong relation between this sparseness measure and unit's interpretability.
28

---

E
Further Screenshots of Psychophysics Trials
This HIT an academic experiment Image classiiiation.
Please
experiment colorblinn your seriously Impaired.
experiment, groups images Your task choose one out of two acditional Images which resembles group Images better.
looks
Please note that you can only perticipete once in this HIT, anonymiied responses
study,
Figure 20: Screenshot of the initial overview of the HIT presented to workers considering the task. inform participants that they consent to their amonymized data being used for a scientific study.
29

---

Which mage the center matches the Right Examples better?
Left
KERNELS
WALL
Which mage the matches Examples better?
Fesmples
Examples
Figure 21: Screenshots of two of the twelve possible instruction trials to explain the task to participants in the natural condition after the participant has given the correct response. See Fig. 22 for examples the other condition.
30

---

Which imge the center matches the Right examples better??
Left
Which the center matches the Examples better?
Right Examples
Figure 22: Screenshots of two of the twelve possible instruction trials to explain the task to participants in the synthetic condition after the participant has given the correct response. See Fig. 21 for examples the other condition.
31

---

Which mage the center matches the Right Examples better?
Left
Which the center matches the Examples better?
Right Esamples
Cortinue
Figure 23: Screenshots of two of the twelve possible instruction trials to explain the task to participants
in the synthetic condition before the participant has given response (top) and after the participant has given the wrong response (bottom).
32


Reviews:
[{"review_id": "X5M2lCDe1q", "reviewer": "Reviewer_fNyx", "summary": "The authors investigate whether features associated with neurons in vision models become more or less interpretable as these models are scaled up. They utilize mechanical turk and a forced association test to see whether human workers are able to clearly associate exemplar images (or synthetic feature activations) to strongly vs. weakly activating sets of input images for given neurons. They find that across multiple scales and classes of vision models, there is no significant increase in interpretability when increasing scale. They provide a number of additional findings on the shortcomings of feature visualizations, and trends (or lack thereof) in interpretability as a function of neuron specificity and the location of neurons within a network. In addition to these interesting findings, they release an annotated dataset of neurons and their associated intepretabilities which should prove a great boon to mechanistic interpretability researchers more generally.", "questions": "It would be interesting to know how the effectiveness of synthetic feature visualizations varies for different models, as there is some discussion of the difficulties involved in using existing methods to generate features for the different models. Figure 12 mostly answers this as one can infer the “difference between synthetic and natural interpretability for each model” but it is not directly shown. It might be nice to add a figure with e.g. “3 non-cherry picked examples of synthetic feature visualizations” for each model, alongside the difference in synthetic vs natural interpretability. Perhaps this is not interesting as the feature visualizations are generally quite poor (as per Sec. 4.2).", "limitations": "Limitations are thoroughly discussed.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "Overall this paper provides a meaningful contribution to the field of interpretability research; both through its own investigations and through the accompanying release of the IMI Dataset. \n\n## Originality and Significance\nThe psychoanalytic approach toward addressing questions about interpretability has thus far been relatively underexplored, and is used to great effect in this work. The findings themselves seem general enough to guide future research on interpretability methods for vision models, but the release of the associated dataset may be more impactful still, as it will allow researchers with fewer resources to work on superior algorithmic approaches for discovering features / Interpretable neurons.\n\nThe additional experiments carried out in App. B.4 are also very interesting, as these undermine some intuitions prevalent in the community regarding which kinds of simple flags are likely to indicate interpretable neurons (though the annotations don’t explicitly measure the levels of polysemanticity, thus allowing this to remain as a potential confounder).\n\n## Quality and Clarity\nThe methodology and findings are mostly well-presented, and only minor changes are suggested (see Weaknesses)\n", "weaknesses": "The methodological approach seems comprehensive and well-suited towards addressing the questions of interest. I have no major criticisms.\n\nThe two minor criticisms are on:\n* **Claims** - the paper repeatedly makes claims of the form “mechanistic interpretability of larger models is not better”, which is supposed to be short-hand for “the interpretability of features associated with neurons in larger vision models is not better”. I think this amounts to an overstatement of the valid claim without being any more concise, as reframing “feature interpretability” to “mechanistic interpretability” is a generalization (there are other forms of mechanistic interpretability which may be unaffected by these findings, such as circuit discovery). Collapsing “larger vision models” to “larger models” is fine, given that this is obvious. Additionally, one might wish to argue that “feature interpretability” is too broad, but the field’s current definition of features is quite loose, and the paper acknowledges issues around superposition and synthetic feature generation, so this is fine.\n\n* **Presentation** - The paper is very well presented for the most part. The only part that was slightly difficult to parse was Fig.2 - here I found the examples in the appendices more illuminating (partly due to a nitpick regarding the caption, mentioned below). I don’t think there would be any loss of generality if this figure were replaced with one of these examples from the appendix (as the caption explicitly states that the query may be a natural exemplar or synthetic visualization)\n\n## Nitpicks:\n* 74: -> “was their usefulness .. experimentally quantified”\n* 114: do you mean “higher density of interpretable units”?\n* Fig 2 Caption: It was slightly unclear at first that “two extremely activating” meant one extremely positively, and one extremely negatively activating; this made the following “pick the more positively activating query” slightly confusing.\n* Fig 3 Caption: The caption is split into part “A” and “B”, but these are not labelled pictographically. Changing to “left” and “right” would be easiest.\n* 147 - “keep” -> “kept”, as latter part of sentence is past tense\n* 174 - “paradigm shift” seems quite strong\n* 300 - The footnote referencing superposition should be places before the period\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nThank you for your thorough review and your thoughtful comments. We were delighted by your enthusiasm for our IMI dataset and by your assessment of our work as a *“well-presented”*  and *“meaningful”* study with *“very interesting findings”*.\nWe also updated the caption of Figure 2 to present the setup more clearly and incorporated your other suggestions.\n\nPlease find our responses to your points below:\n\n**Q:** *“reframing ‘feature interpretability’ to ‘mechanistic interpretability’ is a generalization”* \\\n**A:** Thank you for raising this point. We agree that using precise expressions is important, and we did not mean to overstate the generality of our work. We used ‘mechanistic interpretability’ as a short form for understanding how neural networks work. As there are different approaches for this, we now rephrased our paper to more precisely characterize our paradigm as a sub-form of mechanistic interpretability (i.e., ”per-unit mechanistic interpretability”) to avoid overly general statements. \n\n**Q:** *“How [does] the effectiveness of synthetic feature visualizations var[y] for different models?”* \\\n**A:** Thank you for this suggestion. We included a new figure demonstrating typical feature visualizations by randomly choosing three units per model and depicting a positive and a negative feature visualization for them in Appendix A5 (see Figure 10 of the attached PDF for a preview). We find no obvious and substantial degradation of the feature visualizations for any model. "}, {"review_id": "sc4It88a8j", "reviewer": "Reviewer_fVHW", "summary": " This paper starts with the motivation of  a recent growing field of mechanistic interpretability, in which the goal is to attempt to reverse engineering a given neural network (or possibly any function). The main hypothesis is that since recent models have grown in size in terms of number of samples and number of parameters, if models perform more human-like task, the extracted features might become more interpretable. The authors then perform a large-scale human psycho/meta-physical study, in which they ask participants which of the two presented group of images are more aligned with a selected neuron’s response. The correctness of the response then serves as a metric for how interpretable a neural network is, in which they find neural networks from recent years (2017-2023), despite being larger in parameters and datasets size, are not mechanistically interpretable, which means study subjects are unable to identify the correct group of images. Last but not least, the authors release the data collected from participants as a dataset, to motivate the need for an automated formula/system to promote interpretability within any trained model. ", "questions": "1. Have the authors considered performing a similar experiment but with language tasks, since language is a more natural form of interpretability and method of showing one’s understanding? For instance, finding nearest neighbors of activation maps (as vectors), and ask whether the neighbors form any semantic clusters.\n2. The definition of mechanistic interpretability stated by the authors: “understanding the internal information processing of deep neural networks by attempting to reverse engineer them.” It is not immediately clear to me what it means when the authors stated in the conclusion that we need to optimize for this property. Can the authors elaborate on that? Does it imply we should design neural networks differently? Does it mean we should change the learning objectives we use to train the networks? Do the authors mean that we should design models with the ability to reverse engineer them? But wouldn’t that imply we already know how neural network works?  \n", "limitations": "The authors have adequately addressed the limitations in their Limitation section.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Overall, the paper is well-written and its contents are well-organized. I did not run into trouble understanding the paper. \n\nInterestingly, the work is heavily influenced by Olah 2017, which looks into activation maps of individual neurons of a neural network and find “object detectors” such as “dog neurons.” It is also somewhat parallel to works done back in 1950s, when neuroscientists would probe neurons in monkey visual cortex and find “edge” detectors from simple/complex cells. Hence I believe this experiment, although somewhat similar to what previous has done in terms of methodology, is an important contribution to the field of mechanistic interpretability, and to the entire field of interpretability/study of neural networks on a unit level in general. Although the connection to neuroscience in mentioned in the related work, personally I would suggest adding a short sentence to the introduction regarding its parallel to neuroscience. \n\nThe setup of the experiment is also reasonable and sound. Authors have sufficiently considered the potential biases of the experiment and have addressed them accordingly in the design of the experiment. The experiment is also done with a wide range of models (both CNNs and ViTs) and datasets of different scales. I believe the experiments are well thought out and done. \n", "weaknesses": "In this work, the authors constructed the hypothesis “If models make more human-like decisions, this might hint at a closer alignment between the extracted features and human perception.” While ImageNet classification is often considered standard for benchmarking, it is hard to argue that better models of predictive performance/larger models implies more human-like decisions. A better benchmark would have been comparing models that can perform multiple tasks, which humans do. An alternative solution would be to perform the same experiment on architectures, but that would obviously induce a much higher financial cost. I think the authors can better address why ImageNet is an indicative task for human-like decision in their Method or Introduction. \n\nSmall typo. In Training Details on OpenReview website, authors wrote: “we do not train but onyl evaluate models”, onyl -> only\n\nBut other than the above, there are no major weakness in this work.  \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for your constructive review and for assessing our work as a *“well-written”* and *“important contribution to the field of interpretability”* with *“reasonable and sound”* experiments.\n\nPlease find our responses to your points below:\n\n**Q:** *“I would suggest adding a [...] sentence to the introduction regarding its parallel to neuroscience”* \\\n**A:** You are right that the field of mechanistic interpretability has been inspired by approaches from neuroscience. We edited our manuscript and added this information in the introduction.\n\n**Q:** *“Have the authors considered [...] language tasks”* \\\n**A:** Thank you for the suggestion. We did not investigate language in this work but agree that it would be one of the most natural and dense formats for an explanation. We have updated the Related Work section to include a paper [3] that takes steps towards using natural language to summarize sets of highly activating exemplars. \n\n**Q:** *“It is not immediately clear [...] what it means [...] that we need to optimize for this property [interpretability]”* \\\n**A:** Thank you for raising this point. You are correct that we mean one needs to consider model interpretability throughout the model design and training and cannot simply hope to obtain an interpretable model through existing/commonly used techniques for scaling models up. Here, we use the term “interpretable models” to refer to models where one has knowledge about what each unit in the network does and how it contributes to the overall feature extraction process for a substantial part of the network. You are also correct that this knowledge would drastically simplify reverse engineering networks. We extended the conclusion of our manuscript to include this information and explain the point more clearly.\n\n\n**Q:** *“Why is ImageNet an indicative task of human-like decisions?”* \\\n**A:** This is an excellent question, and we have edited our manuscript to clarify our stance on this matter, as we did not mean to imply that ImageNet performance is indicative of human-like decision-making. Having said this, there is previous work [1,2] suggesting that scaling up models and datasets results in models that exhibit more human-like behavior, as measured by error consistency. We argue that it is conceivable that this behavioral similarity is caused by more human-like decision strategies, which in turn are characterized by their reliance on (non-spurious) features. Hence, we are interested in investigating whether this also results in a more interpretable feature extraction process. To make this connection more direct, we now integrate a new plot that shows the per-unit interpretability of a model as a function of their “human-likeness” measured by an established benchmark [1] (see Figure 4 of the attached PDF). We have updated our manuscript accordingly and included this information in the main text. \n\n[1] Partial success in closing the gap between human and machine vision. Geirhos et al. NeurIPS 2021. \\\n[2] Scaling Vision Transformers to 22 Billion Parameters. Dehghani et al. ICML 2023.\n"}, {"review_id": "IHt7IY3E4A", "reviewer": "Reviewer_q3wq", "summary": "In this paper, the author performed a large-scale psychophysical experiment to see whether scales improve mechanistic interpretability. They found that there is no scaling effect for interpretability; the latest large models do not provide better mechanistic interpretability than older models. The dataset with user response is publicized so that it can be used in future studies that aim to optimize interpretability.\n\nI think the paper is novel in that it answers a new question that was not explored in this way. However, since it lacks new theoretical results, substantial methods, or findings, I believe the NeurIPS datasets and benchmarks track would be a more suitable venue for the paper.", "questions": "1. Is the hypothesis that scaling improves mechanistic interpretability a common perception? ", "limitations": "1. I am not sure whether the hypothesis that scaling improves mechanistic interpretability is a common perception. If not, the utility of the dataset would be limited. ", "rating": 7, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The approach to answering the question of whether scales affect mechanistic interpretability via large-scale psychological experiments is novel.\n2. The experiment was performed on various axes such as model sizes, dataset sizes, and model architectures. \n3. The large-scale psychological experiment dataset would be a valuable resource for further research\n4. analyzed the dataset using multiple ways.\n5. The paper was well written.", "weaknesses": "1. The paper does not provide substantially new methods or findings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nThank you for your helpful review and for acknowledging the novelty and breadth of our experimental work as well as for describing our dataset as a *“valuable resource for further research”*.\n\nPlease find our responses to your points below:\n\n**Q:** *“Is the hypothesis that scaling improves mechanistic interpretability a common perception?”* \\\n**A:** This is a valid question, and we understand your concern about the prevalence of this hypothesis in the research community. We have updated our manuscript to better convey our motivation to investigate this question. Previous work [1, 2] demonstrates that an increased scale results in models that show more human-like behavior, as measured by error consistency. It is conceivable that this behavioral similarity is caused by more human-like feature extraction/processing strategies. However, our experimental results indicate that this is not the case: Scale does not implicitly optimize interpretability, and more human-like behavior does not automatically correspond to a more interpretable feature extraction process. We believe this insight will be of interest to a substantial part of the research community, as is also evident by the positive sentiment of the other reviewers. We updated our manuscript accordingly and included this information in the main text as we agree that this motivation is important for readers.\n\n**Q:** *“[...] does not provide substantially new methods or findings”* \\\n**A:** It is true that we do not develop a new method. Instead, we see the novelty of our study in obtaining a better understanding of how different model design choices — such as model/training dataset scale and training objective — influence a model’s interpretability, as measured by a common technique. To meet this end, we scale up an established experimental paradigm and conduct a large-scale evaluation of various models and conditions. This allows us to answer the research question posed above: None of the investigated design choices actually have the effect of making models more interpretable. We believe this to be an important result, as it strongly implies that better interpretability is orthogonal to other optimization criteria and should be an explicit goal on its own.\n\n\n[1] Partial success in closing the gap between human and machine vision. Geirhos et al.. NeurIPS 2021. \\\n[2] Scaling Vision Transformers to 22 Billion Parameters. Dehghani et al. ICML 2023. \n"}, {"review_id": "qm1GD8Bh0y", "reviewer": "Reviewer_6B8X", "summary": "This paper presents an empirical evaluation of how interpretable the individual neurons in different models are, with specific focus on how increasing model scale effects this. This is done via large scale human experiments via amazon mechanical turk, and they find no evidence that larger models would be more interpretable, instead having weak evidence towards large models being less interpretable. They also release their evaluation data to help create automatic evaluation in the future via models to predict interpretability to a human.", "questions": "- How many trials were conducted? The main paper states more than 100k, but appendix A.3 only states 40k, where is the difference?\n\n- Why was the alternative image chosen from least activating images? Why not for example use a randomly chosen image as the other option? Or perturbed versions of highly activating images as done by Zimmermann et al.?", "limitations": "Yes, good discussion. ", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- Sound experimental setup and statistical analysis\n- Studying an important problem\n- Larger scale than previous experiments, finding additional evidence towards previous findings such as feature visualizations being less informative than natural images, and robust models having more informative feature visualizations\n- Clearly written\n- Good discussion of related work", "weaknesses": "I think the experimental setup could be improved in a few ways:\n- I think 84 neurons per model, while bigger than previous might not be enough, especially if wanting to compare between layers in a single model, as this leaves less than 2 neurons per layer for ResNet-50 for example, which is not large enough sample size. This might also be part of the reason why most findings lack statistical significance. \n- While the costs make it harder to scale, could have been better to narrow down focus i.e. look at fewer models/layers within these models or use fewer evaluators per neuron.\n- Top-5% and top-15% activations for medium and large are too big of a jump in my opinion, top-15% rarely has any meaningful pattern for a neuron. Would be more interesting to see closer results like in the top-1% range\n\nFinally results are not surprising, I wasn't expecting scale alone or the other features studied here to have a significant effect on the interpretability. This reduces the impact of the work but it is still good information to have confirmed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for your constructive review of our work and for describing it as *“studying an important problem”* with a *“sound experimental setup/analysis”*.\n\nPlease find our responses to your points below:\n \n**Q:** *“How many trials were conducted”* \\\n**A:** We apologize for the confusion: The numbers in Appendix A.3 corresponded to a preliminary version of our dataset and were outdated. We updated this section now: “In total and excluding pilot experiments, we collect data for 120'745 trials, of which 69'330 pass the quality checks.”. \n\n**Q:** *“Why was the alternative image chosen from the least activating images”* \\\n**A:** Thank you for raising this question. We followed the reasoning of [1], who propose to use both strongly and weakly activating images as query images with the goal of making the task easier for humans. When the human participants are presented with both of these query images, there are effectively two valid strategies for solving the task: either by seeing that the weakly activating image is indeed weakly activating or by seeing that the strongly activating query image is strongly activating. By not showing the weakly activating query image, we could make the task harder - however, as the human performance is not yet reaching the ceiling performance of 100%, we argue that the current task difficulty is chosen reasonably. \n\n**Q:** *“84 units per model might not be enough”* \\\n**A:** We also pondered this question and performed an a priori power analysis (Sec. A.5) to decide on how many units to test while balancing the cost of the overall experiment. We based our analysis on what we believed to be a relevant effect size (a difference in group means of 10 percentage points) so that significance would also imply relevance. We could have sampled more units per model at the expense of dropping a few models from the comparison, but all that would have meant is that smaller differences would have become statistically significant. We updated our manuscript to include a discussion of this more prominently.\n\n**Q:** *“Top-5% and top-15% activations [...] are too big of a jump [...]. Would be more interesting to see closer results like in the top-1%”* \\\n**A:** Thank you for this insightful comment. We agree that the top-15% is a rather big jump both in activations and, expectedly, also in task difficulty. This setting was meant to test the limit of the usefulness of existing per-unit interpretability methods; hence, we also included the more conservative setting of top-5%. Nevertheless, we now conducted another experiment for the top-1% and updated Figure 7 (see Figure 7 of the attached PDF). Here, we also see a substantial (albeit smaller than for the top-5%) drop in performance for both models investigated. We updated our manuscript accordingly. \n\n**Q:** *“not surprising [...] but good information to have [this result] confirmed”* \\\n**A:** We are happy that you find our results interesting! Whether or not the finding is perceived as surprising, of course, depends: We believe there are reasonable grounds to hypothesize that scaling up models and datasets promotes more human-aligned feature extraction/processing given that the overall model behaviors also get more aligned with humans, according to [2,3]. Testing this hypothesis was the primary motivation for this work. We integrated this reasoning more clearly in the main text.\n\n[1]  Exemplary Natural Images Explain CNN Activations Better than State-of-the-Art Feature Visualization. Borowski et al. ICLR 2021. \\\n[2] Partial success in closing the gap between human and machine vision. Geirhos et al. NeurIPS 2021. \\\n[3] Scaling Vision Transformers to 22 Billion Parameters. Dehghani et al. ICML 2023. \n"}]

Author rebuttal:
We would like to thank all reviewers for their time and their constructive feedback. We appreciate their assessment of our work as *“an important contribution to the field of mechanistic interpretability”* with *“reasonable and sound experiments”* (fVHW), a *“great boon to mechanistic interpretability researchers”* with *“interesting findings”* (fNyx) and a *“novel approach”* producing *“valuable resource[s]”* (q3wq). All reviewers seemed to agree that our work was *“well/clearly written”* (q3wq, fVHW, 6B8X), *“well organized”* (fVHW), and *“well-presented”* (fNyx). We hope that our work is indeed *“general enough to guide future research on interpretability methods for vision models”* (fNyx).

Inspired by your helpful comments, we incorporated the following changes into the next/final revision of our paper:

- We conducted an additional experiment to better capture the decrease in subject performance as the task difficulty increases, sampling query images from the 99th percentile (see Figure 7 of the attached PDF).
- We added a figure to the Appendix that offers a qualitative impression of the feature visualizations used in the experiments (see Figure 10 of the attached PDF).
- We updated the caption of Figure 2 to increase the clarity of this illustration and to make the visualization of our task setup more accessible.
- We created a new figure that analyzes how predictive human-aligned behavior (measured by [1]) is for high per-unit interpretability scores (see Figure 4 of the attached PDF).
- We updated our manuscript to fix minor mistakes and reduce the risk of misunderstandings.


Meta-review: This paper studies vision models that differ in scale, architecture, training paradigm and dataset size, and wants to know whether those factors influence the mechanistic interpretability of those models as evaluated in a large scale study on mechanical turk. They find that none of these factors has an effect on the interpretability of individual units in the setting they evaluated. In other words, more recent, better performing models are not automatically more interpretable in this setting. They release their dataset and call for models to be designed to be more interpretable. The reviewers were unanimously positive about this paper, and mentioned it was well written, interesting to interpretability researchers, has well designed experiments, a good set of vision models, and could guide future research in (interpretable) vision models.
Decision: Accept (spotlight)