Measuring Mechanistic Interpretability at Scale Without Humans

Authors: roland s zimmermannl
david  klindt
wieland  brendel
Venue: ICLR_cc_2024_Workshop_Re-Align
Published: 2024-02-01 22:20:18

Abstract: In today’s era, whatever we can measure at scale, we can optimize. So far, measuring the interpretability of units in deep neural networks (DNNs) for computer vision still requires direct human evaluation and is not scalable. As a result, the inner workings of DNNs remain a mystery despite the remarkable progress we have seen in their applications. In this work, we introduce the first scalable method to measure the per-unit interpretability in vision DNNs. This method does not require any human evaluations, yet its prediction correlates well with existing human interpretability measurements. We validate its predictive power through an interventional human psychophysics study. We demonstrate the usefulness of this measure by performing previously infeasible experiments: (1) A large-scale interpretability analysis across more than 70 million units from 835 computer vision models, and (2) an extensive analysis of how units transform during training. We find an anticorrelation between a model's downstream classification performance and per-unit interpretability, which is also observable during model training. Furthermore, we see that a layer's location and width influence its interpretability.

Main paper: 
 related work: 
 Mechanistic Interpretability While the overall field of explainable AI tries to increase our understanding of neural networks, multiple subbranches with different foci exist (Gilpin et al., 2018) . One of these branches, mechanistic interpretability, aims to improve our understanding of neural networks by understanding their building blocks (Olah, 2022) . An even more fine-grained branch aims to interpret individual units of vision models (Bau et al., 2017; Zhou et al., 2018; Bau et al., 2020; Morcos et al., 2018; Olah et al., 2017) . We focus exclusively on this branch of research. This line of research for artificial neural networks was, arguably, inspired by similar efforts in neuroscience for biological neural networks (Hubel & Wiesel, 1962; Barlow, 1972; Quiroga et al., 2005) .Different studies set out to understand the behavior and sensitivity of individual units of vision networks -here, a unit can, e.g., be (the spatial average of) a channel in a convolutional neural network (CNN) or a neuron in a multilayer perceptron (MLP) . The level of understanding obtained for a unit is commonly called the per-unit interpretability; by averaging over a representative subset of units in the network, one obtains the per-model interpretability (Borowski et al., 2021; Zimmermann et al., 2023) . With the recent progress in vision-language modeling, a few approaches started using textual descriptions of a unit's behavior (Hernandez et al., 2022; Kalibhat et al., 2023) . However, the majority still uses visual explanations which are either synthesized by performing activation maximization through, e.g., gradient ascent (Olah et al., 2017; Erhan et al., 2009; Mahendran & Vedaldi, 2015; Nguyen et al., 2014; Mordvintsev et al., 2015; Yosinski et al., 2015; Nguyen et al., 2017) , or strongly activating dataset examples (Olah et al., 2017; Borowski et al., 2021) .With the onset of large language models (LLMs) and the increasing interest in them, there is also now an increasing interest in mechanistic interpretability of them (e.g., Elhage et al., 2021; Olsson et al., 2022; Bricken et al., 2023) .Quantifying Interpretability Rigorous evaluations, including falsifiable hypothesis testing, are critical for research on interpretability methods (Leavitt & Morcos, 2020) . This also encompasses the need for human-centric evaluations (Borowski et al., 2021; Kim et al., 2022) . Nevertheless, such human-centric evaluations of interpretability methods are only available in some sub-fields. Specifically for the type of interpretability we are concerned about in this work, i.e., the per-unit interpretability of vision models, two methods for quantifying the helpfulness of explanations to humans were introduced before: Borowski et al. ( 2021) presented a two-alternative-forced-choice (2-AFC) psychophysics task that requires participants to determine which of two images elicits higher activation of the unit in question, given visual explanations (i.e., images that strongly activate or deactivate the unit, see Fig. 1A ) of the unit's behavior. Zimmermann et al. (2021) extended this paradigm to quantify how well participants can predict the influence of interventions in the form of occlusions in images. While these studies used their paradigms to evaluate the usefulness of different interpretability methods, Zimmermann et al. (2023) leveraged them to compare the interpretability of multiple models. Due to the reliance on human experiments, they could only probe the interpretability of 767 units from nine models. We now automatize this evaluation to scale it up by multiple orders of magnitude to more than 70 million units across 835 models.Automating Interpretability Research To increase the efficiency of interpretability research and scale it to large modern-day networks, the concept of automated interpretability was proposed, first in the domain of natural language processing (Bills et al., 2023) . This approach uses an LLM to generate textual descriptions of the behavior of units in another LLM. Follow-up work by Huang et al. (2023) , however, pointed out potential problems regarding the correctness of the explanations. To benchmark future fully automated interpretability tools, acting as independent agents, Schwettmann et al. ( 2023) introduced a synthetic benchmark suite inspired by the behavior of neural networks. In computer vision, there are also efforts to automate interpretability research (Hernandez et al., 2022; Zimmermann et al., 2023) . Hernandez et al. (2022) and Oikarinen & Weng (2022) map visual to textual explanations of a unit's behavior using automated tools, hoping to increase the efficiency of evaluations. Zimmermann et al. (2023) introduced the ImageNet Mechanistic Interpretability (IMI) dataset, containing per-unit interpretability annotations from humans for 767 units, meant to foster research on automating interpretability evaluations. 
method: 
 We now introduce our fully automated interpretability measure, Machine Interpretability Score (MIS), visualized in Fig. 1 . Borowski et al. (2021) proposed a psychophysical experiment for quantifying the per-unit interpretability of vision models, i.e., how well humans can infer the sensitivity of a unit in a vision model from visual explanations. Here, a unit can be a channel in a CNN, commonly averaged over space, a neuron in an MLP, or arbitrary linear combinations of different units. The experiment uses a 2-AFC task design (see Fig. 1A ) to measure how well humans understand a unit by probing how well they can predict which of two extremely activating (query) images yields a higher activation, after seeing visual explanations. Specifically, two sets of explanations are displayed: highly and weakly activating images, called positive and negative explanations, respectively. See Appx. A.1 for a more detailed task description. We build on top of this paradigm but replace human participants with machines, resulting in a fully automated interpretability metric that requires no humans.Definition of the Machine Interpretability Score Let I denote the space of valid input images for a model. For a specific explanation method and a unit in question, we denote the unit's positive and negative visual explanations as sets of images E + ⊆ I and E -⊆ I, respectively. Further, let Q + ⊆ I and Q -⊆ I be the sets of query images with the most extreme (positive and negative) activations. The task by Borowski et al. (2021) can now be expressed as: Given explanations E + and E -and two queries q + ∈ Q + and q -∈ Q -, which of the two queries matches E + and which E -more closely? An intuitive way to solve this binary task is to compare each query with every explanation and to match the query images to the sets of explanations based on the images' similarities.To formalize this, we introduce a perceptual (image) similarity function f : I × I → R computing the scalar similarity of two images (Zhang et al., 2018) , and an aggregation function a : R K → R reducing a set of K similarities to a single one. This allows us to define the function s : I × I K → R that quantifies the similarity of a single query image to a set of explanations: s(q, E) := a ({ f (q, e) | e ∈ E }) .(1) To decide whether a single query image is more likely to be the positive one, we can compute whether it is more similar to the positive than the negative explanations. We can compute this now for both the positive and the negative query images and get:EQUATIONEQUATIONThe classification problem will be solved correctly if the similarity of q + to E + relative to E -is stronger than those of q -. This means we can define the probability of solving the binary classification problem correctly asp(q + , q -, E + , E -) := σ α • ∆ + (q + , E + , E -) -∆ -(q -, E + , E -) (4)where σ denotes the sigmoid function and α is a free parameter to calibrate the classifier's confidence.We define the Machine Interpretability Score (MIS) as the predicted probability of making the right choice, averaged over N tasks for the same unit. Across these different tasks, the query images q + , q -vary to cover a wider range of the unit's behavior. If the explanation method used is stochastic, it is advisable to also average over different explanations:EQUATION)The MIS is not a general property of a unit but depends on the method used to generate explanations. One might define a general score by computing the maximum MIS over multiple explanation methods.Choice of Hyperparameters. We use the current state-of-the-art perceptual similarity, DreamSim (Fu et al., 2023) , as f . See Appx. B for a sensitivity study on this choice. DreamSim models the perceptual similarity of two images as the cosine similarity of the images' representations from (multiple) computer vision backbones. These were first pre-trained with, e.g., CLIP-style training (Radford et al., 2021) and then fine-tuned to match human annotations for image similarities of pairs of images. We use the mean to aggregate the distances between a query image and multiple explanations to a single scalar, i.e., a(x 1 , . . . , x K ) := 1/K K i x i . To choose α, we use the interpretability annotations of IMI Zimmermann et al. (2023) : We optimize α over a randomly chosen subset of just 5% of the annotated units to approximately match the value range of human interpretability scores, resulting in α = 0.16. Note that α is, in fact, the only free parameter of our metric, resulting in very low chances of overfitting the metric to the IMI dataset. We use the same strategy as Borowski et al. (2021) ; Zimmermann et al. (2021) and Zimmermann et al. (2023) for generating new tasks (see Appx. A.2). As they used up to 20 tasks per unit, we average over N = 20. See Appx. C for a sensitivity study. 
results: 
 This section is structured into two parts: First, we validate our Machine Interpretability Score (MIS) by showing that it is well correlated with existing interpretability annotations. Then, we demonstrate what type of experiments become feasible by having access to such an automated interpretability measure. Our experiments use the best-working -according to human judgements (Borowski et al., 2021 )visual explanation method, dataset examples, for computing the MIS. We demonstrate the applicability of our method to other interpretability methods (e.g., feature visualizations) in Appx. D. Note that different explanation methods might require different hyperparameters for computing the MIS. Both query images and explanations are chosen from the training set of ImageNet-2012 (Russakovsky et al., 2015) . When investigating layers whose feature maps have spatial dimensions, we consider the spatial mean over a channel as one unit (e.g., Borowski et al., 2021) . We ignore units with constant activations from our analysis as there is no behavior to understand (see Appx. E for details).We validate our MIS measure by using the interpretability annotations in the IMI dataset (Zimmermann et al., 2023) , which will be referred to as Human Interpretability Scores (HIS). The per-unit annotations are responses to the 2-AFC task described in Sec. 3, averaged over ≈ 30 participants. IMI contains scores for a subset of units for nine models. 2023), we measure their interpretability and compare them to randomly sampled units. Strikingly, the psychophysics results match the predicted properties: Units with the lowest MIS have significantly lower interpretability than random units, which have significantly lower interpretability than those with the highest MIS. Errorbars denote the 95 % confidence interval.First, we reproduce the main result of Zimmermann et al. (2023) : A comparison of nine models in terms of their the per-unit interpretability. We plot the HIS and MIS values (averaged over all units in a model) in Fig. 2A and find very strong correlations (Pearson's r = 0.98 and Spearman's r = 0.94).Reproducing the model ranking is strong evidence for the validity of the metric, as no information about these rankings was explicitly used to create our new measure.Next, we can zoom in and look at individual units instead of per-model averages. Fig. 2B shows MIS and HIS for all units of IMI. The left figure clearly shows a strong correlation (Pearson's and Spearman's r = 0.80). The interpretability scores in IMI are a (potentially noisy) estimate over a finite number of annotators. We estimate the ceiling performance due to noise (sampling 30 trials from a Bernoulli distribution) to equal a Pearson's r = 0.82 (see Appx. B for details). The right figure shows an alternative visualization, which bins the units according to their MIS and averages the HIS to reduce this noise -highlighting that the two scores correlate strongly. We can conclude that the MIS explains existing interpretability annotations well -both on a per-unit and per-model level.While the previous results show a strong relation between MIS and human-perceived interpretability, they are of a descriptive (correlational) nature. To further test the match between MIS and HIS, we now turn to a causal (interventional) experiment: Instead of predicting the interpretability of units after a psychophysics evaluation produced their human scores, we now compute the MIS before conducting the psychophysics evaluation. We perform our experiment for two models: GoogLeNet and a ResNet-50. For each model, IMI contains interpretability scores for 96 randomly chosen units.We look at all the units not tested so far and find the 42 units yielding the highest (Easiest, average of 0.99 for both models) and lowest (Hardest, average of 0.63 and 0.59, respectively) MIS, respectively. Then, we use the same setup as Zimmermann et al. (2023) and perform a psychophysical evaluation 2023) from a noisy average over a few units for a few models to all units of 835 models. The models are compared regarding their average per-unit interpretability (as judged by MIS); the shaded area depicts the 5th to 95th percentile over units. We see that all models fall into an intermediate performance regime, with stronger changes in interpretability at the tails of the model ranking. Models probed by Zimmermann et al. (2023) are highlighted in red.on Amazon Mechanical Turk with 236 participants. We compare the (newly) recorded HIS for the three groups of units in Fig. 2C . The results are very clear again: As predicted by the MIS, the HIS is highest for the easiest and lowest for the hardest units. Further, the HIS is close to the pre-hoc determined MIS given above. This demonstrates the strong predictive power of the MIS and its ability to be used for formulating novel hypotheses.After confirming the validity of the MIS, we now change gears and show use cases for it, i.e., experiments and analyses that were truly infeasible before due to the high cost, both time and money, of human evaluations.Zimmermann et al. ( 2023) investigated whether model or training design choices influence the interpretability of vision models. Although they invested a considerable amount of money in this investigation (≥ 12 000 USD), they could only compare nine models via a subset of units. We now scale up this line of work by two orders of magnitude and investigate all units of 835 models, almost all of which come from the well-established computer vision library timm (Wightman, 2019) . These models differ in architecture and training datasets but were all at least fine-tuned on ImageNet. See Appx. F for a list of models. Putting this scale into perspective, achieving the same scale by scaling up previous human psychophysics experiments would amount to absurd costs (≥ 1 B USD). Following previous work, we ignore the first and last layers of each model (Zimmermann et al., 2023) .When sorting the models according to their average MIS (Fig. 3 ) they span a value range of ≈ 0.80 -0.91. The strongest differences across models are present at the tails of the ranking. Note that GoogLeNet is ranked as the most interpretable model, resonating with the community's interest in GoogLeNet as it is widely claimed to be more interpretable. The shaded area denotes the 5th to 95th percentile of the distribution across units. This reveals a strong difference in the variability of units for different models; further, as the upper end of the MIS is similar across models (≈ 95 %), most of the change in the average score seems to stem from a change in the lower end, with decreasing width of the per-unit distribution for higher model rank.To investigate the difference in how the MIS of units is distributed between different models, we select 15 exemplary models and visualize their per-unit MIS distribution in Fig. 4B . Those models were chosen according to the distance between 5th and 95th percentile (five with highest, average, and lowest distance). While models with low and medium variability have unimodal left-skewed distributions, the ones with high variability have a rather bimodal distribution. Note that the distribution's second, stronger mode has a similar mean and shape to the overall distribution for models with low variability. The first mode is placed at a value range slightly above 0.5, corresponding to the task's chance level, indicating mostly uninterpretable units. This suggests that a subset of uninterpretable We show the average interpretability of units from the most common layer types in vision models (BatchNorm, Conv, GroupNorm, LayerNorm, Linear). We follow Zimmermann et al. (2023) and restrict our analysis of Vision Transformers to the linear layers in each attention head. While not every layer type is used by every model, we still see some separation between types (see Fig. 16 for significance results): Linear and convolutional layers mostly outperform normalization layers. Models are sorted by average per-unit interpretability, as in Fig. 3 . units (see Fig. 22 for examples) can explain most of the models' differences in average MIS. We analyze this further in Fig. 19 , where we compare the models in terms of their worst units. We see a similar shape as in Fig. 3 , but with a larger value range used, resulting in stronger model differences.Previous work analyzed a potential correlation between interpretability and downstream classification performance. However, in a limited evaluation, it was found that better classifiers are not necessarily more interpretable (Zimmermann et al., 2023) . A re-evaluation of this question is performed in Fig. 4A and paints an even darker picture: Here, better performing ImageNet classifiers are less interpretable (Pearson's r = -0.5 and Spearman's r = -0.55).Among training procedures and architecture, the analyzed models also differ in the required resolution of their input. While previous work focused only on models with a single resolution (Zimmermann et al., 2023) , we can now see whether the resolution influences interpretability. However, Fig. 17 suggests that there is no influence.Next, we can zoom into the results of Fig. 3 and investigate whether there are differences between different layers. First, we are interested in testing whether the layer type is important, e.g., are convolutional more interpretable than normalization or linear layers? In Fig. 5 , we sort the models by their average MIS over all layer types but show individual points for each of the five most common types (Conv, Linear, BatchNorm, LayerNorm, and GroupNorm). The number of points per model may vary, as not all models contain layers of all types. The figure shows a benefit of Conv over BatchNorm layers, which themselves are better than LayerNorm layers. Linear layers, if present, outperform both Batch-and LayerNorm as well as Conv layers. While the differences are small, they are statistically significant due to the large number of scores collected (see Fig. 16 ).Second, we analyze whether the location of a layer inside a model plays a role, e.g., are earlier layers more interpretable than later ones? The average per-unit MIS (for each layer type) is shown in Fig. 6A as a function of the relative depth of the layer. A value of zero corresponds to the first and a value of one to the last layer analyzed. The scores are averaged in bins of equal count defined by the relative layer depth to enhance readability. The resulting curves all follow a similar, almost sinusoidal, pattern: They start high, decrease in the first fifth, then increase steadily until they drop in the last tenth again.Third, it is interesting to probe the influence of the width of layers on their average interpretability.Based on the superposition hypothesis (Elhage et al., 2022; Olah et al., 2020; Arora et al., 2018; Goh, 2016) , one might expect wider layers to be more interpretable as features do not have to form in superposition (i.e., as polysematic units) but can arise in a disentangled form (i.e., as monosemantic units). Fig. 6B shows the relation between MIS and relative layer width. We use the relative rather than the absolute width to reduce the influence of the overall model and show the results of models with different architectures on the same axis. Note that, nevertheless, there might be other confounding factors correlated with the width, e.g., the layer depth. While we only see a moderate correlation for BatchNorm layers, we find a stronger one for Conv/Linear layers. It is unclear what causes this difference in behavior. However, we see this as a hint that one way to increase a model's interpretability is to increase the width (and not the number) of layers.In the last set of experiments, we demonstrate how the MIS can be used to analyze models in a fine-grained way and obtain insights into their training dynamics. For this, we train a ResNet-50 on ImageNet-2012, following the training recipe A3 of Wightman et al. (2021) , for 100 epochs. To better understand the dynamics through the training -most importantly during the first epoch -we zoom in to find out which units cause this strong change in MIS. Fig. 7 shows the change in MIS during the first epoch for each layer separately (ordered by their depth within the network). Surprisingly, we see that the change in MIS is dominated by a set of BatchNorm layers, namely the last ones of each ResNetBlock, whose MIS increases drastically. Moreover, we detect a small trend of later layers improving more strongly than earlier ones but generally do not see a difference between Conv and BatchNorm layers.1 0 0 1 0 1 1 0 1 1 0 2 1 0 2 1 0 3 1 0 3 1 0 0 1 0 1 1 1 1 1 1 1 1 1 2 1 1 2 1 1 3 1 1 3 1 2 1 1 2 1 1 2 2 1 2 2 1 2 3 1 2 3 2 0 1 2 0 1 2 0 2 2 0 2 2 0 3 2 0 3 2 0 0 2 0 1 2 1 1 2 1 1 2 1 2 2 1 2 2 1 3 2 1 3 2 2 1 2 2 1 2 2 2 2 2 2 2 2 3 2 2 3 2 3 1 2 3 1 2 3 2 2 3 2 2 3 3 2 3 3 3 0 1 3 0 1 3 0 2 3 0 2 3 0 3 3 0 3 3 0 0 3 0 1 3 1 1 3 1 1 3 1 2 3 1 2 3 1 3 3 1 3 3 2 1 3 2 1 3 2 2 
conclusion: 
 This paper presented the first fully automated interpretability metric for vision models -the machine interpretability score (MIS) -which is experimentally shown to be well aligned with human interpretability labels. We verified the alignment through both correlational and interventional experiments. We expect our MIS to enable experiments previously considered infeasible due to the costly reliance on human evaluations. To stress this, we demonstrated the metric's usefulness for formulating and testing new hypotheses about a network's behavior: Based on the largest comparison of vision models in terms of their per-unit interpretability so far, we investigated potential influences on their interpretability, such as a layer's depth and width. Most importantly, we find an anticorrelation between a model's downstream performance and its per-unit interpretability. Further, we performed the first detailed analysis of how the perceived interpretability changes during training.While this paper considerably advances the state of interpretability evaluations, there are some open questions and potential future research directions. Most importantly, the performance of our MIS on a per-unit level is close to the noise ceiling determined by the limited number of human interpretability annotations available. This means that future changes in the MIS measure (e.g., based on other image perceptual similarities) might require additional human labels to determine the significance of performance improvements. Additional human labels could also be leveraged to improve the MIS by following Fu et al. (2023) to fine-tune the image similarity directly on human judgments. In another direction, using vision language models for computing the MIS could be interesting as this might also provide a textual description of a unit's sensitivity (Hernandez et al., 2022) . Finding a differentiable approximation of the MIS will be valuable for explicitly training models to be interpretable (Zimmermann et al., 2023) . Note that while this paper looked at the interpretability of channels and neurons, it can also be used for analyzing arbitrary directions in activation space. Thus, we expect the MIS to also be valuable for researchers generally looking for more interpretable representations of (artificial) neural activations (e.g., Graziani et al., 2023) . Finally, exploring whether this concept of interpretability quantification can be expanded to LLMs is an exciting direction. 
which query image matches the positive explanations better?: 
 Negative Explanations (Minimally Activating)Fig. 1 : Definition of the Machine Interpretability Score. A. We build on top of the established task definition proposed by Borowski et al. (2021) to quantify the per-unit interpretability via human psychophysics experiments. The task quantifies how well participants understand the sensitivity of a unit by asking them to match strongly activating query images to strongly activating visual explanations of the unit. See Fig. 9 for examples. B. Crucially, we remove the need for humans and fully automate the evaluation: We pass the explanations and query images through a feature encoder to compute pair-wise image similarities (DreamSim) before using a (hard-coded) binary classifier to solve the underlying task. Finally, the Machine Interpretability Score (MIS) is the average of the predicted probability of the correct choice over N tasks. C. The MIS proves to be highly correlated with human interpretability ratings and allows fast evaluations of new hypotheses.With the arrival of the first non-trivial neural networks, researchers got interested in understanding their inner workings (Krizhevsky et al., 2012; Mahendran & Vedaldi, 2015) . For one, this can be motivated by scientific curiosity; for another, a better understanding might lead to building more reliable, efficient, or fairer models. While the performance of neural networks has seen a remarkable improvement over the last few years, our understanding of information processing has progressed more slowly. Nevertheless, understanding how complex models -e.g., language models (Bricken et al., 2023) or vision models (Olah et al., 2017; Zimmermann et al., 2023) -work is still an active and growing field of research, coined mechanistic interpretability (Olah, 2022) . A common approach in this field is to divide a network into atomic units, hoping they are easier to comprehend. Here, atomic units might refer to individual neurons or channels of (convolutional) layers (Olah et al., 2017) , or general vectors in feature space (Elhage et al., 2022; Klindt et al., 2023) . Besides this approach, mechanistic interpretability also includes the detection of neural circuits (Cammarata et al., 2020; Elhage et al., 2022) or analysis of global network properties (Nanda et al., 2023) .The goal of understanding the inner workings of a neural network is inherently human-centric: Irrespective of what tools have been used, in the end, humans should have a better comprehension of the network. However, human evaluations are time-consuming and costly due to their reliance on human labor (Zimmermann et al., 2023) . This results in slower research progress, as validating novel hypotheses takes longer. Removing the need for human labor by automating the evaluation can open up multiple high-impact research directions: One benefit is that it enables the creation of more interpretable networks by explicitly optimizing for interpretability -after all, what we can measure at scale, we can optimize. Moreover, it allows more efficient research on explanation methods and might increase the overall understanding of neural networks. While efforts to build such measures for language models exist (Bills et al., 2023) , there is no common approach yet for vision models.The present work is the first to introduce a fully automated interpretability measure (Fig. 1A & C ): the Machine Interpretability Score (MIS). By leveraging the latest advances in image similarity functions aligned with human perception, we obtain a measure that is strongly predictive of human-perceived interpretability (Fig. 1C ). We verify our measure through both correlational and interventional experiments. By removing the need for human labor, we can scale up existing evaluations by multiple orders of magnitude. Finally, we demonstrate potential workflows and use cases of our MIS. 
 
 Appendix: A.1 TASK DESIGN Our proposed MIS builds on the 2-AFC task designed by Borowski et al. (2021) to conduct human psychophysics experiments. An example of such a task is given in Fig. 9 . This task aims to probe how well (human) participants can detect the sensitivity of a unit of a neural network based on visual explanations of it. Understanding the unit's sensitivity should allow participants to distinguish between a stimulus eliciting highly activating from one yielding low activation. Therefore, the task shows the participants two such images, called query images, and asks them to pick the image eliciting higher activation. To solve the task, participants also see two sets of visual explanations: Positive explanations describe the patterns the unit activates strongly for, while negative activations show patterns the unit weakly responds to. For constructing tasks, we follow Zimmermann et al. (2023) . Specifically, this means that we use K = 9 (positive and negative) explanations in each task. We restrict explanations to natural dataset examples to reduce complexity but note that the same setup can also be applied to other visual explanations, such as feature visualizations. To choose query images and explanations, we proceed as follows: For each unit, we determine the N • (K + 1) most and least activating images, respectively. Out of these, the N • K most extreme images are used as explanations, the others as query images.The N • K potential explanation images are uniformly distributed across tasks according to their elicited activation level (see (Borowski et al., 2021; Zimmermann et al., 2023) for more details).As stated in Sec. 3, we used DreamSim (Fu et al., 2023) as the underlying perceptual similarity f for all experiments shown so far. We now repeat the experiments on IMI in Sec. 4.1.1 with two alternative similarity measures: LPIPS (Zhang et al., 2018) and DISTS (Ding et al., 2022) . While all three measures are based on learned image features, DreamSim leverages an ensemble of modern vision models trained on larger datasets compared to LPIPS and DISTS, which use AlexNet (Krizhevsky et al., 2012) and VGG16 (Simonyan & Zisserman, 2015) trained on ImageNet, respectively. According to Fu et al. (2023) , DreamSim clearly outperforms LPIPS and DISTS on image similarity benchmarks.When comparing MIS based on DreamSim with one based on LPIPS and DISTS on a per-model level (see Fig. 10 ) one sees very similar results and strong correlations between each MIS and HIS. This might suggest that the choice of the similarity function to use has little influence on the quality of MIS. The picture, however, changes when zooming in and looking at per-unit interpretability (see Fig. 12 ). Now, it becomes evident that the MIS based on DreamSim outperforms that based on LPIPS and DISTS, indicated by the higher correlation and smaller spread of the point cloud. We, therefore, conclude that DreamSim is the best perceptual similarity available for computing machine interpretability scores. We compare DreamSim with two earlier perceptual similarity metrics, LPIPS and DISTS. All three lead to similar results on IMI (cf. Fig. 2A ). See Fig. 12 for comparing these similarity functions on a per-unit level. standard deviation.To put the difference in performance between the perceptual similarities on a per-unit level into context, we estimate the noise ceiling of the data: As the HIS for a single unit is a (potentially) noisy estimate over (up to 30) human decisions, it has some uncertainty.To take this into account, we run a statistical simulation, in which we model individual human responses as binary decisions from a Bernoulli distribution whose mean equals the unit's HIS. We can now simulate human decisions by sampling from the distribution. Then, we compute the correlation between MIS and simulated HIS and repeat the process 1 000 times. The resulting noise ceiling is compared to the correlations obtained when using LPIPS, DISTS, and DreamSim in Fig. 11 . We see that DreamSim's performance is very close to the noise ceiling for estimating the per-unit human interpretability.As described in Sec. 3, we compute the MIS by averaging over N = 20 tasks. This choice was initially motivated by previous work by Borowski et al. (2021) . We investigate now how this choice influences the MIS. For this, we perform two experiments for GoogLeNet (see Fig. 13 ). First, we use the method for constructing tasks described before in Appx. A.2 to create 20 tasks per unit and then compute how the MIS changes when only using the first i = 1, . . . , 19 tasks compared to all 20. While this setting is straightforward to analyze, it does not reflect how the number of tasks influences the MIS computation in practice: Using the task creation above, the chosen number of Fig. 12 : LPIPS and DISTS Perform Worse than DreamSim when Comparing Individual Units.We compare DreamSim with two earlier perceptual similarity metrics, LPIPS and DISTS. While LPIPS and DISTS perform similarly to DreamSim on a per-model level of IMI (cf. Fig. 12 ), they lead to worse performance on a per-unit level.tasks influences the creation of all tasks, e.g., adding one more task changes which images are used for previous tasks. Therefore, in the second experiment, we again measure how the MIS changes when using i = 1, . . . , 19 tasks compared to 20, but recreate all tasks when increasing their number.For both settings, we see that the residual converges to zero, with a slower convergence in the more realistic setting. Fig. 13 : Convergence of MIS. We investigate how MIS changes depending on the number of tasks N that it is computed over. Here, we distinguish between two settings. In (a), we simulate that adding another task does not change the selection of query images and explanations in earlier tasks; in (b), this is not the case. While the former is easier to analyze due to a reduced level of randomness, note that the latter is the more relevant setting in practice. For both cases, we visualize the average absolute difference in MIS estimated for < 20 and N = 20 tasks.The experiments in Sec. 4 compute the MIS for one type of explanation, namely strongly activating dataset examples. We now demonstrate that the same approach easily generalizes to other visual explanations: feature visualizations. We do not tune any hyperparameters but re-use the same as presented in Sec. 3 for dataset examples as explanations. In Fig. 14 we repeat the experiment from Fig. 2A and again see a strong correlation between MIS and HIS. Fig. 14 : MIS Generalizes Well to Other Explanation Types. We find a high correlation between MIS and HIS for other explanation types (feature visualizations). See Fig. 2A for the corresponding results for using natural dataset examples as explanations.After training a network, it might happen that some of its units effectively become non-active/constant for any relevant image. We here call a unit constant if the difference between maximally and minimally elicited activation by the entire ImageNet-2012 training set is less than 10 -8 . As mentioned at the beginning of Sec. 4, we excluded those units in our analysis, as they do not present any interesting behavior that is worth understanding. Note that this does not mean that it will not be interesting to understand why such units exist. In Fig. 15 , we display the ratio of constant units for each model. For most models, we see a low number of constant units: Specifically, we see that out of the 835 models investigated, 256 do not contain any constant units, 89 contain more than 1 % and 22 more than 5 %. Note that we here used the same notion of units as in the rest of the paper, meaning that we take the spatial mean of feature maps with spatial dimensions (e.g., for convolutional layers). Fig. 15 : Ratio of Constant Units. We compute the ratio of units constant with respect to the input (over the training set of ImageNet-2012) for all models considered. While the ratio is low for most models, it becomes large for a few models.In addition to the 9 models investigated by Zimmermann et al. (2023) (GoogLeNet, Robust (L2) ResNet-50, DenseNet-101, WideResNet-50, Clip ViT-B32. ViT-B32), we include one more model suggested by them (Robust (L2) ResNet-50) and 825 models from timm (Wightman, 2019) .c3 in1k, poolformerv2 m48.sail in1k, tf efficientnet b1.aa in1k, edgenext base.usi in1k, tf efficientnet es.in1k, tresnet l.miil in1k 448, resnet152.a1h in1k, mixnet s.ft in1k, resnet50.am in1k, rexnet 100.nav in1k, xcit large 24 p8 224.fb dist in1k, deit3 base patch16 224.fb in22k ft in1k, xcit tiny 24 p8 384.fb dist in1k, coat lite medium 384.in1k, focalnet small srf.ms in1k, vit base patch8 224.augreg in21k ft in1k, convnext tiny hnf.a2h in1k, visformer small.in1k, vit small r26 s32 384.augreg in21k ft in1k, vgg16 bn.tv in1k, eca nfnet l1.ra2 in1k, xcit small 12 p8 224.fb in1k, beitv2 base patch16 224.in1k ft in22k in1k, cs3edgenet x.c2 in1k, vit base patch16 clip 384.laion2b ft in12k in1k, xcit small 12 p16 224.fb dist in1k, convformer b36.sail in1k 384, bat resnext26ts.ch in1k, caformer b36.sail in1k, dla34.in1k, crossvit 18 dagger 240.in1k, tf efficientnetv2 s.in21k ft in1k, focalnet base srf.ms in1k, convformer b36.sail in22k ft in1k 384, resnet34.tv in1k, resmlp 24 224.fb distilled in1k, convnext base.clip laion2b augreg ft in12k in1k, caformer s18.sail in1k 384, resnetaa50.a1h in1k, beitv2 base patch16 224.in1k ft in1k, convformer m36.sail in22k ft in1k, inception resnet v2.tf ens adv in1k, mobilenetv2 110d.ra in1k, resnext101 32x4d.fb swsl ig1b ft in1k, regnetx 008.tv2 in1k, convnext small.in12k ft in1k 384, levit conv 128.fb dist in1k, volo d3 224.sail in1k, nest tiny jx.goog in1k, mobileone s2.apple in1k, fastvit t8.apple dist in1k, halo2botnet50ts 256.a1h in1k, mobilenetv2 140.ra in1k, caformer m36.sail in1k, seresnet50.ra2 in1k, hardcorenas d.miil green in1k, convformer b36.sail in1k, regnety 320.swag ft in1k, volo d4 448.sail in1k, tf efficientnet b2.ns jft in1k, sebotnet33ts 256.a1h in1k, vit small patch32 224.augreg in21k ft in1k, vit base patch32 224.sam in1k, resnetv2 50d gn.ah in1k, mobileone s4.apple in1k, coat small.in1k, tf mixnet l.in1k, resnet34.a2 in1k, regnetx 032.pycls in1k, resnetaa101d.sw in12k ft in1k, lcnet 100.ra2 in1k, repvgg b1.rvgg in1k, crossvit 15 240.in1k, edgenext x small.in1k, repvit m1 5.dist 300e in1k, hardcorenas a.miil green in1k, efficientformer l1.snap dist in1k, tf mobilenetv3 large 075.in1k, hrnet w18 small.ms in1k, tf efficientnet b2.in1k, ghostnetv2 130.in1k, ecaresnet26t.ra2 in1k, fastvit s12.apple in1k, xcit tiny 12 p8 224.fb dist in1k, tresnet m.miil in21k ft in1k, fastvit sa24.apple in1k, resnetrs200.tf in1k, convnextv2 nano.fcmae ft in1k, resnet50.ra in1k, resnet34.bt in1k, regnety 002.pycls in1k, focalnet base lrf.ms in1k, dla102.in1k, regnetz e8.ra3 in1k, pvt v2 b0.in1k, xcit medium 24 p8 224.fb in1k, regnety 640.seer ft in1k, resnet200d.ra2 in1k, caformer s36.sail in1k 384, deit3 small patch16 384.fb in22k ft in1k, eca resnext26ts.ch in1k, vgg13.tv in1k, tf efficientnet lite0.in1k, resnet50.b1k in1k, dla60 res2next.in1k, repvit m1 1.dist 300e in1k, convnext base.fb in22k ft in1k, tf efficientnet cc b0 4e.in1k, ese vovnet19b dw.ra in1k, resnetv2 152x2 bit.goog teacher in21k ft in1k, deit base distilled patch16 , resnet50.a3 in1k, convnext atto.d2 in1k, efficientnet el pruned.in1k, volo d2 384.sail in1k, resnext101 32x4d .fb ssl yfcc100m ft in1k, repvit m0 9.dist 300e in1k, regnety 120.sw in12k ft in1k, beit base patch16 384.in22k ft in22k in1k, mobilenetv3 large 100.miil in21k ft in1k, tf efficientnet b0.aa in1k, inception next small.sail in1k, deit base distilled patch16 224.fb in1k, lcnet 075.ra2 in1k, xcit tiny 12 p8 224.fb in1k, resnet101.gluon in1k, dpn92.mx in1k, resnet101.a1 in1k, selecsls60.in1k, beit base patch16 224.in22k ft in22k in1k, convnextv2 tiny.fcmae ft in1k, res2net50 26w 8s.in1k, sequencer2d m.in1k, vit medium patch16 gap 256.sw in12k ft in1k, regnetx 008.pycls in1k, resnet50.a2 in1k, res2net101d.in1k, vit large patch16 384.augreg in21k ft in1k, pvt v2 b2 li.in1k, regnetx 006.pycls in1k, xcit tiny 24 p16 224.fb in1k, pvt v2 b5.in1k, resnext50 32x4d.ra in1k, resnest14d.gluon in1k, caformer m36.sail in1k 384, resnet50.gluon in1k, resnet152s.gluon in1k, flexivit large.600ep in1k, resnetv2 50x1 bit.goog distilled in1k, resmlp 24 224.fb in1k, deit3 large patch16 224.fb in1k, seres-next50 32x4d.gluon in1k, densenet121.tv in1k, resnet152.a3 in1k, ghostnet 100.in1k, tf efficientnet b2.ap in1k, regnetx 002.pycls in1k. While the mean of the per-unit interpretability varies in a rather narrow value range (see Fig. 3 ), we investigate differences in the distribution of scores. Specifically, we are interested in the effective width of the distribution, i.e., how low does the minimal MIS per model go? To make the analysis robust against outliers, we do not use the minimum but instead the 5th percentile. Note that this corresponds to the lower end of the shaded area in Fig. 3 . Compared to the average MIS, we see higher variability across models. Zimmermann et al. (2023) investigate nine different models but test two of them in multiple settings, resulting in 14 distinct experimental conditions to compare. 



Reviews: review: title: title: Great paper
main_review: main_review: First, I would like to congratulate the authors. The paper is really well-written, clear, and offers many interesting insights. I truly enjoyed reading it and believe it could significantly impact the explainability community. I particularly appreciate the interventional study.

I didn’t catch any major concerns (**M**), however, I've identified some minor concerns (**m**) and suggestions that could further enhance the paper's value.

**m.1**: Perhaps the most significant minor concern I have is that the paper relies exclusively on neuron-wise study. The authors acknowledge that superposition may strongly affect their data (and even measure it with their width experiments). Thus, I believe there is a subtle contradiction in your conclusion of fig.7: could it be that more performant models have more concepts and thus exhibit more superposition? I would really love to see a control, as I believe we may have a confounding factor: better-performing models may just be less monosemantic, not less interpretable. In other words, better models learn abstractions that push them to group images that are not perceptually similar. Sorry if I am being picky, but I believe this paper could have a strong impact, and this clarification could strengthen the argument or provide a clearer focus for the study's objectives.

**m.2**: I would be really interested in seeing examples where MIS fails to predict human scores. For example, are human and MIS able to "understand" low-freq/high-freq detectors like those in Schubert & al [1]? It's crucial to consider if the goal of explainable AI (XAI) is solely to uncover concepts/things humans already know, or if the most intriguing cases arise when images surprise us or differ significantly from expectations. 
Put another way, someone could even argue that the neurons grouping images that look perceptually dissimilar may be the most interesting ones? I believe this is a minor issue, as we could use MIS to directly target those neurons, but I would love to see a discussion on that.

**m.3**: Figure 10 suggests that wider layers are more interpretable with less superposition, aligning with the hypothesis. However, it raises a question: is the paper merely measuring superposition? The findings suggest that as models become more performant and data becomes more compressed, superposition occurs, and neuron-wise interpretability diminishes because neurons may not be the correct basis for interpretation (see M.1).
The paper might benefit from explicitly testing if monosemanticity is what the benchmark measures. One approach could be training a model with an L1 penalty on the last neuron to bias the model towards a neuron basis to see if it achieves better performance on the benchmark. This could provide a direct test of the paper's underlying assumptions and contribute to the ongoing discussion about the role of monosemanticity in model interpretability.

**m.5**: Some missing citations for "Quantifying Interpretability" [2,3,4,5,6,7,8,9], as well as for the "interpretable representations of neural activations" [10,11,12,13,14,15]

Again, the issues raised are intended to offer constructive criticism and actionable feedback, reflecting my belief in the paper's potential to have a big positive impact on the community.


References:
- [1] Schubert, et al., "High-Low Frequency Detectors", Distill, 2021.
- [2] Nguyen et al., "The effectiveness of feature attribution methods and its correlation with automatic evaluation scores", NeurIPS.
- [3] Colin et al., "What I Cannot Predict, I Do Not Understand: A Human-Centered Evaluation Framework for Explainability Methods", NeurIPS.
- [4] Hase et al., "Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?", ACL.
- [5] Nguyen et al., "Visual correspondence-based explanations improve AI robustness and human-AI team accuracy", NeurIPS.
- [6] Aodha et al., "Teaching categories to human learners with visual explanations", CVPR.
- [7] Chandrasekaran et al., "Do explanations make VQA models more predictable to a human?", ACL.
- [8] Alufaisan et al., "Does Explainable Artificial Intelligence Improve Human Decision Making?", AAAI.
- [9] Felix Biessmann et al., "Quality Metrics for Transparent Machine Learning With and Without Humans In the Loop Are Not Correlated", ICML.
- [10] Ghorbani et al., "Towards automatic concept-based explanations", NeurIPS.
- [11] Zhang et al., "Invertible concept-based explanations for cnn models with non-negative concept activation vectors", AAAI.
- [12] Fel et al., "Craft: Concept recursive activation factorization for explainability", CVPR.
- [13] Zhang et al., "Efficient neural network robustness certification with general activation functions", NeurIPS.
- [14] Vielhaben et al., "Multi-dimensional concept discovery (mcd): A unifying framework with completeness guarantees", TMLR.
- [15] Fel et al., "A Holistic Approach to Unifying Automatic Concept Extraction and Concept Importance Estimation", NeurIPS.
strength_weakness: strength_weakness: No major concern
title: title: Review of submission 16
main_review: main_review: This paper introduces the machine interpretability score (MIS), a metric that can be used to assess the interpretability of units in neural network layers. The quality of the paper is high and the results are particularly interesting for this workshop. Among the major contributions, the authors introduce a metric to automate the extraction of similarity explanations without requiring human annotations. Some results are highlighted: 1) MIS is strongly correlated with the human interpretability score (HIS), but it does not require annotation. 2) The MIS score is evaluated on more than 800 neural models on all layers, and these neural networks trained on Imagenet show an interesting anticorrelation between task accuracy and MIS. 3) The layer type, width, and depth in the NN are relevant for increasing per-unit interpretability. All these results are very interesting and merit to be discussed further at the workshop. 

I have some remarks for the authors, which could be useful to address in future work or for a conference paper. The following does not count in my evaluation. I noticed that the related work focuses more on mechanistic interpretability rather than explainability, but this lacks a huge piece of research that is going on. The type of explanations that are used are prototype-like explanations (based on a similarity score) that are well known in the literature, see e.g. Guidotti et al. (2018) A survey of methods for explaining black box models. Also, I am not pretty much convinced that explanations of this kind (divided into positive and negative samples) are exhaustive, they only capture binary relations.
It would be interesting to explore other cases and also connect to causal alignment, see Geiger et al. (2021), Causal abstractions of neural networks.
strength_weakness: strength_weakness: N/A
title: title: An interesting paper on an automated metric for the interpretability of vision models offers valuable insights into model training and architecture design.
main_review: main_review: The paper presents the machine interpretability score (MIS), an automated metric for the interpretability of vision models. 
The heart of the method is a predicted probability metric expressing how close an explanation is to a query. The visual explanations are a set of positive and negative images w.r.t. some unit; the queries are images with the most extreme positive and negative activation.
The paper provides a related work section on 1) “units” - the aspect of the NN methods striving to interpret, 2) quantifying interpretability, namely 2-AFC, and 3) automated interpretability research. 
The author presents their method and argues for the hyperparameter choices, followed by a validation of the MIS. 
The author compares MIS with the Human Interpretability Score (HIS), showing a strong correlation on the IMI dataset.
The high performance of MIS is further evaluated on different models, for different layers, and during training.
They make some interesting findings possible through the automated nature of MIS, e.g., that linear layers score higher than others (like conv layers).
The paper concludes with a reflection on the metric usefulness, a list of open questions, and future research directions.

Pros:
- The paper is well-written, and design and experiment decisions are easily understandable
- MIS can offer valuable insights that could benefit various research directions, as presented in the paper. I think I am most excited to use this metric to guide neural architecture choice.

Cons:
- One downside, which may be resolved after thinking more about this method, is its failure cases. For example, one thing that immediately struck me as suspicious is the high MIS for untrained models in Figure 12. I would expect a ~random chance for a model explanation to be correct. (Again, I may be lacking in grasping some crucial aspects of MIS.) I am happy to see Fig. 24/25 starting this discussion. 

In summary, this is a fascinating paper, and I am happy to see more research like it, especially for language models!
strength_weakness: strength_weakness: - good idea, well executed