Measuring Per-Unit Interpretability at Scale Without Humans
Authors: Roland S. Zimmermann
David Klindt
Wieland Brendel
Venue: NeurIPS 2024 poster (2024)
Keywords: interpretability
explainability
deep learning
neural networks
analysis
activation maximization
alignment
evaluation
Primary area: interpretability_and_explainability
TL;DR: We introduce the first scalable method to measure the per-unit interpretability in vision neural networks and demonstrate its alignment to human judgements.

Abstract: In today’s era, whatever we can measure at scale, we can optimize. So far, measuring the interpretability of units in deep neural networks (DNNs) for computer vision still requires direct human evaluation and is not scalable. As a result, the inner workings of DNNs remain a mystery despite the remarkable progress we have seen in their applications. In this work, we introduce the first scalable method to measure the per-unit interpretability in vision DNNs. This method does not require any human evaluations, yet its prediction correlates well with existing human interpretability measurements. We validate its predictive power through an interventional human psychophysics study. We demonstrate the usefulness of this measure by performing previously infeasible experiments: (1) A large-scale interpretability analysis across more than 70 million units from 835 computer vision models, and (2) an extensive analysis of how units transform during training. We find an anticorrelation between a model's downstream classification performance and per-unit interpretability, which is also observable during model training. Furthermore, we see that a layer's location and width influence its interpretability.

Measuring Per-Unit Interpretability at Scale Without Humans
Roland S. Zimmermann MPI-IS, Tübingen Al Center
David Klindt Stanford
Wieland Brendel MPI-IS, Tübingen Al Center
Abstract
In today's era, whatever we can measure scale, we can optimize. So far, mea- suring the interpretability of units in deep neural networks (DNNs) for computer vision still requires direct human evaluation and not scalable. As a result, the inner workings of DNNs remain mystery despite the remarkable progress we have seen in their applications. In this work, we introduce the first scalable method measure the per-unit interpretability in vision DNNs. This method does not quire any human evaluations, yet its prediction correlates well with existing human interpretability measurements. We validate its predictive power through an interven- tional human psychophysics study. We demonstrate the usefulness of this measure by performing previously infeasible experiments: (1) large-scale interpretability analysis across more than 70 million units from 835 computer vision models, and (2) an extensive analysis of how units transform during training. We find an anti- correlation between model's downstream classification performance and per-unit interpretability, which also observable during model training. Furthermore, we see that layer's location and width influence its interpretability. Online version, code and interactive visualizations available brendel-
grenddllgruup.gihubb.ommmms github.io/mis
Introduction
With the arrival of the first non-trivial neural networks, researchers got interested in understanding their inner workings 24 [6] For one, this can be motivated by scientific curiosity; for another, better understanding might lead to building more reliable, efficient, fairer models. While the performance of machine learning models has seen remarkable improvement over the last few years, understanding of information processing has progressed more slowly. Nevertheless, understanding
how complex models - egg., language models (7) or vision models 34
work is still an active 33 A common approach in this
50
and growing field of research, coined mechanistic interpretability
field is to divide a network into atomic units, hoping they are easier to comprehend. Here, atomic units might refer to individual neurons or channels of (convolutional) layers [34], or general vectors in
feature space [12 233. Besides this approach, mechanistic interpretability also includes the detection
of neural circuits g or analysis of global network properties 29
The goal of understanding the inner workings of a neural network is inherently human-centric Irrespective of what tools have been used, in the end, humans should have better comprehension of the network. However, measuring interpretability through human evaluations
cime-consuming and costly due to their reliance on human labor (50] This results in slower research progress, as validating novel hypotheses takes longer. Removing the need for human labor by automating the interpretability measure open up multiple high-impact research directions: First, it enables the creation of more interpretable networks by explicitly optimizing for interpretability can measure at scale, we can optimize. Second, it allows more efficient research on explanation methods and might increase our understanding of neural networks. Due to the lack of a reliable automated measure, previous work resorted limited time-consuming human evaluations, partially producing inconclusive results [e.g., 7 39). highlighting the urgency of finding an automated measure.
after all, what
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

A Task Definition ffor Human Psychophysics Study)
B
Replace Humans with Machine
High Correlation of Humans Machines
Which quary imge matcchee the Postivve Explorations better?
Explonations Miltmslly
(Mooimall Actlating
ueuum warrrrrrreee
Queries
Probablltg Right
Choice
LAverage Tasks
Machine Interpretability
(MIS)
Mochine Interpeetabbiitt
Fig. 1: Definition of the Machine Interpretability Score. A. We build top of the established task definition for quantifying the per-unit interpretability via human psychophysics experiments [ The task measures how well participants understand the sensitivity of a unit by asking them match strongly activating query images to strongly activating visual explanations of the unit. Red and blue squares illustrate the unit's minimally and maximally activating images; shaded and solid squares denote natural test images and explanations, respectively. See Fig. 2 for examples. B. Crucially, we remove the need for humans and fully automate the evaluation: We pass the explanations and query images through a feature encoder to compute pair-wise image similarities (DreamSim) before using (hard-coded) binary classifier solve the underlying task. Finally, the Machine Interpretability Score (MIS) is the average of the predicted probability of the correct choice over N tasks for the same unit. C. The MIS proves to be highly correlated with human interpretability ratings and allows fast evaluations of new hypotheses.
The present work is the first introduce a fully automated interpretability measure (Fig. DA & B) for vision models: the Machine Interpretability Score (MIS). By leveraging the latest advances in image similarity functions aligned with human perception, we obtain measure that is strongly predictive of
interpretability Fig. We verify our measure through both correlational and interventional experiments. By removing the need for human labor, we can scale existing evaluations up by multiple orders of magnitude. Finally, this work demonstrates potential workflows and use cases of our MIS.
2 Related Work
Mechanistic Interpretability While the overall field of explainable AI (XAI) tries to increase our understanding of neural networks, multiple subbranches with different foci exist [1]. One of these branches, mechanistic interpretability, tries to improve our understanding of neural networks by understanding their building blocks [33] An even more fine-grained branch per-unit mechanistic interpretabilityy aims to interpret individual units of vision models B 48 27 34). We focus
exclusively on this branch of research in the present work. This line of research for artificial neural networks was, arguably, inspired by similar efforts in neuroscience for biological neural networks 20 37).
Different studies set out to understand the behavior and sensitivity of individual units of vision networks here, unit can, e.g., be (the spatial average of) channel in convolutional neural network (CNN) or a neuron in a multilayer perceptron (MLP). The level of understanding obtained for unit is commonly called the per-unit interpretability; by averaging over a representative subset of units in the network, one obtains the per-model interpretability [50]. With the recent progress in lision-language modeling, few approaches started using textual descriptions of unit's 's's behavior
[18 21 However, the majority still uses visual explanations which are either synthesized by performing activation maximization through gradient ascent 34 26 30 28
46 31,, or strongly
activating dataset examples 34 6. With the increasing usage of large language models (LLM), there
also now an increasing interest in mechanistic interpretability of them [e.g., 11366 7]
Quantifying Interpretability Rigorous evaluations, including falsifiable hypothesis testing, are critical for research on interpretability methods (2). This also encompasses the need for human- centric evaluations [6] (2). Nevertheless, such human-centric evaluations of interpretability methods are only available some sub-fields. Specifically for the type of interpretability we are concerned about in this work, i.e., the per-unit interpretability of vision models, two methods for quantifying the helpfulness of explanations to humans were introduced before: Borowski et [ presented a wwlternative-foreedeeooo
(2-AFC) psychophysics task that requires participants to determine

---

which of two images elicits higher activation of the unit in question, given visual explanations (i.e., images that strongly activate or deactivate the unit, see Fig. [T]] of the unit's behavior. Zimmermann et al. 49 extended this paradigm to quantify how well participants can predict the influence of interventions in the form of occlusions in images. While these studies used their paradigms to evaluate the usefulness of different interpretabilityy methods, Zimmermann et al. [50] leveraged them to compare the interpretability of models. Due to the reliance on human experiments, they could only probe the interpretability of 767 units from nine models. We now automatize this evaluation to scale it up by multiple orders of magnitude to more than 70 million units across 835 models.
Automating Interpretability Research To increase the efficiency of interpretability research and scale it to large modern-day networks, the concept of automated interpretability was proposed in the domain of natural language processing [5) This approach uses an LLM to generate textual descriptions of the behavior of units in another LLM. Follow-up work by Huang et al. [19], however, pointed out potential problems regarding the correctness of the explanations. Besides automating interpretability research of individual units, there are also efforts for automating the discovery and interpretation of neural circuits and subnetworks [9] 43]. To benchmark future fully automated interpretabilityy tools, acting as independent agents, Schwettmann et al. [41] introduced a synthetic benchmark suite inspired by the behavior of neural networks. In computer vision. there are also efforts to automate interpretability research [8] 50]. Hernandez et al. [88] and Oikarinen and Weng 32) map visual to textual explanations of unit's behavior using automated tools, hoping to increase the efficiency of evaluations. Zimmermann et al. [50] introduced the ImageNet Mechanistic Interpretability (IMI) dataset, containing per-unit interpretability annotations from humans for 767 units, meant to foster research on automating interpretability evaluations,
3 Method
We now introduce our fully automated interpretability measure, Machine Interpretability Score (MIS), visualized in Fig. [ Borowski et al. 6 proposed a psychophysical experiment for quantifying the per-unit interpretability of vision models, i.e., how well humans can infer the sensitivity of unit in vision model from visual explanations. Here, a unit can be channel in a CNN, commonly averaged over space, neuron in MLP, or arbitrary linear combinations of different units. The experiment uses 2-AFC task design (see Fig. [A to measure how well humans understand a unit by probing how well they can predict which of two extremely activating (query) images yields higher activation, after seeing visual explanations. Specifically, two sets of explanations are displayed: highly and weakly activating images, called positive and negative explanations, respectively. See Appx. A. for a more detailed task description. We build on top of this paradigm but replace human participants with machines, resulting in a fully automated interpretability metric that requires no humans.
Definition of the Machine Interpretability Score Let Z denote the space of valid input images for model. For specific explanation method and a unit in question, we denote the unit's positive and negative visual explanations as sets of images E+ C I and E C I, respectively. Further, let @ C I and Q C I be the sets of query images with the most extreme (positive and negative) activations. The task by Borowski et al. [6 can now be expressed as: Given explanations 8t and E- and two queries qt E Q+ and q E Q which of the two queries matches E+ and which E more closely? An intuitive way to solve this binary decision task is to compare each query with every explanation and match the query images to the sets of explanations based on the images' similarities.
To formalize this, we introduce a perceptual (image) similarity function f: x I + R computing the scalar similarity of two images [47), and an aggregation function RR IK reducing set of K similarities to a single one. This allows us to define the function 8: I
x I + IR that quantifies
the similarity of a single query image to set of explanations:
=( == a ((S(q.e) e EE
To decide whether a single query image is more likely to be the positive one, we can compute whether it is more similar to the positive than the negative explanations. We can compute this now for both the positive and the negative query images and get:
A St = a(qt,Et -s(qt, (gg,8tt = ==(q=s(T s(q s(qq (8)).
(2)
(3)
3

---

The classification problem will be solved correctly if the similarity of qt to et relative to E- is stronger than those of q This means we can define the probability of solving the binary classification problem correctly
p(q", q --:""
(4)
where a denotes the sigmoid function and or is free parameter to calibrate the classifier's confidence.
We define the Machine Interpretability Score (MIS) as the predicted probability of making the right choice, averaged over N tasks for the same unit. Across these different tasks, the query images q q vary to cover wider range of the unit's behavior. If the explanation method used is stochastic, is advisable also average over different explanations.
MIS
q.
(5)
Note that the MIS is not general property of unit but depends the explanation method used. A general score can be defined by aggregating the MIS over multiple explanation methods.
Choice of Hyperparameters. We use the current state-of-the-an perceptual similarity, DreamSim 14]. as f. See Appx. foor a sensitivity study on this choice. DreamSim models the perceptual similarity of two images as the cosine similarity of the images' representations from (multiple) computer vision backbones. These were first pre-trained with, e.g., CLIP-style training (38] and then fine-tuned to match human annotations for image similarities of pairs of images. We use the mean to aggregate the distances between query image and multiple explanations to a single scalar, i.e.,
a(#1...., == 1/K Σ; Ti. To choose a, we use the interpretability annotations of IMI (50): randomly chosen subset of just 5% of the annotated units approximately match the value range of human interpretability scores, resulting in 0.16. Note that a is, in fact, the only free parameter of our metric, resulting in very low chances of overfitting the metric to the IMI dataset. We use the same strategy as Borowski et al. [6. Zimmermann et al. [49 and Zimmermann al. [50) for generating new tasks (see Appx. A.2 As they used up to 20 tasks per unit, we average over N = 20. See Appx. D for sensitivity study.
We optimize
4 Results
This section is structured into two parts: First, we validate our Machine Interpretability Score (MIS) by showing that is well correlated with existing interpretability annotations. Then, we demonstrate what type of experiments become feasible by having access such an automated interpretability measure. Our experiments use the best-working - according to human judgements [6] visual explanation method, dataset examples, for computing the MIS. We demonstrate the applicability of our method to other interpretability methods (e.g.. feature visualizations) in Appx. E Note that
different explanation methods might require different hyperparameters for computing the MIS. Both query images and explanations are chosen from the training set of ImageNet-201 44 When investigating layers whose feature maps have spatial dimensions, we consider the spatial mean over channel as one unit [e.g..[6] (6. We ignore units with constant activations from our analysis as there is no behavior to understand (see Appx. for details). The code for all experiments is included in the supplementary material and will be publicly released.
4.1 Validating the Machine Interpretability Score
We validate our MIS measure by using the interpretability annotations in the IMI dataset (00 which will be referred to as Human Interpretability Scores (HIS). The per-unit annotations are responses to the 2-AFC task described in Sec. 3 averaged over 30 participants. IMI contains scores for a subset
of units for nine models
4.1.1 MIS Explains Existing Data
First, we reproduce the main result of Zimmermann et al. [50]: A comparison of nine models in terms of their per-unit interpretability. We plot the HIS and MIS values (averaged over all units in
'wwo models were tested in multiple settings, resulting in 14 distinct experimental conditions compare.

---

B
C
Human Interpretability Score (HIS)
0.90
1.0
1.0
Models
0.85
= =0.44
0.9
Pg = 0.98
0.9
0.80
0.8
0.8
Hardest
0.75
Random (IMI)
0.6
0.7
Easiest
0.70
0.5
0.65
Units
0.6
0.4
P. 0.80
0.60
0.3
Pp 0.80
0.5
Chance
0.6
0.7
0.8
0.9
0.4
0.6
0.8 1.0
GoogLeNet
ResNet
Machine Interpretability Score (MIS)
Model
Fig. 2: Validation of the MIS. Our proposed Machine Interpretabilityy Score (MIS) explains existing
interpretability annotations (Human Interpretability Score, HIS) from IMI (50) well. (A) MIS Explains Interpretability Model Rankings. The MIS reproduces the ranking of models presented in IMI while being fully automated and not requiring any human labor, as evident by the strong correlation between MIS and HIS. Similar results found for the interpretability afforded by another explanation method in Appx. E (B) MIS Explains Per-unit Interpretability Annotations. The MIS also explains individual per-unit interpretability annotations. We show the calculated MIS and the recorded HIS for every unit in IMI and find high correlation matching the noise ceiling at p 0.80 (see Appx. C. (C) MIS Allows Defection of (Non-) Interpretable Units. We use the MIS to perform a causal intervention and determine the least (hardest) and most (easiest) interpretable units in a GoogLeNet and ResNet-50. Using the psychophysics setup of Zimmermann al. (50), we measure their interpretability and compare them to randomly sampled units. Strikingly, the psychophysics results match the predicted properties: Units with the lowest MIS have significantly lower interpretability than random units, which have significantly lower interpretability than those with the highest MIS. Errorbars denote the 95 % confidence interval.
model) in Fig. 2A and find very strong correlations (Pearson's 's = 0.98 and Spearman's ' r== = 0.94). Reproducing the model ranking is strong evidence for the validity of the metric, as no information about these rankings was explicitly used to create our new measure.
zoom in and look at individual units instead of per-model averages. Fig. 23 shows MIS and HIS for all units of IMI. It clearly shows a strong correlation (Pearson's 's and Spearman's s P. = Pp 0.80). The interpretability scores in IMI are (potentially noisy) estimate over a finite number of annotators. We estimate the ceiling performance due to noise (sampling 30 trials from a Bernoulli distribution) to equal Pearson's Pp 0.82 (see Appx. for details). We can conclude that the MIS explains existing interpretability annotations well both on a per-unit and on per-model level.
Next,
4.1.2 MIS Makes Novel Predictions
While the previous results show strong relation between MIS and human-perceived interpretability. they are descriptive (correlational). To further test the match between MIS and HIS, we now turn to causal (interventional) experiment: Instead of predicting the interpretability of units after psychophysics evaluation produced their human scores, we now compute the MIS before conducting the psychophysies evaluation. We perform our experiment for two models: GoogLeNet and a ResNet- 50. For each model, IMI contains interpretability scores for 96 randomly chosen units. We look all the units not tested so far and find the 42 units yielding the highest (Easiest, average of 0.99 for both models) and lowest (Hardest, average of 0.63 and 0.59, respectively) MIS, respectively. Then, we use the same setup as Zimmermann et al. (50) and perform a psychophysical evaluation on Amazon Mechanical Turk with 236 participants (Appx. BB. We compare the HIS for the random units from the IMI dataset and the two newly recorded groups (easy, hard) of units in Fig. 2 The results are very clear again: As predicted by the MIS, the HIS is highest for the easiest and lowest for the hardest units. Further, the HIS is close to the priori determined MIS given above. On this newly collected data, again find high correlation between MIS and HIS (Pearson's 's Pp 0.85.
5

---

WideResilet.s te
1.0
Derosinir 201
Res
VIT-B32
0.9
viT-32
etee
ResNet-50
ConvNexTaB
Robustt
30
0.8
Average
0.7
0.6
100
200
300
400
500
600
700
800
Model (Somted by Average Score)
Fig. 3: Comparison of the Average Per-unit MIS for Models. We substantially extend the analysis of Zimmermann et al. [50] from noisy average over few units for few models to all units of 835 models. The models are compared regarding their average per-unit interpretabilityy (as judged by MIS); the shaded depicts the 5th 95th percentile over units. We that all models fall into intermediate performance regime, with stronger changes in interpretability at the tails of the model ranking. Models probed by Zimmermann al. (50) are highlighted in red.
Spearman's P. 0.81)). This demonstrates the strong predictive power of the MIS and its ability be used for formulating novel hypotheses.
4.2 Analyzing & Comparing Hundreds of Models
After confirming the validity of the MIS, we now change gears and show use cases for it, i.e., analyses that were truly infeasible before due to the high cost of human evaluations required for measuring the per-unit interpretability. These costs prevented fine-grained analyses. Crucially, our understanding of what influences unit's interpretability is still fairly limited. For example, it is unclear whether units of specific layer types are more interpretable, or whether layer's 's position or width influences its units interpretability. Equipped with the proposed MIS we now investigate these relations.
4.2.1 Comparison of Models
Zimmermann et al. [50] investigated whether model or training design choices influence the in- terpretability of vision models. Although they invested a considerable amount of money in this investigation (2 12 000 USD), they could only compare nine models via a subset of units. We now scale up this line of work by two orders of magnitude and investigate all units 835 models, almost all of which come from the well-established computer vision library timm [4]. These models differ architecture and training datasets were all least fine-tuned on ImageNet. See Appx. for list of models. Putting this scale into perspective, achieving the same scale by scaling up previous human psychophysics experiments would amount the absurd costs of more than one billion USD. Following previous work we ignore the first and last layers of each model [50].
When sorting the models according to their average MIS (Fig. 3) they span a value range of a 0.80 - 0.91. The strongest differences across models are present at the tails of the ranking. Note that GoogLeNet is ranked as the most interpretable model, resonating with the community' interest in GoogLeNet as it is widely claimed to be more interpretable. The shaded area denotes the 5th to 95th percentile of the distribution across units. This reveals a strong difference the variability of units for different models; further, as the upper end of the MIS is similar across models ( 95 %), most of the change in the average score seems to stem from a change in the lower end, with decreasing width of the per-unit distribution for higher model rank. Note that the MIS cannot only be computed for the most extremely activating query images (see Sec. but also for less activating ones. Refer to Fig. 21 for a version of Fig. 3 that uses the 2nd/98th percentile instead of the most extremely activating query images.
To investigate the difference in how the MIS of units is distributed between different models, we select 15 exemplary models and visualize their per-unit MIS distribution Fig. 4B Those models were chosen according to the distance between 5th and 95th percentile (five with highest, average, and lowest distance). While models with low and medium variability have unimodal left-skewed distributions, the ones with high variability have rather bimodal distribution. Note that the distribu-

---

A
B
0.92
Highest Medium Lowest
Average Per-unit MIS
0.90
0.88
Variability
0.86
0.84
Models
Pp 0.54
0.80
p.
0.59
0.78
60
70
75
80
85
90
1.0 0.5
1.0 Per-unit MIS
0.5
1.0 0.5
ImageNet Top-1 Accuracy
Fig. 4: (A) Relation Between ImageNet Accuracy and MIS. The average per-unit MIS model is anticorrelated with its ImageNet classification accuracy. Refer to Tab. f a list of the Pareto-optimal models. (B) Distribution of per-unit MIS. Distribution of the per-unit MIS for 15 models, chosen based on the size of the error bar in Fig. 3 lowest (top row), medium (middle row), and highest variability (bottom row). While most models have an unimodal distribution, those with high variability have second mode with lower MIS.
Average Perlayertype MIS
0.90
0.85
0.80
BatchNorm
LayerNorm
0.75
Conv
Linear
GroupNorm
0.70
100
200
300
400
500
600
700
800
Model (Sorted by Average Score)
Fig. 5: Comparison of the Average Per-unit MIS for Different Layer Types and Models. We show the average interpretability of units from the most common layer types in vision models (BatchNorm, Conv, GroupNorm, LayerNorm, Linear). We follow Zimmermann et al. 50 and restrict our analysis of Vision Transformers to the linear layers each attention head. While not every layer type is used by every model, we still see some separation between types (see Fig. [8 for significance results): Linear and convolutional layers mostly outperform normalization layers. Models are sorted by average per-unit interpretability, as in Fig.
tion's second, stronger mode has similar mean and shape to the overall distribution for models with low variability. The first mode is placed at value range slightly above 0.5, close to the task's 's chance level, indicating mostly uninterpretable units. This suggests that subset of uninterpretable units (see Fig. 228 for examples) can explain most of the models' differences in average MIS. We analyze this further Fig. 22 where compare the models in terms of their worst units. We similar shape as in Fig. but with larger value range used, resulting in stronger model differences.
Previous work analyzed potential correlation between interpretability and downstream classification performance. However, a limited evaluation, it was found that better classifiers are not necessarily more interpretable [50]- A re-evaluation of this question is performed in Fig. 4A and paints an even darker picture: Here, better performing ImageNet classifiers are less interpretable (Pearson's -0.5 and Spearman's 's -0.55). A similar analysis investigating the influence of model's input resolution on its interpretability suggests no influence (sec Fig.
Besides analyzing the interpretability of models, one can also use the MIS to analyze interpretability tools. Above, we directly looked the interpretability work proposed leveraging sparse auto-encoders (SAE) to first transform a model's activations into potentially more interpretable basis before analyzing [e.g.. g.... 7] While their application has been
model's s activations: however, recent
mostly limited to language models (with the exception of 23 ). we now apply them to vision models
first exploratory analysis: In Appx. [ we use the MIS to compare the interpretability of a model's original layer and of two competing SAE variants
39 7
and find systematic difference.

---

A Conv/Linear
BatchNorm P. - 0.77
LayerNorm P, 0.85
GroupNorm
B
Conv/Linear
BatchNorm
P; 0.85
P. 0.00
P. 0.70
P, 0.50
Pp 0.85
P= 0.77
Pp 0.86
Pp 0.19
Po = 0.94
= 0.41
0.900
0.88
Average Per-unit MIS MIS
0.875
0.850-
0.86-
0.825-
X
0.84-
0.800-
X
0.775
0.82
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Relative Layer Depth
Relative Layer Width
Fig. 6: (A) Deeper Layers are More Interpretable. Average MIS per layer as function of the relative depth of the layer within the network, grouped by layer types. For each type, the values are grouped into 30 bins of equal count based on the relative depth. The crosses depict the bin averages (correlations are calculated for those, too); for a visualization including the bins' variance see Fig. 23 (B) Wider Layers are More Interpretable. Average MIS per layer as a function of their relative width, grouped by layer types. The values are grouped into 5 bins. See Fig. 24 for visualizations of how the median, 5th, or 95th percentile of MIS depend on the layer width.
4.2.2 Comparison of Layers
Next, we zoom into the results of Fig. F and investigate potential differences between layers. First, we interested in testing whether the layer type is important, e.g., convolutional more interpretable than normalization or linear layers? In Fig. 5] we sort the models by their average MIS over all layer types but show individual points for each of the five most common types (Conv, Linear, BatchNorm, LayerNorm, and GroupNorm). The number of points per model may vary, as not all models contain layers of all types. The figure shows a benefit of Conv over BatchNorm layers, which themselves are better than LayerNorm layers. Linear layers, if present, outperform both Batch- and LayerNorm as well as Conv layers. While the differences are small, they are statistically significant due the large number of scores collected (see Fig. 18).
Second, we analyze whether the location of a layer inside model plays role, c.g., earlier layers more interpretable than later ones? The average per-unit MIS (for each layer type) is shown in Fig.66/) function of the relative depth of the layer. A value of zero corresponds to the first and value of one to the last layer analyzed. The scores are averaged in bins of equal count defined by the relative layer depth to enhance readability. The resulting curves all follow similar pattern: They start high, decrease in the first fifth, then increase steadily until they drop in the last tenth again, resulting in an almost sinusoidal shape.
Third, it is interesting to probe the influence of the width of layers on their average interpretability. Based on the superposition hypothesis [12 35 I [66. one might expect wider layers to be more interpretable as features do not have to form in superposition (i.c., as polysemantic units) but can arise in disentangled form (i.e., as monosemantic units). Fig. 6 shows the relation between MIS and relative layer width. We use the relative rather than the absolute width to reduce the influence of the overall model and show the results of models with different architectures on the same axis. Note that, nevertheless, there might be other confounding factors correlated with the width, e.g., the layer depth. While we only see weak correlation for BatchNorm layers, we find stronger one for Conv/Linear layers. It is unclear what causes this difference in behavior. However, we see this as a hint that one way increase a model's 's interpretability is increase the width (and the number) of layers.
4.3 How Does the MIS Change During Training?
In the last set of experiments, we demonstrate how the MIS can be used analyze models in fine-grained way and obtain insights into their training dynamics. For this, we train a ResNet-50 on ImageNet-2012, following the training recipe A3 of Wightman al. (45). for 100 epochs.
Fig- shows how the average per-unit MIS (left) changes during the training. Notably, the initial MIS (of the untrained network) is already above chance level. Visual explanations (sce supplementary
8

---

Figure 7: Interpretability During Training. For a ResNet-50 trained for 100 epochs on ImageNet, we track the MIS and accuracy after every epoch (epoch O refers to initialization). While the MIS improves drastically in the first epoch, it decays during the rest of the training (left). This results in an antipro- portional relation between MIS and accuracy (right).
0.85
0.80
0.75
25 ImageNet Top-1 Accuracy
Training Epoch
material) indicate high color dependence of this network's units. However, during the first epoch, the MIS still increases drastically to values around 0.93, before it decays over the rest of the training. This indicates non-trivial dynamics of feature learning. which we analyze in Fig. When showing the MIS as a function of ImageNet accuracy during training (right), a strong anticorrelation (ignoring the first points) becomes evident. This aligns with the anticorrelation shown in Fig. 44 While do not have a definite answer for why this is happening, we hypothesize the following: This could be sign of learning dynamics and the order in which features are learned. After initialization, the network can improve the fastest by learning very simple feature detectors (e.g., colors, simple geometric shapes), as those are weakly correlated with certain classes (e.g.. blue colors increase the chance of seeing fish). Those features are casy for humans to understand. Throughout the training. these feature detectors are replaced with more complex ones that are harder to decode. Fig. [25 the least/most activating dataset examples for units with a strong MIS drop between the second and last training epoch, matching our hypothesis. To better understand the dynamics through the training
0.50
Change in MIs 10
0.00
EatchWorm
Conv
AN-Layer Block
1131:11113 1111111111111 1el11111111111 0el1122222222222 0022222322233
33333333333333
53333333333 333333333 333
34344444444444
44444444444444444444444444444444
4444444444
0000000001 0112233011
11111122222202 22000220000000000000000000000000
22233323
33000000000111000000000000000000
033554555535555 55355
000000001
22222222
Layer
2231222333301
Layer (Farly
late)
Fig. 8: Change of Interpretability per Layer During Training. To better understand the peak in interpretability after the first training epoch found in Fig. 7 we display the change in MIS during the first epoch, averaged over each layer. Layers are sorted by depth from left to right, and different colors encode different layer types. The change in interpretability appears moderately correlated with a layer's depth, such that deeper layers improve the strongest, whereas early layers show no improvement. For an extended visualization covering the full training, see Fig. 20
most importantly during the first epoch- change in MIS. Fig. Sshows the change in MIS during the first epoch for each layer separately (ordered by their depth within the network). We detect trend of later layers improving more strongly than earlier ones: The change in MIS is heavily driven by the later layers in the network, whose MIS increases strongly while early laters show improvement at first. In general, do not see difference between Conv and BatchNorm layers.
zoom in find out which units cause this strong
5
Conclusion
This paper presented the first fully automated intepretability metric for vision models: the machine interpretability score (MIS). We verified its alignment to human interpretability score (HIS) through both correlational and interventional experiments. We expect our MIS to enable experiments pre- viously considered infeasible due to the costly reliance on human evaluations. To stress this, we demonstrated the metric's 's usefulness for formulating and testing new hypotheses about a network's behavior through series of experiments: Based on the largest comparison of vision models in terms of their per-unit interpretability so far, we investigated potential influences on their interpretability, such as layer depth and width. Most importantly, we find an anticorrelation between a model's downstream performance and its per-unit interpretability. Further, we performed the first detailed analysis of how the interpretability changes during training.

---

While this paper considerably advances the state of interpretability evaluations, there are some open questions and potential future research directions. Most importantly, the performance of our MIS on per-unit level is close to the noise ceiling determined by the limited number of human interpretability annotations available. This means that future changes in the MIS measure (e.g., based on other image perceptual similarities) might require additional human labels to determine the significance of performance improvements. Additional human labels could also be leveraged to improve the MIS by following Fu et al. [14] to fine-tune the image similarity directly on human judgments. In another direction, using vision language models for computing the MIS could be interesting as this might, in addition numerical score, also provide textual description of unit's sensitivity (18). Finding a differentiable approximation of the MIS will be valuable for explicitly training models to be interpretable [50]. Note that while this paper looked the interpretability of channels and neurons, can also be used to analyze arbitrary directions in activation space. Thus, we expect the MIS to also be valuable for researchers generally looking for more interpretable representations of (artificial) neural activations [7]. Finally, exploring whether this concept of interpretability quantification can be expanded to LLMs is exciting direction.
Author Contributions
RSZ led the project, which DK initiated. DK proposed using perceptual similarity functions to build interoperability metric. RSZ and WB conceived the final formulation of the metric. RSZ conducted all the experiments with suggestions from WB and feedback from DK. RSZ executed the data analysis, except for the estimation of the noise ceiling conducted by DK. RSZ created all the figures in the paper and wrote the manuscript with suggestions from DK and WB.
Acknowledgments
This work was supported by the German Federal Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ: 18099.. WB acknowledges financial support via an Emmy Noether Grant funded by the German Research Foundation (DFG) under grant no. BR 6382/1-1 and via the Open Philantropy Foundation funded by the Good Ventures Foundation. WB a member of the Machine Learning Cluster of Excellence, EXC number 2064/1 Project number 390727645. This research utilized compute resources at the Tübingen Machine Learning Cloud, DFG FKZ INST 37/1057-1 FUGG. The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting RSZ.
10

---

References
[1] Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. Linear Algebraic Structure of Word Senses, with Applications to Polysemy, December 2018. Cited on page 8
[2] Horace Barlow. Single units and sensation: A neuron doctrine for perceptual psychology? Perception, 1:371-94, 02 1972. doi: 10. 10.1066/0010371. Cited on page 2
[3] David Bau, Bolci Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissec- tion: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017. Cited on page
[4] David Bau, Jun Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, and Antonio Torralba. Understanding the role of individual units in deep neural network. Proceedings of the National Academy of Sciences, 177448):3071-33077 September 2020. doi: 10. (0.1073/pnas. http3/paan.1.7375111 1907375117. Cited on page 2
URL htttp:://dot orgooog///1 10. 1073//00s /0t 1907375117
[5] Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Πlya Sutskever, Jan Leike, Jeff Wu, and William Saunders. Language models can ex- plain neurons in language models. https: /pppaaippppbiccccccmmm blob. core.windowe n
neuron- paaron-explaiier/pappeee htapliiierr/pppr//neex.x. index. html 2023. Cited on page 3
[6] Judy Borowski, Roland S. Zimmermann, Judith Schepers, Robert Geirhos, Thomas S. A. Wallis, Matthias Bethge, and Wieland Brendel. Exemplary natural images explain cnn activations better than state-of-the-art feature visualization. In Ninth International Conference Learning Representations (ICLR 2021), 2021. Cited on pages 2
and
[7] Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Con- erly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodke Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke. Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023, https:/trransfommer- circuits.
html. Cited on pages
20 and 21
[8] Nick Cammarata, Shan Carter, Gabriel Goh, Chris Olah, Michael Petrov, Ludwig Schubert, Chelsea Voss, Ben Egan, and Swee Kiat Lim. Thread: Circuits. Distill, 2020. doi: 10.23915/ distill.00024.
Cited on page
[9] Arthur Conmy, Augustine N. Mavor-Parker. Aengus Lynch, Stefan Heimersheim, and Adrià Garriga-Alonso. Towards Automated Circuit Discovery for Mechanistic Interpretability, October 2023. Cited on page 3
[10] Keyan Ding, Kede Ma, Shiqi Wang, and Eero Simoncelli. Image Quality Assessment: Unifying Structure and Texture Similarity. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(5):2567-258 May 2022. ISSN 1939-3539, doi: 10. 10.1109/TPAMI.2022 3045810. Cited on page 17
[11] Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. http:///taansffmme--rrrruuu
httpb//////.aaammeoo//ieeeee. html. Cited on page 2
[12] Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, 7ac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish. Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition. Transformer Circuits Thread, 2022. Cited on pages and
[13] Dumitru Erhan, Y. Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of deep network. Technical Report, Univeristé de Montréal, 01 2009. Cited on page 2

---

[14] Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang. Tali Dekel, and Phillip Isola. DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data, December 2023. Cited on pages
T0 and 17
[15] Leilani H. Gilpin, David Bau, Ben Z. Yuan, Ayesha Bajwa, Michael A. Specter, and Lalana Kagal. Explaining explanations: An overview of interpretability of machine learning. 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA), pages 80-89, 2018. Cited on page 2
[16] Gabriel Goh. Decoding the Thought Vector. Cited on 8
2016.
[17] Mara Graziani, Laura O' M'Mahony, An-phi Nguyen, Henning Müller, and Vincent Andrearezyk. Uncovering unique concept vectors through latent space decomposition. Transactions Machine Learning Research, 2023. Cited on page 10
[18] Evan Hernandez, Sarah Schwettmann, David Bau, Teona Bagashvili, Antonio Torralba, and Jacob Andreas. Natural Language Descriptions of Deep Visual Features, April 2022. Cited on
pages and T0
[19] Jing Huang, Atticus Geiger, Karel D' D'Oosterlinck, Zhengxuan Wu, and Christopher Potts. Rig- orously assessing natural language explanations of neurons. arXiv preprint arXiv:2309.10311
2023. Cited on page 3
[20] D H Hubel and T N Wiesel. Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. J. Physiol., (60(())000-111))) 154, January 1962. Cited on page
[21] Neha Kalibhat, Shweta Bhardwaj, C Bayan Bruss, Hamed Firooz, Maziar Sanjabi, and Soheil Feizi. Identifying interpretable subspaces in image representations. International Conference on Machine Learning, pages
PMLR, 2023. Cited page
[22] Sunnie S. Y. Kim, Nicole Meister, Vikram V. Ramaswamy, Ruth Fong. and Olga Russakovsky. HIVE: Evaluating the human interpretability of visual explanations. In European Conference on Computer Vision (ECCV), 2022. Cited on page
[23] David Klindt, Sophia Sanborn, Francisco Acosta, Frédéric Poitevin, and Nina Miolane. Iden- tifying interpretable visual features in artificial and biological neural systems. arXiv preprint arXiv:2310.11431, 2023. Cited on pages and 7
[24] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. ImageNet Classification with Deep Convolutional Neural Networks. In Peter L. Bartlett, Fernando C. N. Pereira, Christopher J. C. Burges, Léon Bottou, and Kilian Q. Weinberger, editors, Advances in Neural Information Processing Systems 25: 26th Annual Conference
Neural Information Processing Systems 2012. Proceedings of a Meeting Held December 3-6, 2012, Lake Tahoe, Nevada, United States,
pages 1106-1114 2012. Cited pages 1 and
[25] Matthew L.. Leavitt and Ari S. Morcos. Towards falsifiable interpretability research. CoRR, abs/2010. 12016, 2020. URL htttps://arxiv.oo
abs/ 12016 Cited on page
[26] A. Mahendran and A. Vedaldi. Understanding deep image representations by inverting them. In 2015 IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 5188-5196, Los Alamitos, CA, USA, jun 2015. IEEE Computer Society. doi: 10. 1 WWW.CVVPR..01.729115...
URL https: : /doi. ieeecompttersocietyy.. org/ 1109/CVPE, 2015. 7299155.
Cited
on pages and 2
[27] Ari S. Morcos, David G.T. Barrett, Neil C. Rabinowitz, and Matthew Botvinick. On the
importance of single directions for generalization. In International Conference on Learning
Representations, 2018. URL BRRLattpss//oppeervvie.... http://rrwrweeeetoooo/r/er//rr/i
Cited on
page
[28] Alexander Mordvintsev, Chris Olah, and Mike Tyka. Inceptionism:
Going deeper into neural networks, 2015. URL. https: //ai googleblog. habbloo..oo//111151 com/2015/0//
cepptiioniimgoonng going-deeper--... deeper- - into-neural html Cited
page
12

---

Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 2020. doi: //311///siilll.0022..... http://dktilll.b2/2202///eeiiroo [29] Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. 2023. Cited on page
[30] Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 12 2014. Cited on page 2
[31] Anh Nguyen, Jeff Clune, Yoshua Bengio, Alexey Dosovitskiy, and Jason Yosinski. Plug & play generative networks: Conditional iterative generation of images in latent space. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2017. Cited on
page 2
[32] Tuomas Oikarinen and Tsui- Wei Weng. Clip-dissect: Automatic description of neuron rep- resentations in deep vision networks. In The Eleventh International Conference on Learning Representations, 2022. Cited on page
[33] Chris Olah. Mechanistic interpretabilitty, variables, and the importance of interpretable bases, 2022. URL bettps:/traasfforrte witprnnnttooorrrrrutttoossssssse html
circuits,,.... prrrcutts..ubb222///eedde interp--eirrp-ee essay/index
Cited on pages
and
[34] Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2017. doi: 1/.33915dissiill.00000 htpp:://isttll http://disiil.puub201////// / Cited on pages T and
2
[35]
Cited on page
[36]
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads. Transformer Circuits Thread, 2022. https//transforme-crrrccce wwwpeeiitttuururi///2..e wttt//2022/mmcooeeeetteee
-anhiinn-nndiiie induuciin
heads/index. Cited on page 2
[37] Quian Quiroga, Leila Reddy, Gabriel Kreiman, Christof Koch, and Itzhak Fried. Invariant visual representation by single neurons in the human brain. Nature, 135(7045):1002-11 1107, 2005. Cited on page
[38] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision, February Cited on page
2021.
[39] Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, János Kramár, Rohin Shah, and Neel Nanda. Improving Dictionary Learning with Gated Sparse Autoencoders, April 2024. Cited on pages 20 and 21
[40] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211 252, 2015. doi: 10. 000007/s12633-01-00116666
Cited on pages and 34
[41] Sarah Schwettmann, Tamar Rott Shaham, Joanna Materzynska, Neil Chowdhury, Shuang Li, Jacob Andreas, David Bau, and Antonio Torralba. FIND: Function Description Benchmark for Evaluating Interpretability Methods, December 2023. Cited page 3
[42] Karen Simonyan and Andrew Zisserman. Very Deep Convolutional Networks for Large-Scale Image Recognition. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference
Track Proceedings, 2015, Cited page 17
13

---

[43] Aaquib Syed, Can Rager, and Arthur Conmy. Attribution Patching Outperforms Automated Circuit Discovery, November 2023. Cited on page 3
[44] Ross Wightman. Pytorch image models.
httpss//github...cm////gggg.aaai
pytorch-image- modelmmmmelelellsseeammooooo 2019. Cited on pages
and
and
21
[45] Ross Wightman, Hugo Touvron, and Hervé Jégou. Resnet strikes back: An improved training procedure in timm. arXiv preprint arXi::2 2021. Cited on page
[46] Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson. Understanding neural networks through deep visualization. Deep Learning Workshop, International Conference on Machine Learning (ICML), 2015. Cited on page 2
[47] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The Unrea- sonable Effectiveness of Deep Features as Perceptual Metric. In 2018 IEFE Conference on Computer Vision and Puttern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 586-595. Computer Vision Foundation IEEE Computer Society, 2018. doi: 10.1109/CVPR. 20222.018806 Cited pages
and 17
[48] Bolci Zhou, Yiyou Sun, David Bau, and Antonio Torralba. Revisiting the importance of individual units in cnns via ablation. CoRR, abs/1806.02891, 2018. hRt/http///axii org/
abs/1806. 128116666991111 Cited page
[49] Roland S. Zimmermann, Judy Borowski, Robert Geirhos, Matthias Bethge, Thomas Wallis, and Wieland Brendel. How well do feature visualizations support causal understanding of cnn activations? In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang. and Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 11730-11744 Curran Associates, Inc., 2021. URL https: http::t:pprrocceeiinnnneee newrips wwtpppprpppppeeeeeeeee cc/paper_files/ /ttp:///0..........////////66660
pdf Cited on pages
and 16
[50] Roland S. Zimmermann, Thomas Klein, and Wieland Brendel. Scale alone does not improve mechanistic interpretability in vision models. In Thirty-seventh Conference Neural Informa- tion Processing Systems, 2023, URL https:://///
Cited on pages
5
and 34
14

---

A Description of the 2-AFC Task
A.1 Task Design
Our proposed MIS builds on the 2-AFC task designed by Borowski et al. [6] to conduct human psychophysies experiments, An example of such a task is given in Fig. 9
This task aims to probe how well (human) participants can detect the sensitivity of a unit of a neural network based visual explanations of it. Understanding the unit's sensitivity should allow participants to distinguish between a stimulus eliciting high from one yielding low activation. Therefore, the task shows the participants two such images, called query images, and asks them to pick the image eliciting higher activation. To solve the task, participants also see two sets of visual explanations: Positive explanations describe the patterns the unit activates strongly for, while negative activations show patterns the unit weakly responds to. For solving this task, there are two potential strategies: Participants can either recognize a common pattern of the positive explanations in one of the query images, making this the correct choice. Or they detect common pattern of the negative explanations
query image, making the other one the right choice. See Borowski et al. 49 or Zimmermann et al. 50 for alternative descriptions and visualizations
[6]. Zimmermann et al. of the task.
Negative Explanations
Posttive Explanations
Negative Explanations
Positive Explanations
Queries
Queries
Fig. 9: Examples of the 2-AFC Task. For two different units of GoogLeNet one task each is shown. Every task contains a set of negative (left) and positive (right) visual explanations describing which visual feature the unit is sensitive to. the center, two query images in the form of strongly and weakly activating dataset examples are shown, respectively. This means that each one of the two query images corresponds to the positive and the other to the negative explanations. The task is now to choose which query image corresponds to the positive ones.
A.2 Task Construction
For constructing tasks, follow Zimmermann et al. [50]. Specifically, this means that we use K (positive and negative) explanations in each task. We restrict explanations to natural dataset examples to reduce complexity but note that the same setup can also be applied to other visual explanations, such as feature visualizations. To choose query images and explanations, we proceed as follows: For each unit, we determine the N (K + 1) most and least activating images, respectively. Out of these, the N . K most extreme images are used as explanations, the others as query images. The N K potential explanation images are uniformly distributed across tasks according to their elicited activation level (see 66 [50] for more details).
15

---

B Psychophysical Evaluation
In Sec. 4.1.2 we validate the correctness of the proposed MIS in terms of estimating the interpretability for new, not previously analyzed, units. As described above, we use the MIS to create set of the most and least interpretable units (42 units each). We then collect human annotations (i.e., human interpretability scores) for these units by running psychophysical experiment. Specifically, we let humans solve the 2-AFC task described
Appx. A..1 originally introduced by [6]. We use open-sourced setup of Zimmermann et al. [50] and recruit participants from Amazon Mechanical Turk. See Fig. TO for screenshots of the task and the information given to participants on how their anonymized responses will be used. We leverage established quality/attention checks to control for high data quality and restrict the pool of potential participants to experienced workers [49 50]. In total, we recruit 236 participants for targeted hourly rate of 15 USD, i.c., 2. 2.79 USD per task based on the average task duration reported by Zimmermann al. [50]. All procedures conform to Standard 8 of the American Psychological 405 Association's ' "Ethical Principles Psychologists and Code Conduct" (2016).
evaluation. The recruited workers compensated at
academic experiment image dassification.
Please do not participate in esperimers imner if you're colarblind or your viskanis
seniussy impaired.
In this experiment, two groups of images will be presented to you
Your
Images which of imagee better. Please you Your anorymized responses used scientific
group
Below you can see what the experiment looks like. participate
man sestot narree
which mage at the center matches the Right Examples better?
Examples
Continoe
KERNELS
WALL
Fig. 10: Screenshot of the initial overview of the HIT presented to workers considering the task. We inform participants that they consent to their anonymized data being used for a scientific study.
16

---

C Influence of the Underlying Perceptual Similarity on the Machine Interpretability Score
the underlying perceptual similarity f for all experiments shown so far. We now repeat the experiments on IMI in Sec. 4.1.1 with two alternative similarity measures: LPIPS [47] and DISTS [0]] While all three measures are based on learned image features, DreamSim leverages an ensemble of modern vision models trained on larger datasets compared to LPIPS and DISTS, which AlexNet (24] and VGG16 [42 trained ImageNet, respectively. According to Fu et al. [14], DreamSim clearly outperforms LPIPS and DISTS on image similarity benchmarks.
As stated in Sec. used DreamSim 14
When comparing MIS based on DreamSim with one based on LPIPS and DISTS on a per-model level (see Fig. [ one very similar results and strong correlations between each MIS and HIS. This might suggest that the choice of the similarity function to use has little influence on the quality of MIS. The picture, however, changes when zooming in and looking per-unit interpretability (see Fig. 13). Now, it becomes evident that the MIS based on DreamSim outperforms that based on LPIPS and DISTS, indicated by the higher correlation and smaller spread of the point cloud. We, therefore, conclude that DreamSim is the best perceptuall similarity available for computing machine interpretability scores.
0.90
Human Interpretabiiity Score
Models
Models As C.
0.85
- 0.95
Pu 0.98
0.99
0.80
0.70
0.65
0.60
0.6
0.7
0.8
0.9
MIS (LPIPS)
MIS (DISTS)
Fig. 11: LPIPS and DISTS Perform Similarly DreamSim when Comparing Models. We compare DreamSim with two earlier perceptual similarity metrics, LPIPS and DISTS. All three lead to similar results on IMI (cf. Fig. A See Fig. 13 for comparing these similarity functions per-umit level. standard deviation.
Noise Ceiling of Annotations in IMI To put the difference in performance between the perceptual similarities on a per-unit level into context, we estimate the noise ceiling of the data: As the HIS for single unit is (potentially) noisy estimate over (up to 30) human decisions, has some uncertainty. To account for this, we run a statistical simulation in which we model individual human responses as binary decisions from a Bernoulli distribution whose mean equals the unit's 's HIS. We can now simulate human decisions by sampling from the distribution. Then, we compute the correlation between MIS and simulated HIS and repeat the process 1 000 times. The resulting noise ceiling is compared to the correlations obtained when using LPIPS, DISTS, and DreamSim in Fig. 12 DreamSim's s performance very close to the noise ceiling for estimating the per-unit human interpretability.
D Sensitivity of the MIS on the Number of Tasks
As described in Sec. 3 we compute the MIS by averaging over N = 20 tasks. This choice was initially motivated by previous work by Borowski al. [6] We investigate now how this choice influences the MIS. For this, we perform two experiments for GoogLeNet (see Fig. 14h. First, we use the method for constructing tasks described before in Appx. A.2 to create 20 tasks per unit and then compute how the MIS changes when only using the first = I. 19 tasks compared to all 20. While this setting is straightforward to analyze, it does not reflect how the number tasks influences the MIS computation in practice: Using the task creation above, the chosen number of tasks influences the creation of all tasks, e-g., adding one more task changes which images are used for previous tasks. Therefore, in the second experiment, we again measure how the MIS changes
17

---

Sptsrran Condation (HIS, ASS
Nosse Colling
0.8
0.8
0.70
0.80
0.20
0.03
Pearson
02-
0.2
LPPPS
DIST5
DreamSiim
LPIPS
DISTS
DreamSim
Simiiartty
Fig. 12: Best Perceptual Similarity Approaches Noise Ceiling, Considering the noise ceiling, caused by the inherent uncertainty of the HIS, the best perceptual similarity (DreamSim) shows an almost perfect performance. The black bar and shaded area show the mean correlation and standard deviation over 000 simulations, respectively.
1.0
0.9
Mad Someth
0.5
0.7
0.39
00.0.3
3.00
5.52
0.70
Fig. 13: LPIPS and DISTS Perform Worse than DreamSim when Comparing Individual Units. We compare DreamSim with two earlier perceptual similarity metries, LPIPS and DISTS. While LPIPS and DISTS perform similarly to DreamSim on a per-model level of IMI (cf. Fig. 13 they lead to performance per-unit level.
when using = 1.... 19 tasks compared to 20, but recreate all tasks when increasing their number. For both settings, we see that the residual converges zero, with a slower convergence in the more realistic setting.
Absolute Difference in MIS
0.08
Absolute Difference in MIS
0.08
0.06
0.06
0.04
0.04
0.02
0.00
17 Number Tasks
Number of Tasks
(a) New tasks do not influence earlier tasks.
(b) New tasks influence carlier tasks.
MIS. We investigate how MIS changes depending on the number of tasks N that it is computed over. Here, distinguish between two settings. In (a), simulate that adding another task does not change the selection of query images and explanations in earlier tasks; in (b), this is not the case. While the former easier to analyze due to reduced level of randomness, note that the latter is the more relevant setting in practice. For both cases, we visualize the average absolute difference in MIS estimated for 20 and N 20 tasks.
Fig. 14: Convergence
18

---

E
Applying MIS for Different Explanation Methods
The experiments in Sec. compute the MIS for one type of explanation, namely strongly activating dataset examples. We now demonstrate that the same approach easily generalizes to other visual explanations: feature visualizations. We do not tune any hyperparameters but re-use the same as presented in Sec. for dataset examples as explanations. In Fig. 13 we repeat the experiment from and again see a strong correlation between MIS and HIS.
Fig. 2
0.75
Human Interpreta biiretyblltyyyo Score
Models
0.70
0.95
0.60
0.55
0.55
0.60
0.65
0.70
Machine Interpretabillty Score
Fig. 15: MIS Generalizes Well to Other Explanation Types. We find high correlation between MIS and HIS for other explanation types (feature visualizations). See Fig. [or the corresponding results for using natural dataset examples as explanations.
F
Analysis of Constant Units
After training a network, it might happen that some of its units effectively become non-active/cconstnt for any relevant image. We here call unit constant if the difference between maximally and minimally elicited activation by the entire ImageNet-2011 beginning of Sec. 4 we excluded those units in our analysis, as they do not present any interesting behavior that worth understanding. Note that this does not mean that it will be interesting to understand why such units exist. In Fig. [6] we display the ratio of constant units for each model. For most models, we see low number of constant units: Specifically, we see that out of the 835 models investigated, 256 do not contain any constant units, 89 contain more than % and 22 more than 5 %. Note that we here used the same notion of units as in the rest of the paper, meaning that we take the spatial mean of feature maps with spatial dimensions (e.g., for convolutional layers).
training set less than 10 As mentioned the
0.25
Rell of Constant Uniss
0.20
0.10
0.05
0.00
200
400
600
800
by
Fig. 16: Ratio of Constant Units. We compute the ratio of units constant with respect to the input (over the training set of ImageNet-2012)) for all models considered. While the ratio is low for most models, it becomes large for few models.
G Computational Resources
Computing the MIS of unit consists of four steps: (1) determining its visual explanations, (2) finding the strongly and weakly activating dataset samples to be used as the query images of the 2-AFC task, (3) computing the pairwise image similarities, and (4) computing the
Complexity of MIS
19

---

final MIS. Due to the simplicity of the MIS' computation, its cost is neglectable. The complexity of the first step depends on the visualization method used: Gradient-based search algorithms, e.g.. feature visualizations, require hundreds of forward and backward passes (of small batches), while determining dataset examples requires only a single forward pass over sufficiently large dataset. The second step also mostly requires single forward pass over this large dataset. Thus, if dataset examples are used as explanations, this step is free. Performing the third step requires computing the pairwise similarities of the images used in the created tasks. However, as most perceptual similarities, most importantly the leveraged DreamSim metric, are computed as the cosine similarity of an image's features, the step can be greatly simplified: We first compute and store the features for every image in the dataset used to sample the tasks' images. Then, computing the similarities equals only querying two features from hash map and computing their cosine similarity. While this caching approach is not necessary for computing the MIS of a single unit, it becomes important when computing it for thousands of units. In this case, computing and caching the similarities also becomes neglectable, meaning that the computational cost of the MIS is dominated by the first and second steps. In summary, computing the MIS mostly resorts to a single forward pass over a sufficiently large dataset and additional forward/backwarl passes only depending on the visualization technique used.
Resourees Used Due to the aforementioned low computational complexity of the MIS, the experi- ments in Sec. d not require much compute: Evaluating all units of model takes, average and varying depending on the model's 's size, less than one hour on a GPU (e.g.. NVIDIA RTX 2080-TI or V100). Therefore, reproducing the experimental results of this paper requires approximately 1000 GPU hours.
H
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning, specifically the field of Interpretable Machine Learning. The main contribution our work is the presentation of more time- and cost-efficient approach for quantifying how well humans can understand neural activations. potential risk in automating interpretabiliity research is that we will start optimizing for metries that are never fully aligned with human judgments. It is conceivable that this will encourage the design of models that ace our metric but whose inner workings and decision-making processes are still obscure to human observers. This would set false goalposts and potentially come with safety risks if a high score in MIS were mistaken for a white box model that comes with higher trustworthiness. Beyond that, we see many potential use cases for this result (see Sec. 5). that can all advance the state of machine learning. There are potential societal consequences of our work, however, none of which we feel must be specifically highlighted here.
I Analyzing SAES
Sparse Entooennsdder (SAE) have been recently proposed as a means understand the behavior of a network's 's layer better [7]: By finding a new, sparser basis to represent the layer's 's original activation, one hopes to find new artificial computational units that are more monosemantic. These units are expected to be easier to understand, rendering the tasks of understanding the behavior of the entire layer easier, too. While conceptually simple, the implementation and evaluation of SAEs is intricate: Training them requires careful hyperparameter tuning and algorithmic design choices such that the final SAES are as sparse as possible but still faithful to the layer's original activations. However, as no reliable automatic interpretability evaluation has existed so far, evaluating SAFS in terms of how much more interpretable their features are is difficult, resulting in potentially inconclusive results. For example, Rajamanoharan et al. 391 suggested a modification to the usual SAE architecture (Gated SAE) but could not find a statistically significant benefit over the default architecture due to the high and, thus, prohibitive cost of interpretability evaluations.
As the MIS enables cheap interpretabiliity evaluations, we can now pick up this work: In the context of vision models, we train different SAES and Gated SAES and compare their interpretability. Specifically, we train them on activations of one layer of GoogLeNet (mixed4b_3x3) and use different expansion factors and weights of the sparsity loss to obtain different SAES. In addition to their interpretability, we also evaluate models in terms of their sparsity (Lp count) and their reconstruction fidelity, i..., how well they maintain the original model's classification cross-entropy compared to
20

---

random model. In line with [39, Fig. 17 shows that Gated SAES allow a better fidelity vs. sparsity trade-off. terms of their MIS, we do see a systematic difference between the two architectures. Moreover, in light of the high MIS of the original layer (i.e., 91.21 .1 %), we do not see strong benefit of SAEs compared to analyzing the original layer yet.
In another experiment, we trained (vanilla) SAES on another, less interpretable layer (layer2_2 2_co22222222222 of ResNet50). While the units of the original layer have an average MIS of 0.854 we observe that MIS values of up to 0.922 for SAEs with ten times more units than the original layer. See Tab. for sensitivity study on the relationship of an SAE's sparsity and its MIS.
1.00
Fidelity (CE Recovered)
0.920
0.95
MIS
0.90
0.910
0.85
0.905
SAE
Gated SAE
0.900
0.80
510
20
30 40
50 60
Sparsity (LD)
Fig. 17: Comparable MIS for different SAE architectures. We compare two types of SAEs used by Bricken et al. (7] and Rajamanoharan et al. [39 (SAE and Gated SAE, respectively), in terms of their sparsity, reconstruction fidelity and interpretability. While Gated SAEs allow more optimal tradeoff between fidelity and sparsity, they are comparably interpretable as standard SAEs. The SAFs overall MIS is a similar regime as the original layer's (91.21 %), while the sparsity is stronger than that of the original layer (<s 75).
Tab. 17: Sensitivity of SAE's MIS
its Hy Hyperparameters.
Sparsity Weight [10-2]
1.125
2.5
3.75
5.0
6.25
7.5
8.75
10.0
LO Count
233
138
99
75
60
49
41
35
MIS
0.892
0.908
0.916
0.915
0.919
0.918
0.922
0.918
J Details on Models
In addition to the 9 models investigated by Zimmermann et al. [50] (GoogLeNet, ReaNet-50, Clip ResNet-50, Robust (L2) Ro,Net-50, 10nseNet-101, WideResNer-50, Clip VIT-B32. VIT-32)), we include one more model suggested by them (Robust (L2) ResNes-50) and 825 models from timm 44]:
xcit_ xcit_tiny__ tiny__n__yl_____________ 12_p______________11p f21.6_224.h______ 104.fbinily inlk,
vit jatiiiy_patchl6... Baacch6__844augee 384.angry in21k_nn_ppn. inlk,
repghost- repghostnet_000..n inl.. volo_d3 d3 sail saS_sallinl inlk, mobilenetv3 large 1a inlk,, hrnet_w40.mn inlk, convnext nownext_small.nne in 2k ft_ inlk, vit vit_tin____s66,___ 2_________844_0rr0_ in21k_ inlk, vit seltiin_pptche patch16 224.agregg_n211 inli,, inlk. scitlsmall12 p26____________
pit_xx_ 224 inlk,
net Inlikk, regnetz_cl6 ceattt__ll__evvstt evoschhinlk.
poolformer-_mdee sail_ii
vit base ji__baee_ppthh666
2ar.ll_222..gggg_imiee in21k inlk,
regnety_320.rv _o__3320v_2_ilk inIk,
denseneti21.mm
repghostnet_ 150.inkk, cuesmext26ss.ch.nnlee
regnety ww0.::0w__fff..io... ftnik,
vit hase vi__hase_pachh66 p___pahhh__2..aaammmmmm 224.222__iikk
seresnextua stxddddddddddss_o__ hetp/_nn2k__f_aaeeeeeee fok__t_jnlkk2___ 288.
regnety_320.ppp jnlk,
esădarknet caape_mmcssssssseeeee inlk,
resnet10lc.glumn inlk, con-
Umextv2_atto.ff www.tttootacetttttcoo jnikk,
flexivit 60eepp.
mobilenetv2
flexivit_ base. csinoxt5_32xxdddvv._
sosnet152.v_inl seresnext26d 32c4dddb____dddiiiibb
fbnntv3_g.ra_jinkk pool-
former s36.sai__nnkk in1k, jesnext101332x86
jinkd rexnet_ 130.ne_
effiicientii 22tp:/ii__b2222__i1__ convnext faall.ff_jn222_f1 (nnn__ft_iik33844
resne150 gnalh_inll jnlk, eva022 small patch14_3366mmm 1066mmmm_n22k_____ jn2ik_n___i__
gygnety_32..ra__
res2met50dinlk,
convit_small.h inlk,
regneix_ regmeix_ 160pyckk_nnkk, valo dI sall
convmexiv2 Samvnextv2_large..ee femme, Bt (__n22k_inlkk__
ifeefficientnet _0..s.0.......j jf_inlk,, pil_i__244inlk i24.
(it_:mmll_22______01 la___________ds______. jask_ 4pm131.mx_in1k
demvenet169.vv
p2________6_____ 2166222.fb_jnkk pvt_v2_ ww__v2_ba.nnkkk tim_nfnet_fi fl.dm fim__nn inlk, vit largergee_rr pttpppatttchh.ooeeem [44.orrg__n211k fi_ik convnextv2 fomae fttee__f____ f__in22k_in jaik__ikk_3444 garesnet50.m22 inlk, of jf_regne__blle bl.m inik, volo dl sai. inlk, resnet50.ram inlk, hlne_w188 w18 small_y2.m, inlk, convext base llip. biion2b_angreg_. angreg_ff ft. inlk, regnetx_160 10___660022_n inlk, sequencer2d_ Linkk, convnext Jarge.ft fb. in22k inlk, botnet26t 256.c1_________1166 jnik, ge_ wwffccieetteet22..cc _w__age inlk, wide resnet500 2.ramm intk.
resmet10ld.m2_ing inlk, repghostnet_PPe (onghostett220.inikk resnet18a2_ing inlk,
21

---

ball espresnextSrr_iilkk, Sanev225M__evvvall t__etticienmeete2 bttp:/aeeet__23._2222 b3.nn2kk____ 12.ne lambda reseeeeeeeeeeeee2555555 jnnk, sawii_si2taplllediiiiii
2e__mediu__2_______10 innk, repvit d.dist 4S0e_ikk convnext tiny.ni 2k, ft. inlk, xcit Jarge 24_pl6________11111 384.hh dist_ilkk,, resnetv2_50..lt jnlk,, comtnet_ r_rw__wwwww__ _w________ ww__224ww_ in efficientnet_ _es prnneedd dia60_ dia60_res2nttt res2net..nnk effi- cientformer stt.pes___pppppii dist, inlk, cait xxs2__xs24_____ 2244__222hb_p_e diss_ inlk, vit small pat_ssll_paachhl______ sacch66224..agree 224.aagregg_inn saageg___2kk____ ft_ f_innk,, If jf_efficientnet_cc oc bl_ He__ei,nk, efli- cicntvit_ cicntvir_bl.28888 bt.r288_i1__ll.. halonot26tall inlk, mixnet inlk, hmot_ ww4.wssinnllt inlk, regnety_160.v22 (60kk,_ xcit_nano__ samoo_1__pp0 12__p____44________2______ 1__.3884__d_s did_ddssiinn inlk,
regnetx_ 32000yclsssints inlk, seresnextaal0dd 22xdddd___xxdw_aallll inli fi_ikk, efficientvit, b2.r288_ini
scresnext101 1_t.0000133xdddb__m inlk, efficientvic 12.206___2..22111111 vit Hi_base_pate _i__ase_pptchl_______ 2atthho____222.laaiioo 224.Iaion2b ft innk, tf_ #__efficientnet lite2. inl hmot_w18, psddll
deir3_small _mallmlll_____ patehh66222.ff.b
if f__efficientoet_b22.o _2aaa_nkkk
crossvit_ 15__________4444 i40. [[_eflicientnet12 b0..ikk
deitt_smalll pattss_sll_paachh._._.1 wwwphllllcehpppee 224.6_222___bb__ 22w.Ib_inn2k innlk,
inlk, _rak_inkk, haloregnetz, b.aa
eca_nfine_ 10.m2 inlk,
twins_ wwins_pppww_ssee ssppl_smalllikkk
scurennetSuura2.nnl..
latriisall2apple_diii jisk.
22a resne150Ml22 inlk,
vit base bass__atcch2__stee wtt.:l..2.2.2.iio22.... [iinkk, reseeiblur50.bc
resne150.alh inlk, hardeorenas_ e. e.mill _ere__reeennnn preen_inlk,en inlk, contnxxt pano pe_____________ sw inlk, convnext convnext_base.diip base.clip_laiom Jaionn_angreg____n__ At T__jnlk__880 tresner_m miml inikk448, 1es.et inlk, _kklk, poolformerv2_m445 m48.sail inlk, efefiiiieent bllaa_in ja_ink,, edgenext hase usse.usi usi inlk, if =f_efficientnet ik, tres-
basse_paichll____ 224.orig222oorggiog.. in21k innl____iiik__nnli inikk,
net net_Imiil_jnke 448, resnet152ath_ii inlk, mixnet_s.ft s.f.sssff.f... jff_iiik resnet500.m ain_ik,,,,_ink,,, rexnet_ na0 inlk, xcit, large _4___________________ 28__224.h,, diss_jiilk, inlk, deit3, base_ fttee_aatchl__24.2111 224.bb 224.fb_in22_____ ft n_inkk, xcit xcit_tiny__0 p8_______118 f8 dististt_tiiiiiiiii disttiiikkk,, cost medium 184,nlk,, mediuum184.nnIl. focalnet small srf.ms inlk, 224.222222222. (ongrreg_i22Ikff6 in2ik_ft_inik convactt hiriitttnnhooooaeee hof inlk, visformer_sman small vit ( (16__32__88880
vit base_ paac__88___2_aaaarreee
382_3384, fntik_ft_innkkk inlk, beitv2 base cass__attch___2222 224.ink2222inkk ft ft_in22k_inlll inlk, xcit crit_small112.0 (al__l2_p16_22211 22.
vga16_ bo.tv_inkk [.__n Inlk, xoit_small_1n
Le_bbas__pathhl Baathhl___lpp__e111 cllpp3344.aaio222 ft_iniii__iiinn
con-
baL.remexx26iss.c..mo inlk,
caformer 136.sailjnnkk
dla34innk,, crosvii daggerderr__20.li..
(t Lellicientneee2ssinnnlk... innk,,
[ocalnet bas__st..msjilkk
resne134.__ilk.. convformer (atp___66..siiaaa _ii___________i__888
1amip_24_224.11 www_2_4__diiiiiilld_ooo.. dibt_ddiiiled__iikk.
convnext convoexL_baweclipp www..ciippliioooobooooooa lwwgonnnnnugrggg_eee finl22Kknin
resne- caformer_s18.. saiiasssss.siiiiik.. _31
tan50.ath_ ikk heitv2 base pase__atchl6_2222 224in22 inin,,, convformer mantierrmmmssllll in22k ft. inli,, inception_ pesooeerr____eeeeeeee v2.1f enss ady inlk, convnext- mmall.nn122 ft jnlk__84, cont___o________ htwp__/2_bbbbbeee dist_ inlk, volo d3, 224.2222saaaaa inlk, nest_tiny jest_ ji.goog__iik inlk, mobileone s2apple_inlk, fastvit_ fastvit__8.aplle dist inlk, hala2hoimet50ls, 256..ah_innkk inlk, mobilenetv2_1440 140mm140mmmm inlk, caformer_ m36.sail_innt seresmet50.m2 inlk, hardeoremas, d.mii. gmeen inlk, terffoomerr_36.saall jnlkk regnety soeoyy3320swwa_ffhe fiink, volo (0__d4_44,, La..4448.__nnntt of effficientnet b2. so_jit_jinn schotnet33ss 2acie3ttt__6665hbee alaaa_256__hhaann inlk,
mobilenetv2_ I 10d.rajinlk, resmextl01 1emexti01333x4400 32x4d.22 beo4d.fh_swssigggg iglh ft inlk, regnetx, regnetx_008.v22
vit small hatplll__aa____aa_____2_ htwp:/2224.arrennin_ii...e fniik_ftinkkkk
btt_bass__ttchh_222_22_ 224 224.sam_ine sam_ik,, resnetv2_500 _0_____0_h__h_h____ _nnah__inkkk mobileone_s4 satpan___ssapp_ii__e__ inlk, coat_ soat_ssallinnkk, _iiixee__inkk, resnet34.2_jnlkk, regnetx pggeett__22.pyolkkk pyols_nk_ resnetaa sw_dswwjin mr__inl2k_fff ____in2k__f_iikk, inik,, lenet_ 100.00000mmm..mm in www.oddddwwiildi__eeeeee
repogg btwgggbb..ggggilk. crosvit_15_24..in)
x_meexxx_ssmii [ggeeet_x_smallinn
repvit _at____iis__00eee_______ 300e_inkkk, hardcorenas_a.nn hatps__ammiii__eeeeeee inlk,
efficientformer_ll..eo wtw.aa/meeee..speeiiiiaeee. dist_dis__nkk,, [_msbileeett Sarraa_aaar_____aeeeeee Jange_075.nnkk Wnne_ mmilk, inlk, [[_efficientnet_ b2ink,, ghostnetv2_ i30.inkk, ocaresnet26rra2jinl inlk. fattii_sl2appll.ink
pi__y___________________________ 2il__224.bbbbe inlk, tresnet mumil in21k itnkk,, fastvit sa24.aplle inlk, resmetrs200af inlk, conveextv] nemae ft. inlk,, resnet50.ra jaink,, ht inlk, regnety_ pyyll inlk, focalnet base Inf.m, in inlk, regnety (40 seer_ At jnlk, resne1200d.m2 inlk,
dla inlk, regnetz_e8.ae e8.ra_8nnn pvt_y2_ blink,, xeit medium_ 24 p8 caformer_ s36. jalk, _34, deit3 hant_smal__paa patchl_pichl_____ 38..8.4.h in22k, A inlk, eca, resnex226is.chh inlk, vggl3.ty jink,, if efficiientee inlk, resnet50.b1kiimg inlk, dia60_ ins mepvit_ml_in _ddis__00e_iiin__ ese_ pownett9b_dwwer jwlkk resnev2_ convnext Jarge fo (b_jn22k_ff_inl fnikk__nin___,,,
convnext sunvnext_haee.hie in22k n f_ink,, if effficieennee scc 60_ inlk, bitt__is222_bbttooooo_ooo teoche rencher_eee___n211 jt__ijn2kk2_iili jnlki beie_bbs__ddeee base_ddstilled__eeeee pesiilled_pthhh____e 300..3384._bbbb resnec10ld gluoonnnnnn inlk, Barknet53.c2nsjimg jnnk,, poolformerv2 2e1222__6sssssiiiiiaaaaa
convformer_ne (a1eer_m36.si__ ma6.ssil_in22k____ jn_____n2_f_________________ ft_nnik_884,, gigiiionn...goonn_mmmm
gmip_s16 hilp__l6_222.ee_ 224.223_inkk inlk, convformer_sl8.0 s18 inlk, efficientret_em..c. em.eam__nlk in]k,, inception_ v3 llgggouoonniii resmlp_ [ammp__1__200 22_224.fb inlk,
Jesnet_Lmmiiling
ponresnetttdd_ppmemmmiii
resnet152.a2_n111 vit siLssmall_attcb32__22_ Mascb32_384.surreg.ii
ineep-
soo__v.af_adv _dvikk repghostnet_ 130.mlkk, cwit__omm_384.bbb_o jnlk 1ppiit_ml_5.diss--5 jalk, efficientne_el.ee inlk, seres- ne150.a2 jnlk, pits distilled__222...
sa it efffeccienttett ___ Se.inkk densenet20L.v jinik,, (xxpsst55_xxxxdaa._.
csssedarknet Leens_eenll inlk, cait_x24_ mii__x24388... 384.fb 384.ffd8isssiii jilk,, spmssnet_100.mppp. 100.mssp_nnlt inlk, res2net50_144 scinkk, repugy repugg_d2se.rygg inlk,
regnetx_ regnetx_032.v22_. iilkk crossvit_ 18, degggge__gSSiiinl 408.inlk, pit_ distilled_ 224inlk, vggl l.v inlk, lit__nnk,, espresnet50.mn
focrrk Le2ks_inlk, resme150.b inlk,
convexxt22
ftni,,, convnext sonvnext_manoinn22 mmn..nn12k.__iill...
(xmext101_64x4div_iii _4__44d4_xddiid,... sanvnett_nmo.dll inlk, seresnet50.al_in
[_mixnet_manlk xcit_tiny_ 12____2_________ f84.f384fbb inlk,
efficientnet2_ www_tm2__ii inlk, cepption_resntt_oo (ontt_v2.f__iiii inlk, eva_large jmm_Jarg__pachh4____ mmchl4_196.in22k._1 fi_ikkk pvt_y2_bllin inllk, densenet16l.ty.nnl
resnet 5ld.ollnnngg _ilkk, paanety_032.vv2.._ convformer ma6. mal.m_ml_..._lll.3._..... vnextv2 famefeee__ft,ilk,, vit_ Jag.aar_pccch444._.. Baachl4__llp___8111 3tt__epppp_3..___e__ lrinnhhn ft ft_in/2k_inni inlk, joit_coo_218sf____ inst_ hrmet_w48.ms_inlll inlk, reset101aa1 inlk,
dla102x.nIkk, adgenext_smalll www.sssssllwwwwiikk... egnety_016.tv2_img con-
mendium_2__ps 24_____22..6____________________ 224.fb_dist_illl inlk,
tesneus152.dtiilkk,
convectv2_aano.cmmaaa ft inlk. con-
vnextv2_ tuy.tiiaaaeea. wtwas/ftaeefiee jn22k_innkk
sennets.ddsteviill jiikk, selecals42binkk. m33vic_m..disc_jmm inlk,
resnest50d_ inlk, 4pm98.mx inlk, xcit__anoo_i__ p2_.:12_6__22..b__pppa 224.fbjnlkk, gggneu__016.yycs_.in xcit moi__mddim__4444 24____________2________ 224.fo24fbb__ee jilk,
caformer sl8sail_jnnl
sehalomet3its.m2.iil inlk, cinlk
inlk, res2net50_ inikk, Mateeeuu22o_22xdxiiiiiiii 32x4ddii_inkk flexivit_ base. 1200ep_ res2net50_ 20w.:000_w6..willkk...mk
2al16_224.bb_iie dist_inkk,
resnext101 (amnext111_32ABdde 32x.ddv22_2nl inlk,
convnextv2 b...base in22k in1k_384, semnosmet, semmnasnet_055mmm 075.mmsp_inn inlk,,
cait_ xail_xxx24___ 384 fb_ inlk, mobilenetv2 (iciileneet2___2eee0 1200ra inlk, seresmext261
vit_ base
vit_base_patcheee pwwpppaltchhpppeeeepe Bachhhl__e____________ _ipp_384.opoma____ ft iin12kkjnkk nest base base_bx.eooogeee paeejjx.gog__ill
ocaresnetlight.iill mill_nikk
repves_ bovee_bO.rvgg__ig inlk, mixer_ble caee__bl__222mmiii 224.mill_nn2I1 in2lk_ft_inlk,
ccaresnet50rall_mng iillk
inception_nets fectio__nnettttnsssiee sily,ssiliinkkk
regnety (onery_0222yckkinlke vit hat_aae__aatte aasse_atchh2__gg 384 augreg a44eeeggee_in21k in2ik_ filkkfttiii inlk, filnkk
pool- effi-
former_s Homerrss12sse 122psss22saaaiiii inlk,
vit hil_aae__pan case_patch2__eiipp hiiip_p84.ppeai____ ftnn___t_i22kjikkkkk
cientvir hl. adl.222_jnn1,
base pase__atchi6___ clipp334..aaie ft f_ink,, deir biirssmall sii_ssmll__isiiie distilled patch16_2244 221.2224.___i jnlk,, efficientvic 22.0 inlk,
reanest50d.nnk. regnety_ Becpcty_120pppeee 120.pycls_nnkk,
semnasnet_ 100.mmsp.iikk. wide wide resne150_2.n inkk,
small 24 p4_ 24__16_224.0 224.0 inlk.
22

---

resnet101.a5_ail inlk,
IapiitiZapple.ikk [_elllicientnet Lite Litellinkk. .inlk.
LinneeLainnk,
resmip_ 2iiiii__ii_22422222_0 24_224.422 bi_tiit.bdiiiiileeddieeee indl 1esaext50_32xxdl.222
resnev2_152x22 hig.:o/bii.gggeeecceeeeeeeee fnaneher_ii2k___iiiekk
efficient- in22k ft. f_inlk__84,, _84, caformer_s s1. inalliii2kk___ ft inik__84 deit3 hase patp_ppcchh____________ 222.2222222k inlk, vit_ si_mediumm_ patccl_________e Bap________________ sop_334.sw_nnll A f_inlk, sequencer2d_siii mo-
net maik,, convoorer s18,aii [iwffomer_ss8.siill_.
vit heu_hse___mee wwwpepatchhh______2 aanch3__elip_000 clip_384.hiio2h Baahhhhffhhh ft_nn_n___inn_a inlk,,
base aa_2k____iinl fnik,, hileone_ste gaanee____pppeiin__eee _nnk, adgemext_base.n22kk
deit3, medium mesfium_ptthhi6____ patch166222...1 222iil_222__b_be
jnlk,, efficientformerv22 Lsnap. dist inlk, soasvit_small_24..
lambda alb
jnlk, xocptionAlp.aa3
reanext50_32xx0 2aa_a/00__xxxd3_____ _ink,
argve inlk, flexivit largii_lrrge33000p inlk, vit_ vit_base_ passe_pathh3__elli wttpppl..2...eeaaaaf.....m inik,,
xeit_small_ 24_6___2__p1________ ______1_________00 dilki
ropvgg_ repvgg_b3g4.ngg b.g4.rgg_.nnkk resnext50_32x44e 1___3_222ah_hillll
coor_ lite soar_lite_mddingee medium.inkk.
44t lainoobb ft_innniiii ft_nn22__inn inlk, vit base viL_base_patchl... 2pm__24aw_jnn inll,, 20____2m__222iii in22k inilk,
resnext50 1annex150_32xkd.ggee g2xkldgluuenniiill
repvgy b2 inlk,
mixer_bl6 _24 goog in21k Inkk,,
2ll_ge____p1__2222_______ 22sssb_dist____eeeeb__e_pa inlk, repvgg repvgg_a2.rvgg inlk, gernet Lerset_ Lidstev_inlk, mobilevivv (0teeeeeii_....eeeeiiiiii.... convnxxiv2 femme (en2a____i222kiallll innn22 resmet18.. 8.. inlk, ecaresnet50d.miil inlk, coat lill_liese_ii small.inlk, convnext_ xharge fb in22k ftinkk, mobilevitv2 075. inlk, cait x36 184_384.fh fb d_dddi_sikl inlk, efficientformerr22 sl.snap al.ana_ddist_iiii inlk if efficientmetv2 h3.inkk,
resmet50.c2_niie inlk, SambalobotuetS0k_555.... in6k
aa__gghh___iih resnet swsl swsl_iglh_n n f_jnkk
mobileone mobileone_slappphe slapppll_iplll inlk,
resnet6ll.m2. inlk, crossvit tiny. 240,in 38..___8nnnnhn_ke_____ in22k inlk,
mohilevitv__ 1 175.cveetss__ilk inlk,
convnext_ ft__:s___nn___i22kkk___ ftn___nn22____ikkk
caformer_ b36. in22k, n M_inlk__M, 384, resnett (adddra__inkk,,e convit, base.ff inlk, tinynet_ binll, deit3 _ patc___aeecehel____. 384.hl66__8______.. b26.sail_in22k,,__
regnetx_ (annetx_004ivveer 104_rvvv2__nnk cait_xxs3__ f8466_38..b__o_ 38d...__disssiiik convnext nano cacts_aao__oossllll ols.ddhh_nnnkk
liteen inception_ jnlk, dintyydstilled___eee pacicod___pthh______1_1 f2 filkk,
brnet_w 1oo__118..mm....... fasmmm_inn jnlk,
semnct_m.issee _inlk, convforner_s366 336..sil_in222 ftt____tt____ae ft 381,_,444,
deit_smalllpanee paccll____t_____ 2e1666222_b_n_a jnlk,, vit lac_lleg__ptcc___ee 3a..ippp filkk,, 1asvic__8_240.in resnet26.br_nnkk resnet18 Salinkk
deit3 base wwtpeeetceh_____p_paa (aachl6__844.b_in222 [iinkk,
convfomer_,36..siliiim conveex_ssmllll_iinie innl,, selecala60b.in1k el-
licientnet_ iorkainkk, focalnet siitt_sff_ss_illlll
ecaresnet101d.mii.. inlk, regnetx_ inl.
mobileone_ s..pplle__.. seremextaat11d 12....1a_......ai.ini..... loviL_ 192.fb_dist_inkk. focal- Jmmill_nn22, jn2lkin ft inlk, seresnet50.a3 inlk, dla46x ainlk, ex3darknet x x.c2ns inlk,
mobilenetv3_ www.:vspsww..mplkkkkk.. poolformerv2 matoom22mmm.ssiiille
sen_tiny__iee Irf.ms_smiiii inlk, regnety_120..wwg le_in inlk, tresnet_ Jmmil _a__t_____mmmiiimm
if efficientnet siticienteet_b..ppp... anllp.lkk
resnextla1 1onextlo133xxddfff 32x8d.ff_ ssl_yf0e000m ft. inlk, 18__88__________________________ 194.fb_ditt_img inlk, iinll cait, sait_xs24_18.fbb 384.fb inlk, fasivit fastvit,a12..pple_.i jilk,
vit hase ji__hase_pucch66 222.mhle6_2... jn2ik inlk,
xcit_ Jarge_24,
limet, hime_w64.ee m64. inlk, hrmei_w30.mm inlk, of df_efficientte b0.inkk, xcit xcit_tiny_2ee_ Sitiitttn_________ee 2a___24____22_ 22__2__fbb______ dist inlk,
regnety_ 016,,yckk__inkkk
2edd__rssetl01_22ee
beitv2_ hargee_hee....hhh...... watpcpuuhh_h22nillllll 224.221222 Saalkk_fiiiiii [__n22k__nnn inlk,
resnct101.v_inikk repvit_m2.dist_nnkk
coatnet_ mw__22.sw_jnlk, mexivit_small 1200cp_inlh
iiticiintnet_bl.nnll. of_
dia dlal02x2.nllk hardoorenas, fariiiaass_mmi__eeee__ee f.miil_rreenjing
tf_ f__fffccenttett2 b2, inlk, resnet32ts.a22 inlk, xci__tin_11__p$ ptd_y__2___33___bb____ 384.fb_dis_inll pnceptionxv3.v_inn xait 2arre_24_p16.___m 24_p16_224..b_iilll 22k. 03 inlk, repvit (___mm2__iidd 3.dis__40eeinnk,
regnety_ 00_.00004_3a..k... resnext101 32x4d_gluoo_iill
meeetv3_bra22 inlk,
viL_base_patche Seus__pacco__224aaugee www.aggrg__innkkkk. innkk,
es3darknet Leins inlk, dla60x.nnlk,
convnext Lisi..eip_.aionn2b
baggooobunnrggggnaa augreg inlkk__k
regnety_660.deic jnlk,
xcit_ siii_tin___ee___ee 24 p1.___66_..hhh.__11 384.fh f44.th_iitt_ia
eva02 _nlk, frtt__aatch___aa_ m3m jn22k A inlk, volo d2 volo_d2_224.snn (0_222,sail_inl inlk, regnety_ 160.swg le, inlk, wttpp_p2___p___ee___ 224 Izionnoonh_____ ft_nn__k__inllllee_ if f__mixnet_xinlle repvit_ml _idiiss___ii____________ Bals___00_i_illl convnextv2 lemge.eemae ft inlk,, dpai07.mmx_nlll inlk, resnet101.v2_jn1k res2next50.in1k.
vit_ hase patch32_elii _eppsp___p_p__1__
resmip_ 12_ fe__24.fb_iisiilledd.e..... bisdddiilled_iill inlk,
xcit_ adi__medium__ @endium_24_ppe 2ol6__84.fb__ae inlk,inkk, regnety 080_ivy_00_vvv222_1
ception inlk_ dpn68.mx__nnlk,
efficientnet piiiieettet_eesrajniii jsll,,nkk, mnasnet_ mmasset_100.mmsppit
base sit_base__acchh__eee wwwpppattthhllpppp____ wwwpp//3...oomaafttto.o
visformer visformer_tiny.ine inl,, annii dllk0,inlk, segnetz_32.m33jiilt semet154,gluonjinkt efficientmetv2_vv Bwtpeett__w__ae2______ jnlk,, focalnet_smal
if_efficiemmett_Blann
pexivit_small.00ee.nntt
inlk,
seresnet33ts.aa__nnl fbnete_ hatte__000mmpppnllll resmetl8diaa_innt jnlk, resnet34.a3 jnlk, Mla6O__einlk,, efficientnet__ bi___l_________ bl_pruned inlk,
resnet50s.glon gluo.,igguo resmet101 100000222mmm inlk, regnety_040.n33 jjnl convmixer_ 1et___1333snnlleeeeee (obboy__00__vvv2_alk. jalk, reamet122al_iat
iixk geresnext26sschu.nn inlk, bat boss__ptccll__eipp 224.op2222pppmii__f.. inlk, fastvit_ fastvit_ma6..pple map.__p....pplin..... vggl6,tv_nnk, girsmeccttts.ch.iilkk xeit 1itpy_____l1_22211111 inlk,
regnety_o0N,
segett_008.pycl...
resmip__66 36. 66_222..bb 224.fb_distilled _dsttille_iink inlk, regnetz_040) jnak_inkk, inception_nes next hase hase saiessiilink,, inlk, dm din_nfret_ ffddee_f..ddm____________ f0ddmmmmmiim.. jnlk, resmet50.d inlk,
efficientnet h2__hh2pp______ inlk, resnet18.ty_inl inlk, rexnet_ 150me_150.ma_inlk
larreet___rge__pppp__ Latpcdlp_laion___aoee soulib_oou_f_fae ft_nnnti_____a__ jiiikk_ii___20 ghost-
netv2_ 160inkk,
samall_ptcch6__15 38..3.gggggggi2ll..... in2lk___inlk,, convnext wwwagellaagaeanaaeeee jtt_iknnn________ faik____ikk_____ mobilenetv Baaaa_s___________ 075.labh jnlk, regnetz_d8 d8_ evos.ch_jnil
dm ff.nee_fBddd_iiiei
repveg_ jnlk, convoex hirge__lppelp__laee Jaion2b augreg_ pogrog_tt_int ft_in_l__3__,,,,
dpa68bmmx mx_ min_nik resnext101 1nxxxtl0_33xxdffbbb wsl_iglb__t_inn deit3 dcic3_large_ tirge_psrchl__eee ___.3844fb_int jnlk, convformer_ste sai. in in1k_384, repghost-
not_ net_058int inlk fastvit sanrit_s33.appleee. dist dist_inlk,, inlk, resnext50_32xxe 32x4d.22__nni jnlk,, regnetx, repnetx_040.pyott pyolsssss_ wit_haee_ft 150_s16_3844orf
inlk, ecaresnet50r.a2, inlk,
vir hase wtt_haaesschhhhhheeeeeee 16chh16_elip__1 1iiiip224.lniioe
ft fn_ink,, deir3 base hawpbbauseehhehheeeeeeeee 16_ 388.383_4fll
if [f_efficietnetv2_siin
resnetrs50.t_iil1 netx netx_004 000.pycls_innk... mnssuet lamb inlk, aopvg__ inlk, reametv2_50x 2_x0___00________ bit wooog_n22lk_faaeoo inik $24__s224222.1 inlk, regnety_ 004..v2_iiikkk convnext bas fase.fb ib__nn2k__fi__ 381. convaext_tiny.bb 1nyydf_j222k_FF inlk, convnext_tiny.ine ftn.inn22___iikk_B55 384, eca, www.haooexx26sscl_alllll. jinkk, resnet18 gaaiiiggloonnliii inlk, fastvic_=12.pple, jisk, deit base cacee_pachl__222_11
gmiker 24_ee_22_____ 2aaxxrr_4___2.ra_ 224.222 inlk, resnetma50d.ww fal2k_ftinle
treane inlk, resnest10leinkk, reg-
jnlk, lime_w18..n aug inlk, convnext small.b fa_ulllb_inkl, inlk, convformer x36. in1k_384, pit distillee 2aauilledd222.iniil inlk, resnei50..v2_im inlk, nest_ posl_small_js jx.gooy_inll inlk, resmlp_36. 36 $6__224fb_im jnlk, himet smsll gww.l inlk, vit hu__hse__pud puich16 _8. augreg inlk, resnet50.ft. fo_ sws_f_ss____________s_ sww__igls_bsssssseeeo ft inlk, poolforerr_m3d m36.sail_nnk inlk if mobillooett_smmeeee int, regnety_040 pyela inlk, goresner33ts ra2, inlk, resnext50_
resnel33ts.ra2_img inlk, seresnext10], 64x4d
darknetanS3.c2ns. inlk, formerv2 sail inlk, ecn je__resec3tsmm__ime jnlk, repvit_ @opi_m2__5 _indimi__ddi_______ _e___ddst___eee
resnet101s g.w.llooss.goooooo.o
poolformerv2_si st2i2sss22lssiill
ssl_yfcccl00mmmmmmmmmmmmm innk,, pool- of re__renntt0.m._.nn
convnext_picn paccopppccooosllllo dllooolsiiiiin poddd_innk inlk, caformer_s36, wat_:/666___alllll inlk, regnetz_ ranezt__40.ra__aa jnik,, vit_ small_ r26 (2____26_________ (0__x2222200 a2g_e24_ in21k ft f_inik, resnex220s.m2 inlk, mixnet pl. inlk,
deit_ basebbasseeeeeeeee wwtppppachhhhpppee 384.fb jnlk, repvit_ml 450est_400e_in1k
convmixer_1022 1024__20_002______ 21224_20_____ppp0 ps9_p14.
regnoty_06e Sepoyy_044.pyee pyl..........ikkk
23

---

resnet34.gluon rex2net101_26w. 26w__4.iillk Rlnee_...22_nnkk resmet34d.22 inlk, convnextv2 nem..eemeen.mee.eem....moa __in22k__inn 384,, hww.pppppu..se.ilkk..k 1es.et2_111.al___a jalk,, ncii_ _ammo_12___ 22_..8224.0bb______.. dis__iik 24__sm 22w.ab_diissiilllllb resnet50.b2k,
deit3 deit3_small __s_smll____ee _aana___ml_patchll____ f8..h1 inlk, handoorenas samiil_grent green_nnkk, con lite miniánlk, resneri52.vv inlk, lensenethlur121de in y2ggg/on_innkee inlk, vit hase patc_6 _ in4.kr At inlk, xcit scit_small_ small__2__8_____________________ 224.2222hh distt_ inlk, convformer_mae m36. inlk, wiee_wi8__ small _a___salll_l_luoooono
xcit_ scit_nnno__ 12_________________ p_______..f 144.fh_ditt_inn jilk, resnet34.al inlk, convnext atto alka2_inlk, resnet14tc3, inikk, swins_pepvi__lat bang_pppl_.arg..ikkk large.inkk, vit_small_ patchll__hhh___________ 204.222..00 224.angrog_.ue anlk con- mobilevitv2_ wtt.o/_s_s_vvossss_____e in22k j01k_3344 deit3_ aaudiiu_patchh6....m f22.kb_ inlk, ghostnetv2_ 100,i ww3.aile inlk,
resnst26d.glooo gloooonikk,, mobilenevv_smmall 100.ambbinlkk, officiiettne_ iiriiaatnccb___uunnnniiii
mmext_tiny.fb_it _ink, resnct50d.33
scresnexti01 aresncr101_3xxxdggoooo. _allk
hardcorenas baniiillllo___iiiin groen_inlk,
Aiformer_m3..sai__iigg in__2k2222___ inlk,,
ecaresnet50d, ecaresaet50l_ppueedddee iilk,
caformer 3.6mmerr_36..vil... in22k_ _iikk_3384
deit deii_inyy_aalhhle_22 pllpppccl___2aaa._e 224.id__22.bbbkkk
regnety_ Seeeey_s20.eeerr____ inikk
edgenext saatoos__saallssiii inlk, resmlp_bigg_22 24,,_2222222______2 iiclb__n22kk_uu iilkk,
1tt___0000oooiille ft regnety_
regnety_160.sw. 1e__1660.wwwilll Bwtpp/_i_______ inlk,
___eflicientnet bl.ap_in_l.....
res2net50_4Mo 48__2sinkk. coat coat_lite_ tineiiin,
ecu_botnexi28ss (ateeciiiss__6..1___o. 2266__226661ll cl_ikk,, resnetv2_101s [olxl_bin bit.gooog_on2222 in2kk ft frjnlk, ex3darknet focus, mw.2nn inlk, sergg_blg4nn big4. inlk, repugg
xcit_ small 24_ p8_ ff_22.fb__ink,,
cmosvit 9 2__aageer_400p. 240.inkk
convnext large_ lareeet__argemmpp..hie wwp.celip_laion2b.b... laion2b Iaion2b_uugreg._ sugreg_ f__nlk,
xcit nat_n p2_pl6__2200 fel6__22.bb__eee dist, inlk,
ridderessett0, _n1k,
vit ht_base__pue ju__hus__utchh6___ patch16__cipp.p. 224.2penai ftinniik_______ jnlk,,,
skresnet34.mjjnn inlk,
vgg19_ bn.ty bn.tyinkkk, repghostnet_ 100.i regnetv_ Saretv__64.raa_imng innkk, mohilenetv2_ (altieeett__1Omm_miiii convnext_ femto.dl_inlk, resne1266.m2, inlk, regnety 040.ra3 jnlk, skrespeti8.r_jnlt jnlk, caforner_m36 sttp://osssiiiiinf_te_eaaoi
vit bi_basc_pee _t__ass_petchh2____ patp__a__ah____pe__p__ 384.augreg__ni
reg-
netz b16.133 jalk, hmet hmot_w48__st psdddssd..dddle_ec. _nnk, remnst500_44xx400iinlk
xait_xx536_ 224.fb_dist__nnn 2ggnet__066.t2_jnng 016.12
seit_small_24 Ba_pal________________ fo________.f_________3111_______ fe____84_____s___________a______ dilk vi_tiny_________
cangee_in2ll_ffiinll mom_mmnnintl sei__small_2__p6__22
dist_inkk, reg.
caformer_ (03.mmmr_336.sil...... 33....66.____22222 _innk_
poolformer_<44 (attameres_ssssaiiiiiia
regnets_
netz dwt.r/a...nnll... jil_ resnet50d jnlk, seprit_ml.dis__ing oea_afnee_ inlk, resmet500.gluon jnlk, seresmext50 scremeet550.2.wwrrrmmm.. s2nkdrammmiikkk
vit.maall patch1l_______1 Baachh6_384.uureeee 384.auregg_inlk inn..iig xci__nmmo_22 p8__2_2____...b_______________ jnlk,, crossvit ba0e__40.ilk,, resnet50d.ll_in _inlk, con- [ewneett__lngge.femmeeeee
vformer s36.sail in22k_ ft_in_nnllll inlk, convnextv2 large.eemee ft m__in2k__nnk, resnet50.ry_ing jill,, resnet50.cl_nit pi distilled i2k,n 22iiilli222m.ill2222kkkkk w:2.t_ inlk, vit ht_hhase_pale hatce_ 224 mill.in2kk ft inin,,, extsndarknet, x.c2ns inlk,
efficientnet_ bl.ft bl.ft_nnkk, ef__effciietteteelliie
dpm688.ra inlk, If [[_efficientnetv2 blilk, Sypelt__044.ycls__hii inlk, mo_iliilee Jargeeeeeee_eeeeeeeeeeee monge_miimml___ 100inIk, resmetrs 101.tf inlk, ese_ vo___oovee399hrm.. inlk, mixer_116, _1___116222222__2 go,_2224ooo gongoongglkkl ft inlk, Begghoostee__00.inntte repugg.
b2g4rggg appii_mi_Iddist 450__400ei__e inlk,
caformer_st8 ststeesssssssssllssse jn22k ft inlk, efficient- formerv2 sttprri__ss.mmpssss____oo _istk iesnest20e..nlk, levit_com_256.tb dist inlk, resnett8.fb_ fasrttff_sss__ffo00000 ftilk, baattyb inlk, resnet152cgluon inlk, dlal69 bwiink, crossitt 1igggg________uu_ in8. convoext ol oot.oddssssdd..... convnexi_laree large.bb_iill inlk, regnetx_ regnetx_U64.pyckk U64.pycls_iillllll.. inlk, (Sapit_ inlk, seresne1122..222 inlk, vgg19.v_inlk, "gell_baty_ing batt_ikkk, dm iun_nfnetr22 dm inlk, seres- nexcl0ld_32xke 32x8d.ah jnlk,, inception_next sasptiionnntttbaaeeseeeeeeee base 384, tembluaresne266.cl. inlk. reanet2__12x22 bitgoog in21k ft inlk, regnety regnety_n80.pycle p80.08800......mm inlk,
vit wtw.bbasseh___________ 2ttp:/2__222.aagg___2222222 herrog__i21k_eeeee fnikk, if f__mobilenetv3_ng lar______or___..iiik 22t.__24iinIk,
wride_resnettln 2tv___inik
convmixer_788_322 32.in mtt_baee_pacch2_2224.122 22a.222_222.nggg_iilla
fastvit, fastvit_maa6..nn appll dist dist_ink,,
resnet50.fh, _ssssssl_ess 1fc inlk, if mohilenetv small inlk regnetz_el6.ra_ji jnlk, edgenext_xx, inl cmosvit 9_ i_240.inlk, scit_tiny 24 p8 p8_224.n 22e.224fh_hh inlk, regnety_080.ra3 080.ra3 inlk, efficientvir_ll.eee b1.r256_in11 tinynet iinlk, caformer_ b36.sall in1k_384, puL_2__b2,nlk, b2ilk, resme126d. bt inlk, con- vmext_ picoddl _inlk, pic_b__2._____p 224,2kkkk,, convnextv2_pico.oooo peco.eeeeeeeeeee_eeee f_innk, fhnetv3 jnlk flexivit_
regnety_006
resmet g/onon gluon inlk, regnetx_080, p8c.sscs_.ppp..kkk convncxt_ base tf ff_efficienteet_emml emmiinkk vit_ patchl (iich166224400 22..a2greg__illl convit convit_tiny.h fb inlk, ff.dnee__ffddm___________ inlk, resnet50.a3_inl jnlk,, convnext_atto. dttoo.d__innk, efficientnet_d _l__ pl_pruned in Inl resnext50_ 32x4d.22 fitpf___ssssgbbeeeeeee ftikk, dm
volo_d2, d2_ 384.s8iiiiiill innkk, resnet101_32x4d.f 32x4d.ff_ss 1f0_yfc1000__fe inikk repvit_mo_9 0ppvit_m__..dit_____ 9.dist_300e_ini
regnety_ 120.ss/w_wwwww..ooo.o inl2k inlk, ineep-
base_ hat___paahhh______1__116 SWM.i22kk_Lin2kkk.
mobilenev3 Batpe/s__aaa____aiiii 1a0e_000.mmiiiii1 in2li___ii___iila
[__ffficientee__bo. aa
gesnett01.gluonninke ma092.mx_inlk. resnet101 all jnlk, selecsts60.in1k, bas_bbas__pplll 224 224.in22k__ ininkk convnextv2 (in. inlk, regnetx__0n8.pee p0e.s inlk, resne150 inlk, res2net50 26w inlk,, sequencer2d millk, vit hes_mmmdd_mahhhheee 25p_256.sw in A jiinkk, regnetx
jio__nextsmallle saililsailjilkk deit base bis_bass_sstsledddeeeee patchleed_ 22uchl6_224fb_jill 075.ra2 inlk, 1ci__tia_11_____22_2__ 12_____2_____22_bbb__
101d,iik,, vit hil_arggepatchh6,___ patch16_ patch16_184.aurrgg 38...gg..gg....... jitjjnnlkk___ii pvt_y2 biink, regnetx_006 p0cx inlk, xcit, xcit_tiny_2 24________4_________ _ly_22_p11__________________ 21__1_222.20_ 224.bb inlk, pvt_22_bb i..ilkkk,, mamexx50_32xdd.raamm
tesnest44dglonnnnlll inlk, eaformer_m3366 m36, (ilossiliinkk___0 jnlk_ resnet50glummn.in gllk 52s.hll52 inlk, flexivit_ large jnlk, resnetv2_50xx s0xw2_50d___e 1__5000xiiiiip._o bit. gist_listilled jnlk, resmpp_ 22_22222_____n__ deit3 deit3_large_ parge_ 224.fbb__2__b___ filkkk, seres-
nexr50 11,50033xdddppp0 1200_32xd.ggoooonn gluon_inkk inlk, densenet1 121 ty jtnikk resnet|52 jnlk, gbostnet_ in Ik, of www.ffiietteettb22... b2lapinnkk, regnetx 002, 20els pyels_inkks
24

---

K
Additional Results
BatchNorm
Corv
Grouphorm
Noreroorm
Linear
Batchormm
Corv
Grouptorm
Layerto
Linwer
0.01
0.001
Fig. 18: Differences Between Layer Types are Significant. We analyze and test for statistical significances in the differences in MIS between different layer types (see Fig. [ The reported significance levels computed using Conover's test over the per-model and per-layer-type means with Holm's correction for multiple comparisons.
0.91
0.89
Average MIS
0.87
0.85
0.81
0.79
0.77
100
200
300
400
500
Input
Fig. 19: Influence of Input Resolution of MIS. We show the average MIS per model as a function of the model's input resolution. No trend is apparent; models with the same resolution yield different interpretability levels.
Fig. 20: Change of Interpretability per Layer During Training. Detailed version of Fig.
25

---

0.80
Average MIS (Hard)
0.75
Denseetct-201
ConsteXTa
Chi Redhiet
0.70
VIT-332
0.65
Googl Net
0.60
VIT-32
Resiters00
WideRestat-
Rolsist
ResNer-Sa
0.55
0.50
0.45
0.40
100
200
300
400
500
60D
700
800
(Sorted Average Score)
Fig. 21: Comparison of the Average Per-unit MIS for Models for a Different Task Difficulty. Our proposed MIS easily be extended to test than just the extrema of the activation distribution: Instead of choosing the most extremely activating samples as query images, we can sample less strongly activating ones from other parts of the activation distribution. By sampling from the 2nd/98th percentile, we can recompute Fig. on a more challenging version of the underlying 2-AFC task.
WidetesNer-Sa
Resteet- 50
5th Percentile Perunit
VIT.832
Googl ettet
0.7
CriP Roskier .3
NTT-832
Denseejat-200 2011
Robuust ResRee-- 50
0.6
ConuNexr-
100
200
300
400
500
600
700
800
Model (Sorted Score)
Fig. 22: Comparison of the Minimum of the Per-unit MIS for Models. While the mean of the per-unit interpretability varies in rather narrow value range (see Fig, B we investigate differences the distribution scores. Specifically, we are interested in the effective width of the distribution, i.e. how low does the minimal MIS per model go? To make the analysis robust against outliers, we do use the minimum but instead the 5th percentile. Note that this corresponds the lower end of Compared to the average MIS, we see higher variability across models.
the shaded area in Fig. 3
A Conv/Linear
BatchNorm 0.35 Pe
LayerNorm
GroupNorm
Conv/Linear
BatchNorm
P = 0.43
P. 0.40.45 0.45
P: 0.003
0.10 Pg
D, - 0.24
Pp 0.38
Pp 0.39
Pp 0.08
P;
0.91
0.90
Average Per-unit
0.88
0.85
0.85
0.80-
0.82
0.79
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Relative Layer Depth
Relative Layer Width
Fig. 23: (A) Deeper Layers are More Interpretable. Average MIS per layer as a function of the relative depth of the layer within the network, grouped by layer types. For each type, the values grouped into 30 bins of equal count based on the relative depth. The markers shown correspond the bin average, the shaded areas indicate the standard deviation. Correlations are computed for the ungrouped data points. While the standard deviation appears moderately high, note that the found trends are consistent
many bins of various layer types. (B) Wider Layers Interpretable. Average MIS per layer as a function of the relative width of the layer compared to all layers of the same type in the network, grouped by layer types. The values are grouped into 5 bins.
More
26

---

Gonviliner
Batenharm 40-0.80 Pe-
Convrimeer
Betchtiom
Conutaneer
Batctelom P.-
Al
A A0-0033
8.90 Ro
Py
Ap
Pp-m
0.00
0ge Pecandic of unit WS
00.00
Percentte ot Perurtt
0.76
Wedian Pe Be
0.74
0.87
0.99
0.72
0.04
0.85
0.93
Relative Midth
Refactvee Wiltth
Witth
Fig. 24: Wider Layers are More Interpretable. Wider layers within network are moderately interpretable based on the computed MIS. This trend holds for both the per-layer-average (see Fig. ( as well as the 5th percentile, median, or 95th percentile of the per-layer distribution as shown here from left to right. This suggests that the overall distribution shifted higher MIS values for wider layers, compared to just few outliers that positively influence the average value.
Tab. 24: Pareto-optimal Models for Optimizing ImageNet Accuracy and MIS. As Fig. 4 an anticorrelation between ImageNet top-1 accuracy and MIS, we here list the Pareto-optimal models for optimizing both accuracy and MIS at the same time.
shows
Model
ImageNet top-1 1 Accuracy [%]
MIS
GoogLeNet
69.15
0.908
timm:resnet34.33. _nnkkkk
72.97
0.904
timm:resnet50_gnm aw..hhgggghhhhhh.. inlk
81.22
0.901
timm:ccaresnet10ld pruned.miil inlk
82.00
0.985
10mm=evw02_smmlll_pahhh44_3. ________mmmm__222k_____ in22k 112222ff_____ inlk
85.72
0.890
timm:vit wttp:wit_baseepacchhh_oc 22_.2222..2 2tt.2222ree_inn2222
85.8
0.871
timm:caformer_b366 11$6.sail_in!! in1k_384
86.41
0.870
timm:caformer.
1atp:oommmer336..aaa s3i. in22k ft jinkk__3 384
86.86
0,870
tim::caformer_ b36.sai in22k ft in_inlk3300
88.06
0.864
timm:beit22 largeetphh ltt__22.iiikkfiiiiii inlk
88.39
0.39
After First Epoch
After Last Epoch
Least Activating
Most Activating
Least Activating
Most Activating
layerl 2__4
layer2_2 conv2233
layer4 2 conv3:73
layer4 2 comx3:859
MiR
layer4 conv3:200
Fig. 25: How do Dataset Exemplars for Units with Strong MIS Drop Change? To gain better understanding of why the MIS of a ResNet50 drops during training after the first epoch, we display the least/most activating dataset exemplars of four units from the model after the first (left) and after the last (right) epoch. While the explanations after the first epoch seem to focus on casy-to- to-grasp visual features, the units the right react to less clear-cut concepts. The units are among the units with the strongest MIS drop the convolutional layers with the strongest MIS drop.
27

---

(a) Clip ViT-B32, resblocks mlp e_proj unit 573
(b) DenseNet201, block3 layer10 norml, unit 138
DenseNer201, block3 Bacck3_layr299_cmmm 1onee,, unit 39
(d) DenseNet201, block3 B_e3___lee3_______ norm2, unit 123
Fig. 26: Visualization of Units for which MIS overestimates HIS. To showcase the shortcomings of the MIS, we visualize four units for which the MIS predicts an interpretability that is higher than the measured HIS in Fig. 2 See Fig. [7 for the opposite direction. For each unit, we show the 20 most (right) and least (left) activating dataset exemplars.
28

---

Works
Rootoone
(a) Clip ResNct-50, layer4_| 1 _cnv22, unit 430
(b) DenseNer201, block3 las___ller______1 unit 21
(c) DenseNet201, block3 Jayer48 plock3_layer48
norml, unit 1369
(d) VIT-B32, block0 norm2, unit 358
Fig. 27: Visualization of Units for which MIS underestimates HIS. To showcase the shortcomings of the MIS, we visualize four units for which the MIS predicts an interpretability that is lower than the measured HIS in Fig. [B See Fig. 26 for the opposite direction. For each unit, we show the 20 most (right) and 20 least (left) activating dataset exemplars.
(a) RcsML blocks Saccs__10_uue _ine_arrtokens,,,,______________ unit 61
(b) ResMLP-24, blocks. O 0_mlp_eee haap_channc_seeeee fcl, unit 110
(C) GMixer-24, blocks 5 mlp tokens fel, unit 166 Fig. 28: Visualization of Hard Units from Models with High Variability. For the four models with the highest variability in MIS (see 4B). visualize one of the units with the lowest MIS each. For each unit, we show the 20 most (right) and 20 least (left) activating dataset exemplars.
(d) ResMLP-12. blocks. 7 linear tokens, unit 127
29

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The contributions claimed in the abstract and introduction are backed up by experimental results in Sec.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We mention the limitations of our work throughout the paper, e.g., when we introduce it in Sec. 3r in Sec. as well in the appendix Appx.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
30

---

Answer: [NA]
Justification: This paper presents no theoretical results but only empirical findings. Thus, this question does not apply.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: A detailed description of how our proposed metric computed is given
in Sec. 3 The conducted experiments are described in the first paragraph of each subsection
in Sec. The experimental settings are stated in Sec. Appx. A.2 and Appx. B
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
31

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We grant open access to this paper's experimental code. It is shared, along with its documentation, in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://niis.cc.
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
7 //nips. //nips.cc/pub.e. wttpppwwccc..uide////c httpi///guidees/ddduuuiiiisiss http///eeeumiiiiiooooooooooo
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The experimental settings are stated in Sec. 3 Appx. A.2 and Appx. B Furthermore, specific experiments are always described in the first paragraph of each subsection in Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We visualize the uncertainty of our experimental results with error bars, unless this severely degrades the accessibility of figure due to cluttering (c.g., (c.g...igg.1 23 unless stated otherwise, the error bars shown in this paper depict the difference between the % and 95 % percentile of the per-unit distribution.
Guidelines:
The answer NA means that the paper does not include experiments.
32

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We explain the computational complexity our proposed method and the resources required for reproducing our experiments in Appx. G
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We read the Code of Ethics and ensured work follows its guiding principles.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiciio))
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: outline potential positive impacts of our work in Sec. 5 and potential negative impact in Appx. H
33

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considcrations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning conrectly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This study presents analysis tool and no new dataset or powerful model. Therefore, this question does not apply.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: This study uses two datasets (ImageNet [40] and IMI (50) that are introduced and cited in Sec. and Sec. respectively.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
34

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [Yes]
Justification:
This paper introduces a new analysis tool/metric. Its implementation and
further experimental code are published, along with its documentation in the supplementary material.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel as part of their submissions structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: We describe the setup of the conducted psychophysical experiment in Appx. B where also describe the workers' compensation and show screenshots of the experiment.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
35

---

Our experiments did not represent any larger risk than normal computer use. For pure psychophysical experiments with non-offensive stimuli, a choice task, and mouse clicks, we did not consider sending request to our IRB. Participants were informed that they consent their anonymized data being used for scientific study before agreeing to participate.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
36


Reviews:
[{"review_id": "7VTaVfF0vZ", "reviewer": "Reviewer_8Jb5", "summary": "The paper suggests a new automatic measure to asses how interpretable individual units inside vision models are (called MIS). The per-unit metric assesses the similarity of two query images (one should maximize unit activation and one minimize it) to two groups of representative exemplars (top-activating and least-activating images for that unit) using LPIPS. Repeating this for several query image pairs measures how well top/least activating images are consistent with themselves. If MIS score of a unit is high, that means all visual exemplars represent very similar visual concepts (ie unit is monosemantic) and therefore the unit is highly interpretable. If MIS score approaches chance (0.5) the unit is likely less interpretable as visual concepts of visual explanations are broad (ie polysemantic). The metric is shown to be well correlated with human assessment of the same task. Authors show several uses of the metric in several tasks such as assessing the interpretability of a wide range of models, units of different depths and widths along deep nets, training dynamics, and correlation of model interpretability and performance.", "questions": "* Please explain how the scalability of the measure enables to reveal phenomena that were unknown before (e.g. in [50])\n* Can further analysis of unexpected phenomena as described above be done?\n- Why is the SAE experiment performed on highly interpretable layers? Why not perform it on lower interpretable layers?\n- What are the insights for other vision models (not necessarily trained for classification)?\n- Can MIS be applied to other types of explanations (e.g. textual descriptions like Milan, clip-dissect, and MAIA)?\n- Can MIS be extended to testing the full distribution of unit activations, not only extrema?\n- Please explain the key differences and what is shared with the automated evaluation protocol of MAIA [3].\n- Is it true to assume that low variability in top activating exemplars also makes the visual explanation well-legible to users?", "limitations": "authors discuss the limitations of the method in the paper", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* Automating scientific processes is important. \n* The metric is built from reliable tools and well-known 2afc test formatting\n* The authors made efforts to exemplify the use of MIS in several large scale analysis", "weaknesses": "* Although the attempt to provide analysis of large-scale experiments, I do not find any of the conclusions of these very exciting and do not see the contribution of these to future studies: \\\na. Paper shows a study for a large number of networks, but the analysis is a bit insightful; The MIS is very similar across all the networks. The per-unit analysis is also not very surprising showing that shallow layers are not as interpretable as deeper layers. There is an interesting phenomenon at the very beginning and end of the network but the authors do not attempt to explain it.\\ \nb. Anticorrelation between the interpretability of units with classification performance was already shown in [50]. The larger scale experiment of the paper includes many types of networks, trained for different tasks, I wish authors would explore correlation to network performance for a broader set of tasks, not only classification.\\\nc. The training dynamic analysis is interesting, but again the interesting phenomenon of MIS the highest after solely 1 ephoc is not attempted to be explained, making it harder to consume the results.\nd. The only part of the analysis that felt more exciting was the SAE experiment, however, unfortunately, the authors chose to analyze layers with relatively high MIS scores. In that setting, SAE does not seem to improve the interoperability of units. Of course, the interesting experiment would be testing a layer with low MIS to see if MIS improves its interpretability.\n\n* The method relies heavily on existing tools like DreamSim, and well-known tests (eg 2afc setting from [50]). Therefore, there is no actual technical contribution in forming the metric itself. The correlation of MIS with the human scores is not surprising- authors use DreamSim in the exact setting it was trained for, with a high correlation with human judgment being the training objective.\n\n* Visual explanations are only one way to measure interpretability, the method does not suggest tackling any more advanced form of explanation, for example, those of [1],[2],[3].\n\n* An important aspect the MIS is missing, is how legible the visual explanations to end-users (which is at the end what we really care about when we want to measure \"interpretability\"). Is it true to assume that low variability in exemplars always implies visual explanation the is well-legible to users?\n\n* The method does not consider the full distribution of neuron activation bur rather the two extrema.\n\nIn general, I feel like the paper is an immediate extension of [50] (ie as if it was another subsection in it) and not a paper by it's own.\n\n\n[1] \"Natural language descriptions of deep visual features\" iclr 2022\\\n[2] \"CLIP-Dissect: Automatic Description of Neuron Representations in Deep Vision Networks\" iclr 2023\\\n[3] \"a Multimodal Automated Interpretability Agent\" icml 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 8Jb5,\\\nThank you for reviewing our paper. Please let us know whether our responses below addressed all of your questions or whether there are further questions we can answer so that you feel confident in increasing your score.\n\n**Q:** *“I do not find any of the conclusions very exciting”*\\\n**A:** We see our contribution as part of a wider community effort to develop better techniques for understanding the inner workings of neural networks. As such, the MIS is better understood as improving our “microscope” for analyzing networks, rather than providing any specific conclusion. However, as you noted, our improved “microscope” already surfaces a range of interesting phenomena that have yet to be explained. It’s such unexplained phenomena that fuel and guide scientific progress, highlighting how relevant MIS is to the study of networks.\n\nThe improvement MIS provides is quite a step-change: compared to the manual approach of prior work (e.g. [50]), we can now test several orders of magnitude more units & models. This is why we could detect the anticorrelation between a model’s accuracy and interpretability, a relation that [50] failed to substantiate due to a lack of statistical power. It’s these kinds of analysis and phenomena that require automated interpretability metrics and which have been sought after for some time (e.g. see Rev. gaJB or [6]). Now, for the first time, such large-scale interpretability analysis is possible with our MIS.\n\n**Q:** Relation to [50], *“technical contribution”*\\\n**A:** Before our paper, there was no work that allowed large-scale quantification of per-unit interpretability in vision models. With respect to [50], please note:\n(1) The manual approach of [50] (and other prior works) does not scale at all. Our method is the first to scale their type of evaluation up and we succeed by finding a simple but clever way to automate it. We hope you can agree that, ultimately, a new method should be mainly evaluated by its potential impact. This impact is visible, e.g. by the fact that our study finds a clear anticorrelation whereas [50] could only produce inconclusive results due to multiple orders of magnitude fewer measurements.\n(2) We’d also like to emphasize that our technical contribution is far from obvious. It’s not clear why a global perceptual metric (DreamSim) could fit human responses in our 2AFC task. In particular, one might expect (e.g. [7]) that humans solve the task by searching for common local patterns in the reference images and then comparing the most common ones to those of the query images. Hence, the fact that a global perceptual metric fits  human responses so well was quite surprising to us.\n\n**Q:** *“Differences to MAIA”*\\\n**A:** Thanks for asking this important question. Approaches such as MAIA and our MIS tackle completely different problems: MAIA is an automated explanation method, i.e. it tries to find explanations of what a unit does. But it does not allow quantifying *how interpretable* the unit is. On the contrary, our MIS is an *interpretability metric* that tells how interpretable a unit is given some explanations. Such an interpretability metric enables practitioners to increase the level of interpretability of a network, e.g., by model selection or hyperparameter tuning of the model or of an interpretability method.\n\n**Q:** Relation b/w *“diversity in exemplars”* and legibility\\\n**A:** Our MIS leverages an established 2AFC task used in multiple prior works [6, 49, 50]. We will integrate more information from these works on why this task measures how legible explanations are (i.e. how well they explain a unit). Their reasoning can be summarized as follows: The task requires humans to reason about positive (and negative) explanations, to identify the positive (or negative feature) shared by the explanations, and to recognize this feature in the correct query image. Please note that these features rarely correspond to completely unambiguous, clear-cut semantic classes, but identifying the common feature can be challenging. If the explanations are so diverse that humans fail to identify a shared feature, it shows that they cannot understand what the unit is firing for.\n\n**Q:** *“[Extension] to full distribution”*\\\n**A:** The MIS can very easily be extended to test more than just the extrema of the activation distribution: Instead of choosing the most extremely activating samples as query images, we can use less strongly activating ones and sample from other parts of the activation distribution. Please see Fig. 1 of the general response for a version of the paper’s Fig. 3 where we chose query images from the 98th/2nd percentile. As the human understanding (measured by HIS/MIS) even of the extrema is still limited and performance breaks down a lot when moving away from the extreme, we suggest using the MIS with images near the distribution’s tails (e.g. 98th or 99th percentile) to get a strong signal. Once models (let it be base models or SAEs) or explanations improve, it can be insightful to test larger parts of the distribution, too.\n\n**Q:** *“[What about] textual descriptions”*\\\n**A:** We are optimistic that the MIS can be generalized to textual descriptions too. With the recent progress in modeling language-vision models, it is conceivable to replace DreamSim with a language-vision encoder such that the MIS is based on the similarity between textual descriptions and query images. To keep our paper focused, and due to the lack of human interpretability annotations for such textual explanations, we decided to leave exploring this extension to future work. We will add this discussion to the final version of our paper.\n\n**Q:** *“SAE experiment performed on interpretable layers”*\\\n**A:** We chose a layer representing the median interpretability of GoogLeNet to neither make the experiment too hard nor too easy. We will re-run our experiments on a less interpretable model/layer and include it in the final version of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zCqPIkJVue", "reviewer": "Reviewer_P5jG", "summary": "The authors introduce a computational metric for interpretability. Their proposed metric is a computational version of an evaluation metric introduced in previous literature which measure human perceived interpretability. Crucially, they find that this metric correlates well, and since it does not require human studies, it can scale well, and therefore potentially enabling more insights in model interpretability.", "questions": "1. Perhaps a bit of a philosophical question, but how do the authors envision their metric in a computer-human interaction (CHI) context. As said before I am totally sold in terms of time/money saving for interpretability evaluation by avoiding human user experiments. However, I am strongly of the opinion that humans should still be somehow involved in terms of interpretability. After all, they are the ones affected/taking the final decisions. So can the authors expand/comment more on how a highly scalable method can interact with human decision making, which is by nature non-scalable? Let's say that you identified a model which is on average more interpretable than many others (Section 4.2.1), at this point how would a person interested in interpreting the model proceed? Would you show the units with the highest MIS? I think my general question would be: can MIS be used to help an user to fully comprehend their model, or would MIS be useful only for some pre-filtering? \n2. As said above, results seem compalling in common knowledge vision tasks. Do you expect HIS and MIS to have the same degree of correlation in, say, text classification tasks? \n3. If we stay in vision task, what about out-of-distribution (OOD) samples? I somehow would expect HIS and MIS to stop correlating for such samples? If so, this would diminish the contribution of this work.\n4. (L. 209) do you have a reference for the claim \"googlenet [..] is widely claimed to be more interpretable\"?\n5. Do you have an intuition of why would interpretability decrease after the first training epoch? I would rather imagine the, e.g., conv filters to adapt and resemble more the data\n6. Would be interesting to see a \"distributional study\" (similar to figure 4b) also for the width layer comparison. Why are wider layers more interpretable? is the MIS a \"real average behavior\" of units in the wider layers, or it's more because the wider the layer the higher the probability of having by change a unit that appears more interpretable?", "limitations": "Most limitations discussed in the conclusion", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- **Significance**: in time and cost terms, human studies (which are necessary for assessing interpretability) are the main bottleneck. This work is therefore significant as it proposes a way to bypass this.\n- **Results**: results seem overall interesting. I have some doubts here and there. see more below.\n- **Clarity**: paper is in general easy to follow", "weaknesses": "- **Experiments/Applicability**: results seem compelling for vision tasks in a domain that somehow falls in human common knowledge. I am not sure what would be the results in different domains and more complex/specialized vision tasks. More discussion below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer P5jG,\\\nThank you for your valuable feedback and for praising our paper as a *“significant work”* with *”interesting results”*. Please let us know whether our responses below addressed all of your questions or whether there are further questions we can answer so that you can confidently increase your score\n\n**Q:** *“how do the authors envision their metric in a computer-human interaction (CHI) context?“*\\\n**A:** That’s a great question! We completely agree with you that, at its core, interpretability is a human-centered field, and, thus, humans need to be involved eventually. Therefore, we ensured that the proposed MIS explains human interpretability annotations well. We see numerous opportunities for the MIS to interact with/support humans: (1) The MIS might be used to increase a model’s interpretability to simplify the human’s job, e.g., through model selection, hyperparameter tuning or directly optimizing for it (where the latter would benefit from a differentiable approximation of the MIS). (2) The MIS can save time when interpreting models: Instead of attempting to understand incomprehensible units, we can start with particularly easy ones (based on their MIS) and allocate our time accordingly. (3) Such an MIS ordering can also be helpful when finding neural circuits in the network as excluding very uninterpretable units from the search reduces the combinatorial complexity and, thus, computational cost. \n\n**Q:** *“Do you expect HIS and MIS to have the same degree of correlation in, say, text classification tasks?”*\\\n**A:** This is a very interesting question! Assuming access to a sufficiently human-like perceptual similarity function, we expect the MIS to generalize to various data modalities. Given the tremendous progress in language modeling/embedding, we are optimistic the MIS will work on text data, too. Testing this hypothesis requires extensive human psychophysical experiments. We will include this exciting experiment in the outlook paragraph of our paper and hope it will inspire future work.\n\n**Q:** *“what about out-of-distribution (OOD) samples?”*\\\n**A:** Thanks for asking this question. Our experimental results indicate that OOD samples cause no problems. In Appx. E, we demonstrate that the MIS is still highly correlated with the HIS when using synthetic feature visualizations instead of dataset examples as reference images. These images pose a substantial distribution shift as they look very different from natural images. Therefore, we conclude that the MIS works correctly also for OOD samples.\n\n**Q:** *“reference for the claim \"googlenet [..] is widely claimed to be more interpretable\"?”*\\\n**A:** This statement refers to the fact that many interpretability papers focus on this network, producing more and more insights into how it operates. We see now that our statement was imprecise and will revise it accordingly.  \n\n**Q:** *“why would interpretability decrease after the first training epoch?”*\\\n**A:** We have no definite answer to this question yet but hypothesize the following: This could be a sign of learning dynamics and the order in which features are learned. After initialization, the network can improve the fastest by learning very simple feature detectors (e.g., colors, simple geometric shapes), as those are weakly correlated with certain classes (e.g., blue colors increase the chance of seeing a fish). Those features are easy for humans to understand. Throughout the training, these feature detectors are replaced with more complex ones that are harder to decode. As suggested by reviewer gaJB, in later training epochs, the network might also tend to a state of stronger superposition to increase classification performance with the cost of decreased interpretability. In Fig. 3 of the general response, we show visual explanations of units with strong MIS drop between the second and last training epoch. We will use the increased page limit of the camera-ready version to discuss this hypothesis in the main text and include these visualizations in the appendix.\n\n**Q:** *“Why are wider layers more interpretable?”, “distributional study for the width”*\\\n**A:** Our data shows a moderate increase of the per-layer interpretability (i.e., per-unit scores averaged per layer) with increasing layer width. We see the same trend when instead of plotting the per-layer-average we plot the 5th or 95th percentile per-layer (see Fig. 2 of the general response). This suggests that the overall MIS distribution moves to higher values with increasing layer width and the effect is not dominated by few outliers.\n\nOverall, we hope to have answered all of your questions. If you are satisfied with our responses, we would appreciate it if you would increase your score.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c2G0qmHnez", "reviewer": "Reviewer_j42j", "summary": "The paper presents a method to automate a per-unit (e.g. individual neuron, channel in a conv layer) interpretability score for vision models that was previously computed via an expensive human study [50]. They demonstrate that their automated scores are highly correlated with the human measures, and then apply their method on 835 vision models, obtaining a ranking of models by 'interpretability' that is consistent with the much smaller subset considered in [50]. Additional analyses using their method involve inspecting 'interpretability' vs. depth, vs. layer width, vs. layer type, and throughout training. The paper motivates their work by arguing that automating the interpretability score will enable optimizing for it, towards more 'interpretable' models.", "questions": "How would you envision optimizing for the proposed MIS score? What advantages would that yield?\n\nWhy do more accurate models have lower MIS? This makes the idea of optimizing for MIS a harder sell.\n\nIs there any evidence that nodes with higher MIS are easier to name? I'd expect there to be a positive correlation, but I'm not sure how strong it would be. Again, I'm trying to think of way in which MIS can be useful, e.g. in collaboration with other interpretability techniques.\n\nWhat was the correlation between the per-unit HIS and MIS scores for the new human study on the two models?", "limitations": "The primary limitation is that the underlying score this method proxies may not sufficiently align with notions of interpretability that can ultimately be useful. I do not see how the method or the findings of this paper can be operationalized towards more research going forward. I would encourage the authors to devise and convincingly present a way in which this method can ultimately be utilized towards more trustworthy, transparent, or reliable models.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper achieves the goal it sets out to achieve, namely that of automating the human interpretability score of [50].\n- The authors conduct extensive experiments, evaluating a tremendous number of models (835!). \n- The authors offer two forms of validating their main claim: one via correlating their score with the human interpretability scores from a prior study [50], and another via a second (new) human study conducted after developing their method.\n- A number of follow up experiments are considered. Some interesting behaviors observed, like a big jump in MIS (machine interpretability score) for some of the batchnorms during the first epoch of training, and the drop in MIS over the final deepest layers (fig 6a).\n- I am a big fan of the abstract problem: measuring interpretability is a hard problem, and an automatic method could potentially be very useful.", "weaknesses": "While the MIS seems to do well at modeling the HIS, I am struggling to see the merits of such a score, as I find the underlying task too easy and as such, limited in its downstream potential for applications. As I understand it, the psychophysics task used to proxy interpretability simply asks users (given a 'unit' that maps each image to a scalar activation) to match a least activating image to a set of other least activating images and a highest activating image to a set of other highest activating images. This can be done so long as there is any kind of distinctiveness between the highest and least activating images. It also does not account for the well-known superposition phenomenon, in which a single unit may encode multiple concepts, thus hindering its interpretability / ability to steer model behavior; in such a case, one could still pass the psychophysics test by recognizing any one of the multiple concepts encoded by a unit, as the least activating images would not contain those concepts. \n\n^to summarize, I don't think the underlying test that MIS can proxy gives us any valuable signal, as it does not tell us if a unit is aligned with a single concept. I could be wrong, but I find it vital for the authors to demonstrate an application of their interpretability score (e.g. combatting spurious correlations, steering model behavior, uncovering biases, etc). \n\nThe claim 'if we can measure it, we can optimize for it' is unsubstantiated and, in my opinion, misleading. I don't see a straightforward way nor reason to efficiently optimize for MIS.\n\nThe range of values that the MIS outputs empirically is quite tight, making it hard to place much significance on the observed differences. \n\nI find the methods section to be needlessly mathy; I think it obscures the underlying method, which is not too complicated (sort images along a unit, select your query+example sets, use dream sim to get similarity between each example and the queries, average, pick). \n\nThe paper relies very heavily on [50]. It would be nice to see how this method could be incorporated with other popular ideas in current interpretability literature.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer j42j, \\\nThank you for your detailed feedback. Please let us know whether our responses below addressed all of your questions or whether there are further questions we can answer so that you can confidently increase your score\n\n\n**Q:** “find the underlying task to easy”, “does not account for polysementacity”\\\n**A:**  You are right that the underlying psychophysical task checks for a rather rudimentary level of understanding. Note, however, that understanding units at this level is necessary for any deeper understanding. This is important because experimental results show that human understanding of units at this level is rather brittle: Zimmermann et al. (2023) showed for two models that by performing the psychophysical task for slightly less extreme query images, human performance drops strongly (see Fig. 6/7 of their paper). We can reproduce this finding with our MIS for many more models (see Fig. 1 of the general response), showing that this task is not trivial to solve. Further, the task partially accounts for polysemanticity as the most extremely activating images of polysemantic units might not correspond to a single but various concepts, making the task even harder to solve. In conclusion, this means there is still ample room for improvement in future models. \n\n**Q:** “application of interpretability score”, “(advantanges of) optimizing for MIS”\\\n**A:** Thank you for raising this question. We see two types of practical applications for our MIS that go beyond analyzing and understanding neural networks: (1) Optimizing neural networks directly to become more interpretable via gradient descent. (2) Performing model selection based on interpretability. (2) Tuning hyperparameters of other interpretability tools to make them explain networks better. While having a differentiable version of the MIS is required for (1) and would surely benefit (2) & (3), note that a non-differentiable metric still provides valuable insights enabling the latter two directions (although potentially less efficiently). As an example, we performed experiments with sparse auto-encoders (L231ff & Sec. I), revisited and investigated inconclusive results of previous papers, and performed hyperparameter selections. We use the increased page limit of the camera-ready version to highlight these results more in the main text.\n\n**Q:** “range of [MIS] values is tight”\\\n**A:** On a per-unit level, we find that the MIS spans the entire theoretical value range (~0.5 to 1.0) (see Fig. 2B/C). When averaged per model, the effective range indeed becomes tighter. We argue, however, that this is not an issue of the metric but instead shows that models only trained for good downstream performance all learn similar representations (https://arxiv.org/abs/2405.07987) with mediocre interpretability. With an increased interest in interpretable models, we expect future models to achieve higher MIS values. By choosing less extremely activating samples as query images (see Fig. 1 of general response), we can also increase the task’s difficulty to have a meaningful signal also for more interpretable models.\n\n**Q:** “[Incorporation] with other popular ideas in current interpretability literature?”\\\n**A:** A particularly popular topic at the moment is SAEs. As described above, our MIS can be used to make model selection and hyperparameter tuning of SAEs more efficient, enabling large-scale sweeps. Further, it is conceivable to use the MIS as a guiding signal when finding interpretable circuits in a network: by excluding particularly uninterpretable units from the search, one might reduce the computational cost of finding circuits. Finally, as the MIS also works with explanations other than dataset examples (see Appx. E), it can be used with future explanation methods, too (e.g., MACO [1]).\n\n**Q:** “Why do more accurate models have lower MIS?”\\\n**A:** This is an important question. We hypothesize that this is related to the phenomenon of superposition: With limited capacity, one way for models to obtain higher downstream performance is to represent features in superposition/entangle them. However, at the same time, this makes units harder to interpret as they don’t correspond to individual features anymore, explaining the lower MIS. We politely disagree with your assessment that this result makes it difficult to sell “optimizing for MIS”. First, note that we see only a correlation, which does not mean that there has to be a tradeoff between performance and interpretability, as this would assume a causal relation between these two variables. Second, note that if accuracy and MIS were positively correlated, then there would be no need to optimize for MIS as one would get this for free. On the contrary, our results show clearly that one should not hope to automatically get very interpretable models by only optimizing for high accuracy. Instead, we need to optimize for both accuracy and interpretability/MIS.\n\n**Q:** “[are] nodes with higher MIS are easier to name?”\\\n**A:** Interesting question! To verify the correctness of our MIS, we used the human psychophysical data of [50]. This data also contains scores indicating how confident humans were when making their choices. Regarding your question, one might say that if humans find it easier to name units, they will have high confidence in solving the 2AFC task. Interestingly, we find a high linear/rank correlation of 80% between this confidence score and our MIS. Thus, we conclude that nodes with higher MIS tend to be easier to name.\n\n**Q:** “correlation between the per-unit HIS and MIS scores for the new human study?”\\\n**A:** Based on your suggestion, we computed the correlation between MIS and HIS for the new human experimental data. Specifically, we compute the correlation for all units used for creating Fig. 2C and again find a high correlation ($\\rho_p = 0.85, \\rho_s = 0.81$).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QLLtG4xOPS", "reviewer": "Reviewer_gaJB", "summary": "The paper proposes a novel method for measuring per-unit (e.g. per-neuron) interpretability of vision models, which is based on a DreamSim-based automation of the 2-AFC task. The method, which is called the Machine Interpretability Score (MIS) is found to be highly correlated with human measures of interpretability, as well as capable of making novel predictions about what units humans will find more and less interpretable. The authors then use MIS to compute average per-unit interpretability for 835 computer vision models, and highlight their findingings (such as the negative correlation between accuracy and average per-unit interpretability, and the increased interpretability of deep layers) which would have been impossible to accomplish without the novel metric.", "questions": "* Here's something I don't quite understand: for a unit, why do its weakest-activating dataset samples seem to all be monosemantic? Wouldn't we expect them to be more random than that?\n* It's unclear to me why one might expect \"wider layers to be more interpretable\" when comparing across models—sure, this seems true for a fixed model architecture and training set, but among the models being analyzed, wouldn't the models with larger layers be more likely to be larger models trained on more data?\n* Is there a way to quantify how superposition affects MIS? Do you have any thoughts on what percentage of the decreasing MIS throughout training is explained by increased superposition in neurons?\n* Do you have thoughts on training autoencoders to directly optimize for a combination of MIS and reconstruction loss?\n\nMinor questions:\n\n* In Figure 3, the red points simply represent the location of the models tested by Zimmerman and not the HIS determined by Zimmerman for those models, correct?\n* What models form the pareto frontier in Figure 4A?\n* In what sense is the word \"define\" used on line 122?", "limitations": "The authors adequately address the limitations as they come up.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper is a triumph of the genre: it is written in an incredibly clear fashion, presents limitations as they arise, and has an excellent appendix which directly addresses most questions that came up as I was reading the paper.\n\n* Originality: the contribution solves a longstanding problem in per-unit interpretability in a novel and scalable way. It builds upon some of the best work in the field, and runs experiments that, as the paper mentions, would cost billions of dollars to complete using current approaches.\n* Quality: the paper is of outstanding quality. It is exceptionally clear, and the appendix contains a host of highly useful sensitivity analyses. I especially appreciate Appendices C (justification of the choice of DreamSim) and I (application to sparse autoencoders).\n* Clarity: It's very clear what the results are, and what the contribution is, and where the claims are supported. The authors do a good job of demonstrating that MIS explains existing data; then, that it makes novel predictions, which they support with a human experiment (!); and finally, have lots of detailed figures explaining their experiments which use MIS.\n* Significance: human-free per-unit interpretability has long been sought in vision model mechanistic interpretability, and this provides a possible solution.", "weaknesses": "No significant weaknesses.\n\n\n\n\n\nMinor nitpicks:\n\n* The way the query images are selected is only mentioned in the appendix; this might be worth quickly mentioning in the main paper (basically, the fact that they're also highly-activating images)\n* In Figure 14, unless I'm misunderstanding something, there's a strange difference in y-axis scaling. I reckon these should be aligned.\n* Line 195 has an extra full stop. (Or, if you were trying to evoke a sense of mystery, has one less than needed...)\n* Figure 17 says \"interpreability\"; should this be \"interpretability\"?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer gaJB, \\\nThank you for your valuable feedback and for praising our paper as a *“triumph of the genre”* with *”outstanding quality”* and finding it *”solve[s] a longstanding problem”*. Please let us know whether our responses below addressed all of your questions or whether there are further questions we can answer so that you feel even more confident in our work. \n\n**Q:** “The way the query images are selected is only mentioned in the appendix”\\\n**A:** Thank you for your suggestion. We will make use of the increased page limit in the camera-ready version of the paper to move this information to the main text.\n\n**Q:** “Why [are] weakest-activating samples [not] more random?”\\\n**A:** We agree with you that it is not obvious why the least activating samples are also clustered, and all contain the same visual feature. However, one can think of these features as “counterparts”/”anti-features” to the features displayed by the maximally activating samples: A unit only has high activation if just the feature but not the antifeature is present, which potentially allows the network to learn more specific feature detectors.\n\n**Q:** “Is there a way to quantify how superposition affects MIS?”\\\n**A:** That’s a great question and suggestion! While we can’t think of a precise quantification right now, we can offer a thought experiment. Polysemantic units respond (strongly) to multiple concepts. If those different features elicit similar activation ranges, the query and reference images used to compute the MIS differ, resulting in harder tasks and lower MIS. We think that quantifying through human studies how well a drop in MIS is correlated with units being polysemantic will be an interesting follow-up for our work. Thank you for the suggestion!\n\n**Q:** “why [might] one [...] expect \"wider layers to be more interpretable\"?”\\\n**A:** Thanks for raising this question! In this plot, we compare the relation of a layer's relative width and its interpretability, i.e., we ask whether wider layers of a network are more interpretable than narrower ones of the same network. We chose this relative comparison exactly to circumvent your concern. We think our observation might be explained by the superposition hypothesis: Narrow layers do not have sufficient capacity to represent different concepts individually through single units and instead have to leverage polysemantic units. We will expand on this connection in the camera-ready version of our paper.\n\n**Q:** “thoughts on training autoencoders to directly optimize for MIS and reconstruction loss?”\\\n**A:** We think that using our proposed MIS to increase the interpretability of networks is very exciting — both for auto-encoders used to make large networks interpretable and for directly making networks more interpretable. While the current computation of the MIS can be used for non-gradient-based optimization (e.g., hyperparameter grid search), optimizing it scales inefficiently with parameter count. A challenge for future work will be to find a differentiable approximation of the MIS that can be optimized using gradient descent, circumventing the efficiency issue. We believe such an approximation can be defined when training with sufficiently large batch sizes, and hope to explore this further in follow-up work. \n\n**Q:** “Do the red points in Fig. 3 represent the location of models tested by Zimmermann et al.?”\\\n**A:** Yes, that is correct. We will update the caption to ensure future readers do not mistake them for the results of Zimmermann et al.\n\n**Q:** “What models form the pareto frontier in Figure 4A?”\\\n**A:** Thank you for this suggestion! We determined the Pareto frontier of Fig. 4A using the paretoset python package and will include the following table in the camera-ready version of our paper:\n\n| Model                                            |   Acc [%] |   MIS |\n|:-------------------------------------------------|----------:|------:|\n| googlenet                                        |     69.15 | 0.908 |\n| timm:resnet34.a3_in1k                            |     72.97 | 0.904 |\n| timm:resnet50_gn.a1h_in1k                        |     81.22 | 0.901 |\n| timm:ecaresnet101d_pruned.miil_in1k              |     82    | 0.895 |\n| timm:eva02_small_patch14_336.mim_in22k_ft_in1k   |     85.72 | 0.890  |\n| timm:vit_base_patch8_224.augreg_in21k_ft_in1k    |     85.8  | 0.871 |\n| timm:caformer_b36.sail_in1k_384                  |     86.41 | 0.870  |\n| timm:caformer_s36.sail_in22k_ft_in1k_384         |     86.86 | 0.870  |\n| timm:caformer_b36.sail_in22k_ft_in1k_384         |     88.06 | 0.864 |\n| timm:beitv2_large_patch16_224.in1k_ft_in22k_in1k |     88.39 | 0.839 |\n\nWe found it interesting that although purely convolutional networks with high accuracy exist, all points on the Pareto frontier with high accuracy belong to transformer architectures.\n\n**Q:** “In what sense is the word \"define\" used in L122”\\\n**A:** We use the word define here in the same sense as one defines a statistical/machine learning model. Would you prefer the word “model” instead of “define” here?\n\n**Q:** “Figure 14 [...] difference in y-axis scaling”\\\n**A:** Thanks for the pointer, we will update this figure in the final version of the paper to use an aligned y-axis.\n\n**Q:** Typos in L195 and caption of Fig. 17\\\n**A:** We corrected these typos.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear reviewers,\
Thank you for your valuable feedback. We are delighted that you praise our paper as a *“triumph of the genre”* with *”outstanding quality”* (Rev. gaJB) and finding it’s results *”overall interesting”* (Rev. P5jG) and *“potentially very useful”* (Rev. j42j) for an *”important”* topic (Rev. 8Jb5).

Based on your feedback and questions, we implemented the following changes in our paper for the rebuttal:

- We explained and demonstrated how the MIS can be computed not just for the extreme of the activation distribution but the rest of the distribution, too (Rev. 8Jb5 & j42j) (Fig. 1 in attached PDF)
- We conducted a “distributional study” (Rev. P5jG) on the relation between layer width and interpretability (Fig. 2 in attached PDF)
- We computed the correlation between MIS and HIS on the new data collected for Fig. 2C. (Rev. j42j)
- We determined the Pareto frontier of models in terms of their accuracy-interpretability tradeoff (Rev. gaJB)
- We explored reasons for why the interpretability of a ResNet50 decreases during training (Rev. P5jG) by visualizing units with a particularly strong drop in MIS (Fig. 3 in attached PDF)

Please let us know whether our responses below addressed all of your questions or whether there are further questions we can answer so that you feel more confident in our work and can increase your score.

Meta-review: After a productive discussion period, the reviewers came to consensus that this paper was above threshold for acceptance. This approach holds great promise as a general purpose method for understanding large-scale model representations and strategies, and should prove impactful for AI practitioners trying to characterize the success and error modes of their models and cognitive scientists who want to compare models and biological brains. I recommend this paper for acceptance.
Decision: Accept (poster)