SlowGuess/NeurIPS_2023_OCR huggingface dataset:
A fundamental problem with using natural images to characterize the receptive field of individual units (apart from idiosyncrasies of the used dataset) is that visual features do not usually appear in isolation, resulting in ambiguity. For example, highly activating ImageNet-exemplars for a unit sensitive to feathers would probably depict birds, making it hard to isolate feathers as the crucial visual feature instead of beaks, claws, or a background of greenery or blue sky.
The promise of Feature Visualization is to circumvent these limitations by synthetically generating images that only contain visual features contributing to high unit activation. The procedure starts with an initial random noise image and performs gradient ascent on the activation achieved by this image at the unit of interest. Following established work [e.g. 4, 54], a unit is defined as one feature map of a convolutional layer, where the activation across the feature map is aggregated by calculating the mean, just like for natural stimuli. To prevent mode collapse of the generated batch of feature visualizations, i.e. to truthfully capture the receptive field of so-called polysemantic units that show sensitivity to multiple different concepts, a regularization term is added to the loss to diversify the images.
We build on an existing implementation [38] and extend it to support various models flexibly. Previous implementations had two critical hyperparameters: the number of gradient ascent steps to be performed and the weight used for the diversity term. As earlier work mainly focused on the GoogLeNet model, hyperparameters were tuned for it. We find, however, that these fixed values do
Due to technical issues, some workers participated more than once. However, we exclude their data in our analysis and recollect the missing data by recruiting new participants.
not generalize well to other models, but their optimal<sup>6</sup> values heavily depend, among other factors, on the model and location of the unit within the network — in extreme cases, the ideal value can even be different for two units of the same layer in the same network. Therefore, using any fixed value would introduce an unfair bias for or against some models. Furthermore, since a larger weight for the diversity term hinders the optimization, the number of necessary gradient ascent steps depends partially on the diversity weight, meaning these parameters cannot be set independently.
To overcome the latter problem of choosing an appropriate number of optimization steps, we implement an adaptive procedure that interrupts the optimization when the gradients become small. The procedure performs at least  $2^{\prime}500$  steps of gradient ascent and records a trajectory of the observed gradient magnitude. We smooth these trajectories with a large sliding window and halt optimization once the average gradient magnitude in the last window is larger than in the second-to-last window.
To solve the first problem, we determine the diversity weight for each unit individually as follows. We first record the maximum and minimum activation achieved by natural dataset samples for the unit. Then, we generate feature visualizations without diversity and assert that they achieved a stronger activation. We then try to find the largest possible diversity value that still produces images that achieve at least as strong activations as all dataset samples. To do so, we first perform an exponential search starting at a diversity of 1, increasing by a factor of 10 in each step. Once the value becomes too large, we perform 6 steps of binary search between the largest diversity value still known to work and the final value tested in the exponential search. If no value tested during the binary search worked, we return the lower bound of the search range, i.e. the images generated in the end are always guaranteed to be at least as activating as the strongest natural images. Generating one batch of Feature Visualizations, i.e., one step of the procedure, takes between two and 90 minutes on an Nvidia 2080Ti GPU, depending mostly on the width of the layer of the unit, since the diversity term scales quadratically. A qualitative comparison of feature visualizations generated for the different models considered in this work can be found in Fig. 9.
For ViTs, feature visualization could theoretically be performed using the same method by maximizing the activation at the position-wise feedforward layers. However, just applying the existing methodology does not lead to visually coherent images. Ghiasi et al. [16] present a method for adapting the procedure to ViTs that seems to produce intelligible images, but one step of their algorithm just adds large-scale noise to the visualizations, effectively performing a random search in image space to find activating images. Removing this augmentation or reducing the scale of the noise leads to unintelligible images again. In light of these issues, we chose not to evaluate ViTs in the synthetic condition.
Judged by the first authors.
Figure 9: Qualitative Comparison of Feature Visualizations. For each model, we randomly choose three units and display the maximally (left) and minimally (right) activating feature visualizations generated without the diversity regularizer.
A.5 A Priori Power Analysis
A central question for the experimental design of this study is how many units need to be sampled per model to obtain a result representative of the entire model. Answering this question is non-trivial as there might be large inter-unit interpretability differences within one model. Indeed, this is what we observe as displayed in Fig. 5). While the most naive approach would be to test all units, this is unfeasible due to the associated financial costs. Therefore, we need to find a trade-off between these considerations and keep the number of sampled units as low as possible while still getting representative results. Put differently: What is the lowest number of units one can select while still being reasonably sure that the found effect is statistically significant?
To answer this question, we first ran a pilot study where we controlled for inter-participant differences by showing stimuli from two models (GoogLeNet and Robust ResNet-50) to the same subjects. Participants in this pilot were the study's first authors and other lab members. This means that the obtained data is of high quality, and we can be confident that all participants understood the task. The mean difference in the proportion of correctly completed trials came out to be 0.1, with standard deviations of 0.15 for both interpretability methods, resulting in a relatively large effect size, with Cohen's  $d$  of 0.67. Irrespective of concerns of statistical significance, we deem an effect of this size to be practically relevant; in other words, if the difference in interpretability between two models would be at least 10 percentage points, we would consider this practically relevant. To determine the required number of sampled units at these effect sizes, we then performed an a-priori power analysis using the software G*Power [13] — a standard tool widely used in psychology and the social sciences. To avoid unrealistic assumptions about the shape of the distribution of measurements (the normality-assumption of the t-test will almost certainly not be met because the data points are proportions expected to lie between 0.5 and 1.0), we opted for the non-parametric Mann-Whitney-U test. We assumed an  $\alpha$ -level of 0.01 (subject to Bonferroni-correction to safely conduct up to five significance tests on the same data) and a  $\beta$ -level of 0.95. This analysis yields that at least 86 units are required.
However, the situation is further complicated by the fact that we are comparing values of which we cannot actually take a continuous measurement since we aggregate binary trials to estimate the proportion of correctly completed trials for each unit, i.e. there is measurement noise. This can be modeled as a Binomial distribution, characterized by the parameter  $p$ , the probability of answering correctly in any given trial for units of this model. This gives rise to the question of how many measurements we should take per unit to be able to assess an individual unit's interpretability with any confidence. Accepting a standard deviation of 0.1 in the estimate of each unit's  $p$  results in 30 independent trials per unit.
Another consideration is how many trials one participant can be asked to complete. Earlier work presented up to 24 trials to each participant under similar conditions [54]. Still, again we might be interested in accurately estimating the participant's performance, and each participant incurs some fixed cost for the time spent instructing them and completing the practice trials. On the other hand, MTurk HITs are typically very short. Constructing long tasks, e.g. of 100 trials or more, would increase the risk of participants losing focus or becoming frustrated and just answering randomly. We deemed 55 trials per participant (40 real trials, 10 instruction trials, and 5 catch trials) a suitable balance of these concerns.
Finally, the required number of participants is the total number of trials divided by the number of trials per participant. The total number of trials is, of course, the number of units times the number of necessary measurements per unit, resulting in  $86 \cdot 30 / 40$  trials. As this is not an integer, we opt for using 84 units instead, which brings the number of needed participants to 63.
B.1 Extended Visualizations of Results in Sec. 4
Figure 10: Well-interpretable units do not necessarily stay interpretable in harder tasks. For each unit investigated of the ResNet-50 (first row) and the Clip ResNet-50 (second row) model, we visualize the gap in human performance between the easy and medium (first two columns) and the easy and hard (last two columns) tasks. We show these gaps as functions of the relative layer position (first and third column) and of the human performance in the easy condition (second and fourth column).
Figure 11: Few models have significantly different interpretability scores. The differences in interpretability afforded by synthetic feature visualizations are mostly non-significant (NS) in a Conover test with Holm correction for multiple comparisons; see Fig. 3 for significance values for natural exemplars.
Figure 12: Measured interpretability using different methods is partially correlated. We investigate how the interpretability measured in our psychophysical experiment for the explanation method of natural dataset samples is predictive for that measured using synthetic feature visualizations. The table shows Spearman's rank correlation between the proportions correct when using natural and synthetic explanations. Asterisks denote significant correlations. While we see a strong correlation for some models, this does not hold for all.
(a) Natural exemplars.
(b) Synthetic feature visualizations.
(c) Distribution of confidence ratings.
Figure 13: More confident responses are mostly more correct. We investigate the relationship between the confidence indicated by the participants and the correctness of the given response. For this, we compare the proportion correct for responses with low (i.e.,  $= 1$ ) and high (i.e.,  $= 3$ ) confidence ratings for all models and both natural exemplars (a) and synthetic feature visualizations (b). For the natural exemplars (a), we find that for almost all models, a higher proportion of responses are correct when the associated confidence ratings are higher. For the synthetic condition (b), this only holds for two models, if at all. Additionally, the distribution of confidence ratings (c) shows that natural examples lead to higher confidence scores for all models.
Figure 14: Impact of unit sampling on performance. In Fig. 6, we investigate the effect of task difficulty on performance. Due to an oversight, not all 80 units sampled for this experiment were kept identical between the difficulty levels, but only 63. Here, we visualize the result for only those units that were shared between the difficulty levels. The inconsistency has no relevant qualitative effect on the conclusion: Performance rapidly declines as the task becomes harder.
B.2 Analysis of Quality Checks
(a) Distribution of decisions.
(b) Distribution of values used for decision.
Figure 15: Most participants pass quality checks. For each of the five quality checks outlined in Appx. A.3, we show a distribution over the number of participants that have passed/failed this check (top) and the distribution over the values used by the checks. The black and red lines in the latter indicate the minimally required and the maximally allowed values, respectively.
B.3 Distribution of Activations
Figure 16: Activation distribution is unimodal. We display the distribution of activation for 15 randomly chosen units from GoogLeNet. The activations have been divided by the largest absolute activation per unit to restrict the distribution to values between  $-1$  and  $1$ . The orange and red lines indicate the location of the 85th and 95th percentile as well as that of the 15th and 5th percentile, respectively. It is apparent that the distribution is unimodal and does not feature multiple pronounced peaks/modes at its tail.
B.4 Are Activation Patterns in Feature Maps Predictive of a Unit's Interpretability?
Since we observe large differences in unit-wise interpretability across all networks, a logical research direction is to find out what drives these differences. As an example, we investigate two hypotheses here.
Contrast. First, we investigate whether there is a relationship between a unit's interpretability and the local contrast in the activation maps of convolutional layers caused by validation set images. This is motivated by the idea that if a feature is concentrated at one location in the image, it might be easier to be detected by human observers than if the activation is distributed across the image.
We visualize the relationship between a unit's interpretability and the computed contrast in its activation maps in Fig. 17. There does not appear to be a strong relationship between the two, as supported by low Spearman's rank correlations  $(-0.24\leq \rho \leq 0.14)$ .
Sparseness. Second, we analyze whether the sparseness of activations in a feature map is predictive of a unit's interpretability. This is motivated by the argument that units that sparsely fire over a large dataset are sensitive to a particular image feature that might be easier for humans to detect and understand.
To test this, we investigate two measures of sparseness: First, we compute the fraction of non-positive values (i.e., zeros after ReLU activation) in a unit's feature map averaged over the ImageNet validation set. The resulting data and the units' interpretability scores are shown in Fig. 18. As for the contrast baseline, we see only a weak, non-significant relation between the two. Second, we compute the fraction of images in the ImageNet validation set for which an entire feature map achieves only non-positive values (i.e., zeros after ReLU activation). Analogously to before, the resulting data is shown in Fig. 19, and we find no strong relationship.
We expect the broader impacts of our work to be positive since advancements made with respect to the interpretability of AI systems should increase their transparency and fairness. However, as is always the case for interpretability work, explanations can also give users a false sense of trust in the explained model. This can lead to the deployment of models that, under real-world conditions, give incorrect or undesired results. Too much trust in AI systems can also lead to their deployment in areas
Figure 17: Local contrast of activation maps does not predict a unit's interpretability. We compute the average local contrast in the activation maps caused by validation set images for the sampled units of the investigated convolutional networks. The units' interpretability, measured by the proportion correct, does not appear to be a function of the local contrast.
that are better left in human hands for ethical reasons, such as policing or the justice system. Apart from these general and high-level concerns, we see no direct way in which someone could use the findings and data presented here to cause harm, especially since we do not build an interpretability method but investigate whether models are interpretable.
D Computational and Financial Cost
The most computationally intensive aspect of this work is creating stimuli for the experiments, which can be further subdivided into collecting natural exemplars and producing feature visualizations. The former point is negligible since all that is required is one forward pass over the ImageNet training set for each model. We record the activations on Nvidia 2080Ti GPUs and perform multiple forward passes due to memory constraints, but even if we assume a pessimistic 4 hours of GPU time and full utilization of the GPU at  $250\mathrm{W}$ , this results in  $9\mathrm{kWh}$  power consumption for all models in total. Creating feature visualizations for 100 randomly selected units — we later randomly sample 84 units for each model and kept some stimuli for anticipated later experiments — requires the parallel use of 25 2080Ti GPUs for about 12 hours for all models except ConvNeXt, which takes about 24 hours on average. Since this is done for only seven models because we do not generate feature visualizations for the ViTs, the required electricity amounts to  $600\mathrm{kWh}$ . Assuming our country's consumer electricity price of  $0.4812\mathrm{€} / \mathrm{kWh}$  and the country's typical CO2 emissions per  $\mathrm{kWh}$  of  $428\mathrm{gCO2e} / \mathrm{kWh}$ , both of which are pessimistic estimates given that the experiments ran in a local academic datacenter, these requirements translate to about 300 USD and  $256\mathrm{kg}$  of CO2 equivalent emissions.
The financial cost of this work is dominated by crowdworker compensations. As outlined in Appx. A.3, workers are compensated at an hourly wage of 15 USD, or 2.79 USD / HIT. Since all workers are
Figure 18: Sparseness of activations does not predict a unit's interpretability. We compute the fraction of non-positive values (i.e., zero after ReLU activation) in the feature maps of the units of interest averaged over the ImageNet validation set for all investigated models. We then show a unit's interpretability as a function of this pixel-wise sparseness measure. However, the two do not appear to have a meaningful relationship, as indicated by Spearman's rank correlation shown above each plot.
compensated, even if the results of their HIT do not pass our quality checks, the total cost incurred by the experiment (including the fees paid to MTurk) amounts to around  $12^{\prime}000$  USD.
Figure 19: Sparseness of entire channels does not predict a unit's interpretability. Similar to Fig. 18, we compute the fraction of images for which an entire feature map achieves only non-positive values (i.e., zero after ReLU activation). Analogously to before, we plot a unit's interpretability as a function of the channel-wise sparseness and find no strong relation between this sparseness measure and a unit's interpretability.
Channelwise Sparsity
E Further Screenshot of Psychophysics Trials
Figure 20: Screenshot of the initial overview of the HIT presented to workers considering the task. We inform participants that they consent to their anonymized data being used for a scientific study.
Which image at the center matches the Right Examples better?
Left Examples
Right Examples
Figure 21: Screenshot of two of the twelve possible instruction trials to explain the task to participants in the natural condition after the participant has given the correct response. See Fig. 22 for examples in the other condition.
Left Examples
Figure 22: Screenshots of two of the twelve possible instruction trials to explain the task to participants in the synthetic condition after the participant has given the correct response. See Fig. 21 for examples in the other condition.
Figure 23: Screenshots of two of the twelve possible instruction trials to explain the task to participants in the synthetic condition before the participant has given a response (top) and after the participant has given the wrong response (bottom).