How Well do Feature Visualizations Support Causal Understanding of CNN Activations?
Authors: Roland Simon Zimmermann
Judy Borowski
Robert Geirhos
Matthias Bethge
Thomas S. A. Wallis
Wieland Brendel
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: evaluation of interpretability
feature visualization
activation maximization
human psychophysics
understanding CNNs
explanation method
counterfactual
causal understanding

Abstract: A precise understanding of why units in an artificial network respond to certain stimuli would constitute a big step towards explainable artificial intelligence. One widely used approach towards this goal is to visualize unit responses via activation maximization. These feature visualizations are purported to provide humans with precise information about the image features that cause a unit to be activated - an advantage over other alternatives like strongly activating dataset samples. If humans indeed gain causal insight from visualizations, this should enable them to predict the effect of an intervention, such as how occluding a certain patch of the image (say, a dog's head) changes a unit's activation. Here, we test this hypothesis by asking humans to decide which of two square occlusions causes a larger change to a unit's activation.
Both a large-scale crowdsourced experiment and measurements with experts show that on average the extremely activating feature visualizations by Olah et al. (2017) indeed help humans on this task ($68 \pm 4$% accuracy; baseline performance without any visualizations is $60 \pm 3$%). However, they do not provide any substantial advantage over other visualizations (such as e.g. dataset samples), which yield similar performance ($66\pm3$% to $67 \pm3$% accuracy). 
Taken together, we propose an objective psychophysical task to quantify the benefit of unit-level interpretability methods for humans, and find no evidence that a widely-used feature visualization method provides humans with better "causal understanding" of unit activations than simple alternative visualizations.

How Well do Feature Visualizations Support Causal Understanding of CNN Activations?
Roland Zimmermann'
Judy Borowski"
Robert Geirhos!
Matthias Bethge
Thomas S. Wallis Wieland Brendel
Tubingen Al Center, University of Tübingen, Germany.
Institute of Psychology and Centre for Cognitive Science, Technical University of Darmstadt, Germany.
Shared first authorship, determined by coin flip. firstname iattaameeunittubbnneee Joint supervision.
Abstract
A precise understanding of why units an artificial network respond to certain stimuli would constitute big step towards explainable artificial intelligence. One widely used approach towards this goal is to visualize unit responses via activation maximization. These synthetic feature visualizations are purported to provide humans with precise information about the image features that cause a unit to be activated an advantage over other alternatives like strongly activating natural dataset samples. If humans indeed gain causal insight from visualizations, this should enable them to predict the effect of intervention, such as how occluding a certain patch of the image (say, dog's 's head) changes a unit's activation. Here, test this hypothesis by asking humans to decide which of two square occlusions causes a larger change to a unit's activation. Both a large-scale crowdsourced experiment and measurements with experts show that average the extremely activating feature visualizations by Olah et al. [40] indeed help humans on this task (68 + 4 % accuracy; baseline performance without any visualizations is 60 +: %). However, they do not provide any substantial advantage over other visualizations (such e.g. dataset samples), which yield similar performance (66+-3 % 67 4-3 % accuracy). Taken together, we propose an objective psychophysical task to quantify the benefit of unit-level interpretability methods for humans, and find no evidence that a widely-used feature visualization method provides humans with better "causal understanding" of unit activations than simple alternative visualizations.
Introduction
It is hard to trust a black-box algorithm, and it is hard to deploy an algorithm if one does not trust its output. Many of today's best-performing machine learning models, deep convolutional neural networks (CNNs), are also among the most mysterious ones with regards to their internal information processing. CNNs typically consist of dozens of layers with hundreds or thousands of units that distributively process and aggregate information until they reach their final decision at the topmost layer. Shedding light onto the inner workings of deep convolutional neural networks has been long-standing quest that has so far produced more questions than answers.
One of the most popular tools for explaining the behavior of individual network units is to visualize
unit responses via activation maximization [6 33 38 35 39
36 54 [5]. The idea is to start with
an image (typically random noise) and iteratively change pixel values maximize the activation
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

A Synthetic feature visualizations to show what "causes" activation
B
Human psychophysical experiment:
Synthetic images helpful causal
Given references which query image activates a unit more?
Synthetic
0.70-
whole face or just an eye?
Proportion Carrest
0.65-
Choose
Natural
0.55-
Chance
Synthetic Natural
None
References
Queries
Figure 1: How useful feature visualizations interpret the effects of interventions? A: "Causal" synthetic feature visualizations. B: Human experiment. Given strongly activating reference images (e.g. synthetic or natural), human participant chooses which out of two manipulated images activates a unit more. Note that this trial is made up real trials are often more difficult. C: Core result. While participants are above chance for all visualization types, synthetic images only provide a substantial advantage over no references and not over other alternatives such as natural references.
of a particular network unit via gradient ascent. The resulting synthetic images, called feature visualizations, often show interpretable structures, and are believed to isolate and highlight exactly those features that "cause" a unit's s response [40 50). Some of the synthetic feature visualizations appear quite intuitive and precise. As shown in Fig. DA, they might facilitate distinguishing whether, for example, unit responds to just an eye or whole dog's face.
However, other aspects cast a more critical light on feature visualization's 's "causality": Generating these synthetic images typically involves regularization mechanisms [36 33 38 (35, which may influence how faithfully they visualize what "causes" a network unit's activation. Furthermore, to obtain a complete description of mathematical function, one generally needs more information than just knowing its extrema. In view of this, it an open question how well a unit can be characterized by simply visualizing the arguments of its maxima. Finally, a crucial unknown factor is whether humans are able to obtain causal understanding of CNN activations from these synthetic visualizations.
Given these points, we develop psychophysical experiment to test whether feature visualizations by Olah et al. [40] indeed allow humans to gain causal understanding of unit's behavior. Our task is based on the reasoning that being able to predict the effect of an intervention is at the heart of causal understanding. Understanding the causal relation between variables implies an understanding of how changes in one variable affect another one 45). In our proposed experiment, this means that participants can predict the effect of an intervention if they know the causal relation between image features and unit's activations. Our experiment tests whether synthetic feature visualizations indeed provide information about such causal relations. Specifically, we ask humans which of two manipulated images activates CNN unit more strongly. The interventions we test are obtained by placing an occlusion patch at two different locations in an image. Taken together, this experiment probes the purported explanation method's 's advantage of causality in a counterfactual-inspree. prediction set-up [4].
in form of an image manipulation
Besides feature visualizations, other visualization methods have been used to gain an understanding of the inner workings of CNNs. In this experiment, we additionally test alternatives based on natural dataset examples and compare them with feature visualizations. This is particularly interesting because dataset examples are often assumed to provide less "causal" information about unit's response as they might contain misleading correlations 40 To continue the example above, dog eyes usually co-occur with dog faces; thus, separating the influence of one image feature from the other one using natural exemplars might be challenging.
Our data shows that:
Synthetic feature visualizations provide humans with some helpful information about the most important patch in an image but not much more information than visualizations all.
Dataset samples as well as other combinations and types of visualizations are similarly helpful.
How easily the most important patch identifiable depends on the unit, the images as well as the relative activation strength attributed to the patch.

---

2 Related Work
Feature visualizations are a widely used method to understand the learned representations and decision-making mechanisms CNNs [33
37 As such, several works 46 and other networks or introduce analysis frameworks [65]. In contrast, some researchers question whether this synthetic visualization technique, first introduced by Erhan et [6]., is too intuition-driven [27. and how representative the appealing visualizations in publications are [26] Further, as already mentioned above, the engineering of the loss function may influence their faithfulness [36 33
38
35
39
36
leverage this method to study Inception V
42
50
58
59
66 21 20]; others create interactive tools 66 44
52
38 35), Another challenge is generating diverse feature visualizations to represent the different aspects that one single unit may respond to [42 36]. Finally, our recent human evaluation study [5] found that while these synthetic images do provide humans with helpful information in a forward simulation-inspired task, simple natural dataset examples are even more helpful.
Human evaluation studies are extensively used quantify various aspects of interpretability. As an alternative to pure mathematical approximations [2] 66, 57 63), researchers not only evaluate the understandability of explanation methods in psychophysical studies 7
34 5) but also trust
in these methods [28 64 as well as the human cognitive load necessary for parsing explanations
1 whether humans would follow explained model decision [47]
1 48
A recent study
even demonstrates that metrics of the explanation quality computed with human judgment are more insightful than those without 4
Counterfactuals are a pop- ular paradigm for both creating as well as eval- uating explanation meth- ods. Intuitively, they pro- vide answers to the ques- tion "what should I change to achieve a different out- come?" in the context of machine learning expla- nation methods, usually the smallest, realistic change to a data point is of inter- est. As examples, coun- terfactual explanation meth- ods have been developed for vision- [22] and language- based (62 models as well
Strongly Activating Images
Strongly Activating Image
Which image elicits higher activation?
more confident
more confident
Figure 2: Schematic visualization of an example trial our psy- chophysical experiment. For certain network unit, participants are shown several maximally activating images. While the ones on the left serve reference images, the The top one a natural maximally activating image and the bottom ones copies of said image with square occlusions at different loca- tions. The task is to select the image that activates the given network unit more strongly. Participants answer by clicking the number below the corresponding image according to their confidence level not confident, 2: somewhat confident, very confident). Correct answer: right image.
as for model-agnostic sce- narios [51]. Further, they are set into context of the EU General Data Protec- tion Regulation [60]. Us- tun et al. (56) investigate feasible and least-cost coun- terfactuals, while Mahajan ct al. [32] and Karimi et al. 225 take feature interactions into account. To evaluate rather than create explanation methods, researchers often follow the "counterfactual simulation" task introduced by Doshi- Velez and Kim 14: Humans are given input, output, and an explanation and are then asked "what must be changed to change the method's 's [model's s] prediction to desired output?" Doshi- Velez and Kim
the right serve query images:
Based this task, Lucic et al. [30] test their new explanation method and Hase and Bansal compare different explanation methods to each other.
In this project, we design counterfactuul-inspiee task to evaluate how well feature visualizations support causal understanding of CNN activations. This the first study to apply such paradigm to understanding the causes of individual units' activations. In order to scale the experiments,
3

---

simplify our task by having participants choose between two intervention options, rather than having them freely determine interventions themselves.
3
Methods
We run an extensive psychophysical experiment with more than 12, 000 trials distributed over 323 crowdsourced participants on Amazon Mechanical Turk (MTurk) and two experts (the two first authors) D For more details than provided below, please see Appx. Sec. A.1
Design Principles Overall, our experimental design choices aim at (1) the best performance possible, meaning that we select images that make the signal as clear as possible; (2) generality over the network, meaning that we randomly sample units of different layers and branches (testing all units would be too costly); and (3) easy extendability, meaning that we choose a between- between-participan design (each participant sees only one reference image condition) so that other visualizations methods can be added to the comparisons in the future.
3.1 Psychophysical Task
If feature visualizations indeed support causal understanding of CNN activations, this should enable humans to predict the effect of an intervention, such as how occluding an image region changes unit's activation. Based this idea, employ wooaatteraative forced choice task (chance performance: 50%) where human observers are presented with two different occlusions in an image. and asked to estimate which of them causes smaller change to the given unit's activation (sce Fig. for an example trial). More specifically, participants choose the query image that they believe to also elicit strong activation given of 9 reference images. Such references could for instance consist of synthetic feature visualizations of certain unit (purportedly "causal")), or alternative visualizations. To summarize, the task requires humans first identify the shared aspect in the reference images and to then choose the query image in which that aspect is more visible. Since we do not make any assumptions about whether participants are familiar with machine learning, we avoid asking participants about activations of a unit in the CNN. Instead, we explain that an image would be "favored" by machine, and the task is to select the image which is "more favored". The complete set of instructions shown to participants can be found in Appx. Fig. ( and [0] In addition to each participant's 's image choice, the subjective confidence level and reaction time are also recorded.
3.2 Stimulus Generation
To generate stimuli, we follow Olah et al. [4 and use an InceptionVI network [53] trained on ImageNet [12 [4. Throughout this paper, refer to CNN's channel as "unit" and imply taking the spatial average of all neurons in one channel We test units sampled from 9 layers and 2 Inception module branches (namely 3 x 3 and POOL). For more details on the generation procedures of the respective stimuli, see Appx. A.1.2
We use five different types of reference images:
Synthetic references: The synthetic images are the optimization results of the feature visualization method by Olah et al. 440 with the channel objective for 9 diverse images.
Natural references: The reference images are the most strongly activating dataset samples from ImageNet 22 49).
This a combination of the previous two conditions: the most strongly activating natural and 4 synthetic reference images are used. The motivation is that this condition combines the advantages of both worlds - namely precise information from feature visualizations and easily understandable natural images and, thus, has the potential to give rise to higher performance in the task. Jointly looking at these two visualization types is common in practice [40]
Mixed references:
Cooee and data available at argithub. www.bbbnnndgguuogaaassssutatttad
2 Other papers might refer to channel as "feature map", e.g. [
To reduce compute requirements, use random subset of the training set (72 50%).

---

Blurred references: To increase the informativeness of natural images for this task, we modify them by blurring everything but a single patch. This patch chosen in the same way as in the maximally activating query image (see below). Consequently, this method cues participants to the most important image feature. In way, these images can be seen as an approximate inverse the maximally activating query image and might improve performance on our task.
No references: This a control condition in which participants do not see any reference images and have to solve the task purely based on query images.
To generate query images, we place a square patch of 90 x 90 pixels of the average RGB color of the occluded pixels into most strongly activating image chosen from ImageNet. The location of the occlusion patch is chosen such that the activation of the manipulated image is either minimal or maximal among all possible occlusion locations. These images then yield the distractor and target query images respectively.
3.3 Structure of the Psychophysical Experiment
We test the five different reference image types as separate experimental conditions. In each condition, we collect data from a total of 50 different MTurr participants, each assigned to a single Human Intelligence Task (HIT) consisting of an instruction block, variable number of practice blocks and a main block. The instructions extensively explain hand-crafted example trial (see Appx. Fig. 9 and 10 The blocks of 4 practice trials each which are randomly sampled from pool of 10 trials- have to be repeated until reaching 100% performance; except in the none condition, as there is no obvious ground truth due to the absence of reference images. Finally, 18 main trials follow that are randomly interleaved with a total of 3 obvious catch trials. While feedback is provided during practice trials, feedback is provided in the other trials. At the end, participants can share comments via an optional free-text field. Across all conditions, all participants see the same query images for the instruction, practice and catch trials. In contrast, the query images differ across participants in the main trials: In each reference image condition, we test 10 different sets of query images, each responded by 5 different MTurr participants, hence 50 HITs per condition. The order of the main and catch trials per participant is randomly arranged, and identical across conditions. Each MTurk participant takes part in only one reference image condition (i.e. reference images are deween-participane factor). For more details, see Appx. Sec, Sec. A.1.4
3.4 Ensuring High-Quality Data in an Online Experiment
To ensure that the data we collect in online experiment is of high quality, take two measures: (I) We integrate hidden checks which were set before data collection. Only if a participant passes all five of them do we include his/her data in our analysis. First, these exclusion criteria comprise performance threshold on the practice trials as well as a maximum number of blocks a participant may attempt. Further, they include performance threshold for catch trials, a minimum image choice variability as well as a minimum time spent on both the instructions and the whole experiment. For more details, Appx. Sec. I. (2) Our previous human evaluation study in a well-controlled lab environment found that natural reference images are more informative than synthetic feature visualizations when choosing which of two different images is more highly activating for given unit [5). We replicate this main finding on MTTurk based on a subset of the originally tested units (see Appx. (A.3) which indicates that the experiment's 's environment does not influence this task's outcome. Our decision to leverage a crowdsourcing platform is further corroborated by our result in Borowski al. [5) that there is no significant difference between expert and lay performance.
3.5 Baselines
In order to both set MTurk participants' performance into context as well as evaluate different strategies participants could use perform our task, further evaluate few basclines.
The two first authors answer all 18 trials in all 5 reference conditions on all 10 image sets. As they are familiar with the task design and are certainly engaged, this data serves as an upper human bound.
Expert Baseline:
Center Baseline: In natural images from ImageNet, important objects are likely to be closer the center of the image. If participants were biased to assume that units respond to objects, potential
5

---

strategy to decide which occluding patch produces a smaller effect on the unit's activation would therefore be to choose the image with the most eccentric occlusion. The Center Baseline model performs this strategy for all images.
The Center Baseline is not a perfect measurement of an object-biased strategy because primary objects can appear away from the center. To account for this, the first authors and the last author manually label all trials, choosing the image for which the occlusion hides as little information as possible from the most prominent object in the scene. In approximately one third of the trials (58) (58/189 180), the authors' confidence ratings are very low (reflecting e.g. the absence of a primary object); these cases we repeatedly replace the decisions by random binomial choices. Thus, in the results, we report the estimated expected values, but cannot perform by-trial analysis. For more details, see Appx. Sec. A.1.3
Primary Object Baseline:
Variance Baseline:
Another assumption participants might make is that patch in low-contrast region, e.g. a blue sky, is unlikely to have a large effect on the unit's ' activation. This baseline selects the query image whose content is less affected by the introduction of the occlusion patch. To simulate this, we calculate the standard deviation over the occluded pixels and choose the one of the lower standard deviation.
Saliency Baseline: As complement to the baselines above, this baseline selects the query image whose original pixels hidden by the occlusion patch have a lower probability of being looked at by the participants. This simulates that participants select the image with a patch that occludes less prominent information and is estimated with the saliency prediction model DeepGaze IIE [29]. For more details, Appx. Sec. A.1.3
4 Results
The results shown in this section are based on 7350 trials from MTurk participants, who passed all exclusion criteria, and experts distributed over five conditions. In all figures, Synthetic refers the purportedly "causal", secivation-maxiniiing maximizing feature visualizations, Natural to ImageNet samples, Mixed the combined presentation of synthetic and natural images, Blur the blurred images, and None to the condition with no reference images at all. Further, error bars indicate two standard errors above and below the participant-men mean over network units and image sets, unless stated otherwise.
4.1 No Significant Advantage of Synthetic Feature Visualizations
feature visualizations
A
Performance
B
Reaction Time
provide humans with use- ful information about the image features causing high unit activations and other visualizations do not, participants' accu- racy in our task should be higher given feature vi- sualizations than for all other visualization types or no reference images. This is only partly what we find: On average, ac- curacy for feature visual- izations is slightly higher than when reference images are given (67+t%% vs. 60 + 3%). However, the accuracy for feature visualizations not significantly higher than for other visualization methods (see Fig. BA, dark bars). For the latter, MTurk participants reach between 66 + 3% and 67 + % depending on the visualization type. Statistically, only the condition without reference images is
0.8
Afrt
10
Experts
Progortion Correct
0.6
Chance
0.4
Reaction
4
0.6--022
0.72:0.0
0.6=0
0.00:0.44
0.7000004
0:60:0.55
16
0000-1 22
Synthetic
Natural
Mixed
Blur
None
Symthei Natural Mixed
Blur None
Reference Images
Reference Images
Figure 3: A: Task accuracy. On average, humans reach the same per- formance regime with any visualization method. This holds for both lay participants on MTurk (darker colors) as well as experts (brighter colors). B: Reaction times. MTurk participants need several seconds to answer trial, indicating that they carefully make their decision. For more details see Appx. Fig. 13
*(18 main + 3 catch x50 MTurk participants x5 conditions (18 main + 3 catch x0 expert measurements x5 conditions.

---

different from all other conditions (p < 0.05, Soin-Whitney U test). Taken together, these findings suggest that all visualization methods are similarly helpful for humans in our counterfactual-inge -nnspired task, and that they only seem to offer small improvement over no visualizations at all.
4.1.1 MTurr Participants Carefully Make Their Choices
Similar performances for various conditions such as those found in Fig. 3 might suggest that participants would not give their best when doing our experiment. However, several aspects speak against this: (1) Measurement of the two first authors, i.e. experts who designed and thus clearly understand the task, and certainly engage during the experiment, again show very similar performance (see Fig. AA, bright bars): This estimated upper bound is just - 6% better than MTTurk participant performance. (2) With our strict exclusion criteria, we check for doubtful participant behavior and only include data from participants who pass all five criteria. (3) Reaction times per trial (see Fig. 3)) lie between 4s and s.. This. as well as the fact that participants take longer for the conditions with references than for the None condition, suggest that they carefully make their decisions. (4) Several MTurk participants comments an optional free-text field indicate that they engage in the task: "[...] did my best", "It was engaging", "interesting task". (5) Trial-by- trial responses between MTurk participants are more similar than expected by chance (see Fig. 4B discussed below), which suggests that humans use the available information.
4.1.2 Simple Baselines Can Reach the Same Above- Chance Performance Regime
recision-making strategies can be diverse. To set hu-
A
C
Progatton Corract
Synthetic
man performance into con- text, we evaluate several simple strategies as base- lines: How high is per-
044
Natural
02-
Cohen's Kappa
formance if one always
Cuntar
Object Vardrceeallency Baselines
chooses the query image with an unoccluded center (Center Baseline) or pri- mary object (Object Base- line)? Or such that the more varying or salient image re- gion is unoccluded (Vari- ance and Saliency Base- line)? Fig. 4A shows that these strategies have vary- ing performances with the best ones namely the Ob- ject and Variance baselines reaching 63 + % and 63 %, respectively. Since al- ready these simple heuris- ties, which do not require reference visualizations, can reach the same performance regime as participants, the additional advantage of visualizations (reaching just up to 4 % better performance) appears limited.
B
Nore
Synthetic
6-4
1-6
.145
2+6
4+5
0-15
6-11
Natural-
Variance
12+2
21-6
15-6
166
14-7
8-10
108+0
11+17
Sallency
37+7
02-7
10-5
12-6
10
4-12
11+17
108-0
Basaline
Synthelic
Natural
Satlency
Baesliee
Synthetic
Natural
Nome
Camter
Vertence
Saltenry
Particioants
Baselines
Figure 4: Baseline performances. Simple baselines can reach above chance level B, C: Decision consistency. The mean and two standard errors of the mean Cohen's kappa averaged over partici- pants and image sets quantifies the pairwise consistency of decision patterns 6 While they vary across participants, they are higher between conditions with natural references and highest between the Saliency Baseline and other conditions. For more details, see Appx. Fig 15
4.2 By-trial Decisions Show Systematic but Fairly Low Agreement
While accuracy is the most common metric evaluate task performance, it does not suffice to
compare two systems' decision decision-making processes [31 19 I8
Instead, a quantitative trial-by-trial error analysis necessary to ascertain or distinguish strategies. Here, we use Cohen's 's kappa [10 to
'Only the Object Baseline has an error bar because in trials with, e.g. no clear primary object, we replace decisions by random binomial choices. The reported values are the estimated expectation value and standard deviation.
'There is no data for the Object Baseline because about one third of the trials do not have a clear answer from the three author responses. For more details, see Appx. A.1.3
7

---

calculate the degree of agreement in classification while taking the expected chance agreement into account. value of corresponds to perfect agreement, while a value of O corresponds as much agreement as would be expected by chance. Negatíve values indicate systematic disagreement.
In Fig. A and C, we plot consistency between MTTurk participants of the same and different reference conditions as well as between MTurk participants and baselines. Since Cohen's 's kappa only allows for comparisons of two decision makers, we compute this statistic for all possible pairs across image sets, and report the mean over participants and image sets and two standard errors of the mean. All values between participants as well as between participants and baselines are in an intermediate regime (up to 0.40). This suggests that there is systematic agreement. but also quite some room for subjective decisions, Among participant-basele baseline comparisons, highest agreement is found for the saliency baselind I. while lowest agreement is found for the Center Baseline. Within participant to participant comparisons, decision strategies for conditions involving unmodified natural images (Natural, Mixed) are more similar to each other as well as slightly more similar to other strategies than the Synthetic, Blur or None condition to other strategies. Within the Synthetic condition, participants are relatively inconsistent. We hypothesize that due to the fact that humans are more familiar with natural images, they use more consistent information from these types of reference images and, thus, their decisions are more similar.
4.3 Performance Varies across Units, Image Sets and Activation Differences, but Less So for Reference Conditions
Having found that feature visualizations do not offer an overall advantage over other techniques, we now ask: Is performance similar across units, query images and their activation differences?
As evident from Fig. 5 performance varies by unit, but usually not much by reference condition: While only one unit (layer 2, POOL) is clearly below chance level, many units reach around average performance and a few units stand out with high performances (e.g. layer 8, POOL). Further, the five reference conditions are relatively close to each other for most units. Finally, on the image set level, we observe fairly high variance probably partly due to the limited number of participants per image set (see Appx. Fig. [4].
Units and Image Sets
Fig. further illustrates the different difficulty levels as well as the strong unit- and mengeddepend:ncy For the shown easy unit (Fig, A the (presumably yellow-black) feature is fairly clearly identifiable and visible in the diverse reference and query images. In contrast, for the shown difficult unit (Fig. (B), the unit's feature selectivity is unclear not only in the reference but also in the query images.
Activation Differences We hypothesize that our task might be easier if the difference in activations between the two interventions of the query images is larger. In Fig. 7 and B. we plot by-image-set performance against the relative activation differences, i.e. the difference between activations elicited by the two manipulated images normalized by the unperturbed query image's 's activation. The figure shows that even though we select query images as the most strongly activating images for unit, the relative activation differences vary widely. Furthermore, human performance indeed tends to increase with higher relative activation difference, confirming our hypothesis. This trend is stronger in the PoOL than in the 3 x branch as quantified by the Spearman's 's rank correlations in Fig. 7CC
5 Discussion & Conclusions
Explanation methods such as feature visualizations have been criticized as intuition-driven [27]. and it is unclear whether they allow humans to gain a precise understanding of which image features "cause" high activation unit. Here, we propose an objective psychophysical task to quantify how well these synthetic images support causal understanding of CNN units. Through time- and cost- intensive evaluation (based on 24, 439 trials taking more than 81 participant hours including all pilot and reported experiments), we put this widespread intuition to quantitative test. Our data provides no evidence that humans can predict the effect of an image intervention (occlusion) particularly well when supported with feature visualizations. Instead, human performance is only moderately above
FFrom different perspective, this result can be seen as confirmation that the CNN learned to look at the "important" part of the image for downstream classification.
8

---

A
3 Branch
B
Pooling Branch
Trials
Synthetic Mixed None
Natural
Blur
Accuracy
0.6
Chance
0.4
2
5 Layer
Layer
Layer
Figure 5: While for some units predicting the effect of an intervention is relatively easy, for most units performance is close or just above chance. A and show the performance per unit in the main trials separated by branch (3 x 3 and POOL respectively) and layer. C shows the performance per unit in the hand-pice hand-picked trials used as catch trials (hence the *), though selected from those MTurk participants who pass the exclusion criteria without the catch trial exclusion criterion. Note that each represents averages over participants and image sets.
A
Reference Images
Query Images min
B
Reference Images
Query Images min
max
max
synthetic
syntresid
natural
natural
blur
Figure 6: Example reference and query images for a unit with high (A) and low (B) performance from layer and of the PooL branch, respectively.
A
x Branch
B e-g
Pooling Branch
1.00-
the
0.75
Branch
Accuracy
dacs mo
References
3
Pool
Chance
Synthetie
0.047
0.487"
0.50
Natural
0.162
01.477
Mixed
0.080
0.488'
Blur
0.159
0.597
0.25
None
0.242"
0.432"
All
0.135
0.491
Synthetic Mixed
None
0.00
Natural
Blur
0.2
0.6
1.0
0.2
1. Relative activation difference
0.6
1.4
Relative activation difference
Figure 7: Performance tends increase with the relative activation difference between query images. This effect is stronger for the PoOl branch (B) than for the 3 x 3 branch (A) quantified by Spearman's rank correlations (C). Stars signal significance (p < .05). Note that each dot in A and B represents the participant-average i.e. there is one dot combination of layers, branch and image set. For an alternative visualization see Appx. Fig. 16

---

baseline condition where humans are not shown any visualization at all. and similar to that of other visualization methods such as simple dataset samples. Further, by-trial decisions show systematic but fairly low agreement between participants. Finally, task performance depends on the unit choice, image selections and activation differences between query images. These results add quantitative evidence against the cenerally-assumeel usefulness of feature visualizations for understanding the causes of CNN unit activations.
Our counterfactualiinspin task is the first quantitative evaluation of whether feature visualizations support causal understanding of unit activations, but it certainly not the only possible way to evaluate causal understanding. For example, our interventions are constrained to occlusions of fixed size and shape, imposing an upper limit on the precision with which the occlusions can cover the part of the image that is most responsible for driving unit's 's activation. Future work could explore more complex intervention techniques, extend our study to more units of Inception V1 as well as to different networks, and investigate additional visualization methods. Thanks to the between-partin participant design, new conditions can be added to the data without the requirement to re-run already collected trials.
Taken together, the empirical results of our quantitative evaluation method indicate that the widely used visualization method by Olah al. 40 does not provide causal understanding of CNN activations beyond what can be obtained from much simpler baselines. This finding is contrary to wide-spread community intuition and reinforces the importance of testing falsifiable hypotheses in the field of interpretable artificial intelligence (27). With increasing societal applications of machine learning, the importance of feature visualizations and interpretable machine learning methods is likely to continue to increase. Therefore, it is important to develop understanding of what we can and cannot - expect from explainability methods. We think that human benchmarks, like the one presented in this study, help to expose precise notion of interpretability that is quantitatively measurable and comparable to competing methods or baselines. The paradigm we developed in this work can be easily adapted to account for other notions of causality and, more generally, interpretability as well. For the future, we hope that our task will serve as a challenging test case to steer further development of feature visualizations.
Author Contributions
The idea to test how well feature visualizations support causal understanding of CNN activations was born out of several reviewer and audience comments on our previous paper (5) The first idea of how to test this in a psychophysical experiment came from TSAW. JB led the project. JB, RSZ, WB and TSAW jointly improved the experimental set-up with input from MB and RG. RSZ led and JB helped with the implementation and execution of the experiment: JB led and RSZ contributed to the generation of stimuli. RSZ and JB both coded the baselines, and TSAW guided the replication experiment with statistical power simulations. The data analysis was performed by RSZ and JB with advice and feedback from RG, TSAW, WB and MB. TSAW and WB provided day-to-day supervision. While JB and RSZ created the first draft of the manuscript, RG and TSAW heavily edited the manuscript and all authors contributed to the final version.
Acknowledgments
We thank Felix A. Wichmann and Isabel Valera for helpful discussion. We further thank Ludwig Schubert for information on technical details via slack distill. pubb In addition, thank colleagues for helpful discussions, and especially Matthias Kümmerer, Dylan Paiton, Wolfram Barfuss, and Matthias Tangemann for valuable feedback on our task, and/or technical support. Moreover, we thank our various reviewers and other researchers for comments on our previous paper inspiring us to investigate causal understanding of visualization methods. And finally, we thank all our participants for taking part in our experiments.
Funding
The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting JB, RSZ and RG. This work was supported by the German Federal Ministry of Education and Research (BMBF) through the Competence Center for Machine Learning (TUE.AI, FKZ O1IS 8039A) and the Bernstein Computational Neuroscience Program Tübingen (FKZ 01 01GQ1002), the Cluster Excellence Machine Learning: New Perspectives for Sciences (EXC2064/1), ), and the German Research Foundation (DFG, SFB 1233, Robust Vision: Inference Principles and Neural Mechanisms, TP3, project number 276693517), 7). MB and WB acknowledge funding from the MICrONS program of the Intelligence Advanced Research Projects
10

---

Activity (IARPA) via Department of Interior/Interior Business Center (Dol/IBC) contract number DI6PC00003. WB acknowledges financial support via the Emmy Noether Research Group on The Role Strong Response Consistency for Robust and Explainable Machine Vision funded by the German Research Foundation (DFG) under grant no. BR 6382/1-1.

---

References
[1] Ashraf M. Abdul, Christian von der Weth, Mohan S. Kankanhalli, and Brian Y. Lim. COGAM: measuring and moderating cognitive load in machine learning model explanations. In Regina Bernhaupt, Florian 'Floyd' Mueller, David Verweij, Josh Andres, Joanna McGrenere, Andy Cockburn, Ignacio Avellino, Alix Goguey, Pernille Bjøn, Shengdong Zhao, Briane Paul Samson, and Rafal Kocielmik, editors. CHI '20: CHI Conference on Human Factors in Computing Systems, Honolulu. HI. USA. April 25-30, 2020, pages 1-14. ACM, 2020.
doi: 10.1145/
3313831.337661.
[2] Elvio Amparore, Alan Perotti, and Paolo Bajardi. To trust or not to trust an explanation: using leaf to evaluate local linear xai methods. PeerJ Computer Science, 7:e479, 2021.
[3] Lucas Beyer, Olivier Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are we done with imagenet? arXiv preprint arXiv:2006.07/55 159, 2020.
[4] Felix Biessmann and Dionysius Irza Refiano. A psychophysics approach for quantitative comparison of interpretable computer vision models. arXiv preprint arXiv:1912.000 2019,
[5] Judy Borowski, Roland Simon Zimmermann, Judith Schepers, Robert Geirhos, Thomas S. A. Wallis, Matthias Bethge, and Wieland Brendel. Exemplary natural images explain {cnn] activations better than Learning Representations, 2021.
feature visualization. In International Conference on
[6] Santiago A Cadena, Marissa A Weis, Leon A Gatys, Matthias Bethge, and Alexander S Ecker. Diverse feature visualizations reveal invariances in early layers of deep neural networks. In Proceedings of the European Conference on Computer Vision (ECCV), pages 217-232, 2018.
[7] Carrie J Cai, Jonas Jongejan, and Jess Holbrook. The effects of example-based explanations in a machine learning interface. In Proceedings of the 24th International Conference on Intelligent User Interfaces, pages 258-262.2 2019.
[8] Nick Cammarata, Gabriel Goh, Shan Carter, Ludwig Schubert. Michael Petrov, and Chris Olah. Curve detectors. Distill, 2020. doi: https:/ddaiiil.pub//00eeerrrrris pptpo/00/ccciiiisuueeeeeeeeeeeee
[9]
Nick Cammarata, Gabriel Goh, Shan Carter, Chelsea Voss, Ludwig Schubert, and Chris Olah. Curve circuits. Distill, 6(1 6(1):e00024-000 2021.
[10] Jacob Cohen. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37-46, 1960.
[11] Joshua de Leeuw and Benjamin A Motz. Psychophysics in a web browser? comparing response times collected with javascript and psychophysics toolbox in a visual search task. Behavior Research Methods, 48(1):1-12 2016.
[12] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Computer Society Conference Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA, pages 248-255. IEEE Computer Society, 2009. doi: 10.1 109/CV/R.2009955222
[13] William K Diprose, Nicholas Buist, Ning Hua, Quentin Thurier, George Shand, and Reece Robinson. Physician understanding, explainability, and trust in a hypothetical machine learning risk calculator. Journal of the American Medical Informatics Association, 27(4):592 -000, 2020.
[14] Finale Doshi-Velez and Been Kim. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv: 1 001.7702200088 2017.
[15] Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Brandon Tran, and Aleksander Madry. Adversarial robustness as a prior for learned representations. arXiv preprint
M.Xiv:1906.0094: 2019.
[16] Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of a deep network. University of Montreal, 1341(3):1, 2009.
[17] Ruth Fong, Mandela Patrick, and Andrea Vedaldi. Understanding deep networks via extremal perturbations and smooth masks. In 2019 IEEF/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November 2, 2019, pages 2950-2955 IEEE, 2019. doi: 10. 10.1109/CCCV22299000000
12

---

[18] Christina M Funke, Judy Borowski, Karolina Stosio, Wieland Brendel, Thomas SA Wallis, and Matthias Bethge. Five points to check when comparing visual perception humans and machines. Journal of Vision, 2163)1 2021.
[19] Robert Geirhos, Kristof Meding, and Felix A. Wichmann. Beyond accuracy: quantifying trial-by-trial behaviour of cnns and humans by measuring error consistency. In Hugo Larochelle. Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin, editors, Ad- vances in Neural Information Processing Systems 33: Anmual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12 12, 2020, virtual, 2020.
[20] Gabriel Goh, Nick Cammarata, Chelsea Voss, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. Multimodal neurons in artificial neural networks. Distill, 6(3):
e30, 2021.
[21] Nicolas Gonthier, Yann Gousseau, and Saïd Ladjal. An analysis of the transfer learning of convolutional neural networks for artistic images. arXiv preprint arXiv:2011.02727, 2020.
[22] Yash Goyal, Ziyan Wu, Jan Ernst, Dhruv Batra, Devi Parikh, and Stefan Lee. Counterfactual visual explanations. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference Machine Learning. ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 2376-2384, PMLR, 2019.
[23] Peter Green and Catriona J MacLeod. Simr: package for power analysis of generalized linear mixed models by simulation. Methods in Ecology and Evolution, 7(4):493-492) 2016.
[24] Peter Hase and Mohit Bansal. Evaluating explainable AI: Which algorithmic explanations help users predict model behavior? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5540-5552, Online, 2020. Association for Computational Linguistics. doi: 10.18653/v aal.20..cl-mii....
[25] Amir-Hossein Karimi, Gilles Barthe, Borja Balle, and Isabel Valera. Model-agnostic counter- factual explanations for consequential decisions, 2020.
[26] Nikolaus Kriegeskorte. Deep neural networks: a new framework for modeling biological vision and brain information processing. Anmual review of vision science, 1:417-446, 2015.
[27] Matthew I. Leavit and Ari Morcos. Towards falsifiable interpretability research. arXiv preprint arXiv:2010. /2016, 2020.
[28] Brian Y. Lim, Anind K. Dey, and Daniel Avrahami. <i>why and why not</i> explanations improve the intelligibility of context-aware intelligent systems. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '09, page 2128-2128, New York, NY, USA, 2009. Association for Computing Machinery. ISBN 9781605582467. doi: 10. 1518701.1519023
[29] Akis Linardos, Matthias Kümmerer, Ori Press, and Matthias Bethge. Calibrated prediction in and out-of-domain for state-of-the-art saliency modeling, 2021.
[30] Ana Lucic, Hinda Haned, and Maarten de Rijke. Why does my model fail? contrastive local explanations for retail forecasting. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pages 90-98, 2020.
[31] Wei Ji Ma and Benjamin Peters. A neural network walks into a lab: towards using deep nets as models for human behavior. arXiv preprint arXiv:2005.02188 2020.
[32] Divyat Mahajan, Chenhao Tan, and Amit Sharma. Preserving causal constraints in counterfactual explanations for machine learning classifiers, 2020.
[33] Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by inverting them. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pages 5188-5196. IEEE Computer Society, 2015. doi: 10.1 P0..///V...01...2999
[34] Sina Mohseni, Jeremy E Block, and Eric Ragan. Quantitative evaluation of machine learning explanations: A human- grouaddg benchmark. In 26th International Conference Intelligent User Interfaces, pages 22-31, 2021.
[35] Alexander Mordvintsev, Christopher Olah, and Mike Tyka. Inceptionism: Going deeper into neural networks, 2015.
13

---

[36] Anh Nguyen, Jeff Clune, Yoshua Bengio, Alexey Dosovitskiy, and Jason Yosinski. Plug & play generative networks: Conditional iterative generation of images in latent space. In 2017 IEEE Conference on Computer Vision and Patlern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 3510-3520. IEEE Computer Society, 2017. doi: 20.1109/CVP..2017..37.
[37] Anh Nguyen, Jason Yosinski, and Jeff Clune. Understanding neural networks via feature visualization: A survey, In Explainable AI: interpreting, explaining and visualizing deep learning, pages 55-76. Springer, 2019.
[38] Anh Mai Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pages 427-436. IEEE Computer Society, 2015, doi: 11L.0/CV..00...2222888..
[39] Anh Mai Nguyen, Alexey Dosovitskiy, Jason Yosinski, Thomas Brox, and Jeff Clune. Syn- thesizing the preferred inputs for neurons in neural networks via deep generator networks. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Gar- nett, editors, Advances in Neural Information Processing Systems 29; Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spuin, pages 3387-3395, 2016.
[40] Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2(11): e7, 2017.
[41] Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. An overview of early vision in inceptionvl. Distill, 2020. doi:
[42] Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 2020. doi: 10.23915/distill000005 https:/diisdllpuu//20//iiciiiii pubb200000ccccuiszzzomiii
[43] Chris Olah, Nick Cammarata, Chelsea Voss, Ludwig Schubert, and Gabriel Goh, Naturally occurring equivariance in neural networks. Distill, 512x001,5( 2020.
[44] OpenAI. OpenAl Microscope. httpp::/miicroccoee..eeaaaccc. cessed on 09/12/2020).
2020. (Ac-
[45] Judea Pearl. Causality. Cambridge university press, 2009.
[46] Michael Petrov, Chelsea Voss, Ludwig Schubert, Nick Cammarata, Gabriel Goh, and Chris Olah. Weight banding. Distill, 6(4):c00024-009 2021.
[47] Forough Yoursabzi-Sangdelt Daniel Goldstein, Jake M Hofman, Jennifer Wortman Vaughan, and Hanna Wallach. Manipulating and measuring model interpretability. arXiv preprint arXiv:1802.078101 2018.
[48] Grusha Prasad, Yixin Nie, Mohit Bansal, Robin Jia, Douwe Kiela, and Adina Williams. To what extent do human explanations of model behavior align with actual model behavior? arXiv preprint arXiv:2012. 2020.
[49] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision 115(3):2111255 2015. doi: 10. 1007/s1 1263-015-0816
[50] Ludwig Schubert, Chelsea Voss, Nick Cammarata, Gabriel Goh, and Chris Olah. High-low frequency detectors. Distill, 6(1):0002240055, 2021.
[51] Shubham Sharma, Jette Henderson, and Joydeep Ghosh. Certifai: Counterfactual explanations for robustmess, transparency, interpretability, and faimess of artificial intelligence models. arXiv preprint arXiv: wew.. 2019.
[52] Stefan Sietzen, Mathias Lechner, Judy Borowski, Ramin Hasani, and Manuela Waldner. Inter- active analysis of enn robustness. Computer Graphics Forum (Proceedings of Pacific Graphics 2021), 40(7), 2021.
[53] Christian Szegedy. Wei Liu, Yangqing Jia, Pierre Sermanet, Scott F. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In IEEE Conference Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA,
14

---

USA, June 7-12, 2015, pages 1-9. IEEE Computer Society, 2015. doi: 10.1109/CVPR.2015. 7298594.
[54] Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[55] Dimitris Tsipras, Shibani Santurkar, Logan Engstrom. Andrew llyas, and Aleksander Madry. From imagenet to image classification: Contextualizing progress on benchmarks. In Interna- tional Conference Machine Learning, pages 9625-9635. PMLR, 2020.
[56) Berk Ustun, Alexander Spangher, and Yang Liu. Actionable recourse in linear classification. In Proceedings of the Conference on Fairness, Accountability, and Transparency, pages 10-19, 2019.
[57] Arnaud Van Looveren and Janis Klaise, Interpretable counterfactual explanations guided by prototypes. arXiv preprint arXiv:1907.0258/ 2019.
[58] Chelsea Voss, Nick Cammarata, Gabriel Goh, Michael Petrov, Ludwig Schubert, Ben Egan, Swee Kiat Lim, and Chris Olah. Visualizing weights. Distill, 6(2)::00024-0007, 2021.
(59) Chelsea Voss, Gabriel Goh, Nick Cammarata, Michael Petrov, Ludwig Schubert, and Chris Olah. Branch specialization. Distill, 6(4)::000224-008, 2021.
[60] Sandra Wachter, Brent Mittelstadt, and Chris Russell. Counterfactual explanations without opening the black box: Automated decisions and the gdpr, 2018.
[61] Eric Wong, Shibani Santurkar, and Aleksander Mądry. Leveraging sparse linear layers for debuggable deep networks, 2021.
[62] Tongshuang Wu, Marco Tulio Ribeiro, Jeffrey Heer, and Daniel S Weld. Polyjuice: Automated, general-purpose counterfactual generation. arXiv preprint arXiv:2 101.00288,, 2021.
[63] Xi Ye, Rohan Nair, and Greg Durrett. Evaluating explanations for reading comprehension with realistic counterfactuals. arXiv preprint arXiv:2104.04511 045155 2021.
[64] Ming Yin, Jennifer Wortman Vaughan, and Hanna M. Wallach. Understanding the effect of accuracy on trust in machine learning models. In Stephen A. Brewster, Geraldine Fitzpatrick, Anna L.. Cox, and Vassilis Kostakos, editors, Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI 2019, Glasgow, Scotland, UK, May 04-09, 2019, 04455//29005.33005555
page 279, ACM, 2019. doi:
[65] Mohammad Nokhbeh Zacem and Majid Komeili. Cause and effect: Concept-based explanation of neural networks. arXiv preprint Xiw:2105500703: 2021.
[66] Yilun Zhou, Serena Booth, Marco Tulio Ribeiro, and Julie Shah. Do feature attribution methods correctly attribute features? arXiv preprint arXiv:2104 14403, 2021.
15


Reviews:
[{"review_id": "iHOZOm25QFs", "reviewer": "Reviewer_Nxjp", "summary": "This paper put forward an objective psychophysical task to evaluate whether feature visualization could help humans understanding CNN activations. The authors recorded the details of their objective psychophysical task and adopted lots of baselines that ensure their experiments' preciseness. They also analyze the results from different angles.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is a brand-new idea that evaluates the performance of feature visualization totally with psychophysical experiments. However, it is more like a social science document rather than a research paper target for NeurIPS. This paper overflowed with design details, such as 3.1 Psychophysical Task, 3.3 Structure of the Psychophysical Experiment, and 3.4 Ensuring High-Quality Data in an Online Experiment, which could be paced in the supplement. What’s more, the results of the experiment also are trivial. Besides the task accuracy in experiment 1, all other experiments are hard to understand the purpose.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Besides the above mentioned, there are more concerns:\n\n(1) More evidence should be provided to support that why this experiment is sufficient for supporting the claim. Why a similar performance between visualization and nature image can cause the uselessness of visualization explanation? Also, a unit may be sensitive to different patterns (multiclass classification), why nature image fixed in a single class?\n\n(2) The authors do not illustrate how those reference images are assigned. Since the activation of CNN is relevant to units, the reference images should be different if units change. However, this paper doesn't show the details. \n\n(3) It seems maximally activating synthetic image is full of one pattern, and it is understandable if the image is enlarged. However, when the synthetic image is the same size as the natural image, it is hard to recognize the pattern in the image. As the difference in query images is the location of the square patch, it is more reasonable to use a maximum activation patch as a reference rather than the same size image as the reference images. \n\n(4) The authors point out that one advantage of this method is its easy extendability. However, If researchers want to investigate additional visualization methods,  they still need to generate new query images and re-run the collected trials.\n\n(5) This paper lacks novelty, and the results are not interesting. Too many details which seem like placeholders because readers cannot get any inspiration from those paragraphs. \n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10 hours"}, "extra_text": {}}, {"review_id": "erwhW_0q-fS", "reviewer": "Reviewer_VeU5", "summary": "The paper designs and conducts a psychophysical experiment to evaluate how effective unit visualization is in interpreting a neural network. The experiments presents synthetic and natural images obtained through methods of unit activation maximization to human subjects and evaluate how these images could guide them in answering effects of occlusion of images to a network. Through extensive experiment controls at different levels, it arrives at the conclusion that, unit visualization with synthetic images is not particularly useful in helping us understand CNN activations. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The psychophysical experiment design is novel and to my knowledge it is the first study that answers questions about unit interpretability in CNN with actual human behavioral data. \n\nQuality: The amount of controls that are designed into the experiment is very impressive. It is able to rule out most of the competing hypothesis related to this problem. Analysis of the decision patterns, reaction time, as well as comparison to expert evaluator are also very informative. \n\nClarity: The paper is extremely well-written and organized. \n\nSignificance: The behavioral results in this paper has a lot of insights and is important in guiding the field for the next interpretability methods. It would also encourage more validating experiments to be done on other visualization methods in general. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "A potential limitation of the experiment is that, the occlusion based judgement design could bias human subjects to match objects or specific object parts to the occluded images. Units in a CNN, especially the ones in early layers, don't have to function as object detector or object descriptor detector. In this case unit visualization might still be helpful in identifying the general kind of feature that a unit cares about (for example, texture or the horizon) but fails to inform human subjects in the occlusion task. \n\nIt is not surprising to me that the saliency baseline yields similar performance compared to using maximally activated image and that the saliency baseline has the highest decision consistency with other conditions. This result can also be interpreted positively since it demonstrates that, through unit visualization, we verify that the network learns to look at the \"important\" part of the image for downstream classification. In another word, saliency models that are trained with human fixation data could be used not as a baseline but instead as a method in evaluating how well the network learns. \n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "39XZrhBi0vE", "reviewer": "Reviewer_yDAo", "summary": "EDIT AFTER AUTHOR RESPONSE: I read the other reviews and I do not see the need to change my review.\n\nThe paper presents an experiment where users are trained, with or without the use of feature visualizations, to predict which of two images achieves higher activation on a unit from InceptionNet trained on ImageNet.  The query images are obtained by maximally activating and minimally activating occlusions of an natural image.  The general findings of the paper are that while feature visualizations help users get slightly higher accuracy than having no reference at all, there is no significant difference between synthetic visualizations and the highly activating natural references, and furthermore that user behavior is highly similar to the performance of a salience baseline that predicts user gaze on an image.\n", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:  Low. This work is similar to Borowski et al. 2021, but it does a more thorough evaluation encompassing more types of feature visualizations and more baselines.  It also uses an experimental design that is likely to be more powerful, as it compares two images that are obtained by occluding the same original image.  However it is not entirely clear that this methodological change is a strict improvement on Borowski et al. 2021 because such occluded distributions lie outside of the support of the natural image distribution and hence could fail to be representative of neural network activity for natural images, unless the network has been shown to be robust to occlusions.\n\nQuality:  High. A lot of thought was put into the experimental design and the analysis of the conclusions.  In particular, decomposing the variation in accuracy by no only visualization method but also unit and contrast in activation helps build the case that the visualizations do not differ by much.  The details of collecting data on MTurk also build confidence in data quality.\n\nClarity:  The paper is clear and details of the methods are well-motivated and explained.\n\nSignificance:  Moderate.  The paper adds to the findings of Borowski et al. which pose serious questions for the feature visualization field.  Why are human rankings of feature activations so close to chance, and why don't visualizations (natural or synthetic) help more?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.\n\nIt is clear that this work is only a stepping stone, and that more studies are needed to investigate why human predictions of unit activations are so poor, and whether new tools can be developed to increase our ability to make predictions of neural network unit activations.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates whether feature visualizations, such as activation maximization, provide a benefit to humans in predicting model performance over simpler approaches such as the maximally-activating data samples. By performing a set of human experiments, the authors show that a) there is a benefit to these visualizations over no visualizations, but that b) there is no difference between these approaches and the use of natural images. Reviewers all found the experiments to be very carefully designed and rigorous, and praised the overall clarity of the paper. There is certainly a need for more thorough and careful examination of the interpretability methods that are widely used, and I think this paper will prove impactful both because of its observations and as a model for how to conduct such evaluations. I recommend it be accepted as a spotlight. 
Decision: Accept (Spotlight)