Exemplary Natural Images Explain CNN Activations Better than State-of-the-Art Feature Visualization
Authors: Judy Borowski
Roland Simon Zimmermann
Judith Schepers
Robert Geirhos
Thomas S. A. Wallis
Matthias Bethge
Wieland Brendel
Venue: ICLR 2021 Poster (2021)
Keywords: evaluation of interpretability
feature visualization
activation maximization
human psychophysics
understanding CNNs
explanation method

Abstract: Feature visualizations such as synthetic maximally activating images are a widely used explanation method to better understand the information processing of convolutional neural networks (CNNs). At the same time, there are concerns that these visualizations might not accurately represent CNNs' inner workings. Here, we measure how much extremely activating images help humans to predict CNN activations.
Using a well-controlled psychophysical paradigm, we compare the informativeness of synthetic images by Olah et al. (2017) with a simple baseline visualization, namely exemplary natural images that also strongly activate a specific feature map. Given either synthetic or natural reference images, human participants choose which of two query images leads to strong positive activation. The experiment is designed to maximize participants' performance, and is the first to probe intermediate instead of final layer representations. We find that synthetic images indeed provide helpful information about feature map activations ($82\pm4\%$ accuracy; chance would be $50\%$). However, natural images --- originally intended to be a baseline --- outperform these synthetic images by a wide margin ($92\pm2\%$). Additionally, participants are faster and more confident for natural images, whereas subjective impressions about the interpretability of the feature visualizations by Olah et al. (2017) are mixed. The higher informativeness of natural images holds across most layers, for both expert and lay participants as well as for hand- and randomly-picked feature visualizations. Even if only a single reference image is given, synthetic images provide less information than natural images ($65\pm5\%$ vs. $73\pm4\%$). In summary, synthetic images from a popular feature visualization method are significantly less informative for assessing CNN activations than natural images. We argue that visualization methods should improve over this simple baseline.

Published as a conference paper at ICLR 2021
EXEMPLARY NATURAL IMAGES EXPLAIN CNN ACTI- VATIONS BETTER THAN STATE-OF-THES FEATURE VISUALIZATION
Judy Borowski", Roland S. Zimmermann', Judith Schepers, Robert Geirhos, Thomas S. A. Wallis Matthias Bethget, Wieland Brendel* University of Tübingen, Germany
ABSTRACT
Feature visualizations such as synthetic maximally activating images are a widely used explanation method better understand the information processing of con- volutional neural networks (CNNs). At the same time, there are concerns that these visualizations might not accurately represent CNNs' inner workings. Here, we measure how much extremely activating images help humans to predict CNN activations. Using a well-controleed psychophysical paradigm, we compare the informativeness of synthetic images by Olah et al.] (2017) with simple base- line visualization, namely exemplary natural images that also strongly activate a specific feature map. Given either synthetic or natural reference images, human participants choose which of two query images leads to strong positive activa- tion. The experiment is designed to maximize participants' performance, and is the first to probe intermediate instead of final layer representations. We find that synthetic images indeed provide helpful information about feature map activations (82 + 4% accuracy; chance would be 50%). However, natural images - origi- nally intended to be a baseline outperform these synthetic images by wide margin (92 + 2%). Additionally, participants are faster and more confident for natural images, whereas subjective impressions about the interpretability of the feature visualizations by Olah et al. (2017) are mixed. The higher informativeness of natural images holds across most layers, for both expert and lay participants as well as for hand- and randomly-picken
feature visualizations. Even if only a single reference image is given, synthetic images provide less information than natural images (65 = 5% vs. 73 + 4%). In summary, synthetic images from pop- ular feature visualization method are significantly less informative for assessing CNN activations than natural images. We argue that visualization methods should improve over this simple baseline.
INTRODUCTION
As Deep Learning methods are being deployed across society, academia and industry, the need understand their decisions becomes ever more pressing. Under certain conditions, "right to explanation" is even required by law in the European Union (GDPR 2016 Goodman & Flaxman
2017). Fortunately, the field of interpretability
explainable artificial intelligence (XAI) is also growing: Not only are discussions on goals and definitions of interpretability advancing (Doshi-
Velez & Kim 2017: Lipton 2018 Gilpin et al. 20188; Murdoch et al. 2019 Miller 2019 Samek et al. (2200) but the number of explanation methods is rising, their maturity is evol violving (Zeiler &
Fergus 2014 Ribeiro et al. 2016 Selvaraju et al. 2017 Kim et al. 2018 and they are tested and
"Joint first and corresponding authors: firstname Last eesttaaneuurtttuuuiignnn....
1 'Current affiliation: Institute Psychology and Center for Cognitive Science, Technische Universität
Darmstadt
Joint senior authors

---

Published as a conference paper at ICLR 2021
A
Given these reference images...
...which image is strongly activating?
B
Synthetic images are helpful
1.0
Natural eien
Maximally Activating
Propartion Correct
0.8
Choose
Minimally Activating
0.6
Chance
Synthetic
Queries
Synthetic
Natural
Figure 1: How useful are synthetic compared to natural images for interpreting neural network activations? A: Human experiment. Given extremely activating reference images (either synthetic or natural), human participant chooses which out of two query images is also a strongly activating image. Synthetic images were generated via feature visualization (Olah et al. 2017). B: Core result. Participants are well above chance for synthetic images but even better when seeing natural reference images.
used in real- world scenarios like medicine (Cai al., 2019 Kröll et al.) 2020) and meteorology (Ebert-Uphoff & Hilburn 2020).
We here focus on the popular post-hoc explanation method (or interpretability method) of feature visualizations via activation maximization First introduced by Erhan et al. (2009) and subsequently improved by many others (Mahendran & Vedaldi 2015: Nguyen et al., 2015; Mordvintsev et al,, 2015 Nguyen et al. 20166a 2017), these synthetic, maximally activating images seek to visualize features that a specific network unit, feature map or a combination thereof is selective for. However, feature visualizations are surrounded by great controversy: How accurately do they represent CNN's inner workings- or in short, how useful are they? This is the guiding question of our study.
On the hand, many researchers (Graetz 2019) and that "features can be rigorously studied and understood" (Olah et al,, 20206). Also other applications from Computer Vision and Natural Language Processing support the view that features are meaningful (Mikolov et al.) [2013] Karpathy et al. 20155] Radford et al. 2017 Zhou
convinced that feature visualizations
interpretable
2014 Bau al.] 2017 (0200) and might formed in hierarchical fashion (LeCun et al. Güçlü & van Gerven 2013 Gooddellow et al., 2016). Over the past few years, extensive investigations better understand CNNs are based on feature visualizations (Olah et al., 2020b a
al.
2015
Cammarata et al. 22200] Cadena et al. 2018). and the technique is being combined with other ex- planation methods (Olah et al.) 2018 2019).
Carter et al. 2019: Addepalli et al. 2020 Hohman et al.,
On the other hand, feature visualizations can be equal parts art and engineering as they are science: vanilla methods look noisy, thus human-defined regularization mechanisms are introduced. But do the resulting beautiful visualizations accurately show what a CNN is selective for? How represen- tative the seemingly well- interpretable, "hand-picked" (Olah et al. 2017) synthetic images in publications for the entirety all units in a network, a concern raised by e.g. Kriegeskorte/ (2013)?
might be sug- gested by the existence of adversarial examples (Szegedy et al. 2013 Ilyas et al. 2019)? Morcos suggest that units of easily understandable features play a less important role in
What if the features that CNN is truly sensitive are imperceptible instead,
et al. (2018) a network. Another criticism of synthetic maximally activating images is that they only visualize extreme features, while potentially leaving other features undetected that only elicit e.g. 70% of the maximal activation. Also, polysemantic units (Olah et al., 2020b), i.e. units that are highly activated by different semantic concepts, as well as the importance of combinations of units (Olah al., 2017] 2018 Fong & Vedaldi, 2018) already hint at the complexity of how concepts are encoded in CNNs.
One way to advance this debate is measure the utility of feature visualizations terms of their helpfulness for humons. In this study, we therefore design well-controlled psychophysical experi- ments that aim to quantify the informativeness of the popular visualization method by Olah et al. 2017 Specifically, participants choose which of two natural images would elicit a higher activa-
Also known as input maximization maximally exciting images (MEls).
2

---

Published as a conference paper at ICLR 2021
tion in CNN given set of reference images that visualize the network selectivities. We natural query images because real-world applications of XAI require understanding model decisions to nat- ural inputs. To the best of our knowledge, study is the first to probe how well humans can predict intermediate CNN activations. Our data shows that:
Synthetic images provide humans with helpful information about feature map activations.
Exemplary natural images even more helpful.
The superiority of natural images mostly holds across the network and various conditions.
Subjective impressions of the interpretability of the synthetic visualizations vary greatly between participants.
2
RELATED WORK
Significant progress has been made in recent years towards understanding CNNs for image data. Here, we mention a few selected methods as examples of the plethora of approaches for under- standing CNN decision- cation decision (Springenberg al., 2014 Bach et al., 2015) [Smilkvv et al., 2017 Zintgraf et al,, 2017). concept activation vectors show a model's sensitivity to human-defined concepts (Kim et al., 2018), and other methods- amongst feature visualizations focus on explaining individual units
Saliency maps show the importance of each pixel to the classifi-
(Bau et al.) 2020). Some tools integrate interactive, software-like aspects (Hohman et al., 2019
Wang et al., 2020 Carter et al. 2019 Collaris & van Wijk 2020 OpenAl 2020), combine more than explanation method (Shi al. 2020; Addepalli
al. 2000)) or make progress towards automated explanation methods (Lapuschkin et al. 2019: Ghorbani et al. 2019). As overviews, we
recommend Gilpin et al. (2018) Zhang & Zhu (2018); Montavon et al. (2018) and Carvalho et al.
(011
Despite their great insights, challenges for explanation methods remain. Oftentimes, these tech- niques are criticized as being over-engineered; regarding feature visualizations, this concerns the loss function and techniques to make the synthetic images look interpretable (Nguyen al., 2017). Another critique is that interpretability research is not sufficiently tested against falsifiable hypothe- ses and rather relies too much on intuition (Leavitt Morcos 2020).
In order to further advance XAI, scientists advocate different directions. Besides the focus on devel- oping additional methods, some researchers (e.g- Olah al. (2020b)) promote the "natural science" approach, i.e. studying neural network extensively and making empirical claims until falsification. Yet another direction is to quantitatively evaluate explanation methods. So far, only decision- explanation methods have been studied in this regard. Quantitative evaluations can either be real- ized with humans directly or with mathematically grouunddd models as an approximation for human perception. Many of the latter approaches show great insights (c.g. Hooker et al. (2019); ; Nguyen &
Martúnez (2020); Fel & Vigouroux| (2020): Lin et al.) (2020); Tritscher et al. (2020): Tjoa & Guan (2020)) However, recent study demonstrates that metrics of the explanation quality computed without human judgment are inconclusive and do not correspond to the human rankings (Biess-
mann & Refiano, 2019). Additionally, Miller (2019) emphasizes that XAI should build on existing research in philosophy, cognitive science and social psychology.
The body of literature on human evaluations of explanation methods is growing: Various combi- nations of data types (tabular, text, static images), task set-ups and participant pools (experts vs. laypeople, on-site vs. crowd-sourcing) are being explored. However, these studies all aim inves- tigate final model decisions and do not probe intermediate activations like our experiments do. For a detailed table of related studies, see Appendix See. A.3] A commonly employed task paradigm the "forward simulation / prediction" task, first introduced by Doshi- & Kim (2017): Partici- pants guess the model's computation based on an input and an explanation. As there is no absolute metric for the goodness of explanation methods (yet), comparisons are always performed within studies, typically against baselines. The same holds for additional data collected for confidence or trust ratings. According to the current literature, studies reporting positive effects of explanations (e.g. Kumarakulasinghe et al. (2020)) slightly outweigh those reporting inconclusive (c.g. Alufaisan et al. 20200; Chu et al. (2020) even negative effects (e.g. [Shen & Huang 2020
3

---

Published as a conference paper at ICLR 2021
Which image from the center is also a strongly activating image?
more cont
Minimally Activating
Maximally Activating
Figure 2: Example trial in psychophysical experiments. A participant is shown minimally and maximally activating reference images for certain feature map the sides and is asked to select the image from the center that also strongly activates that feature map. The answer is given by clicking on the number according to the participant's confidence level (1: not confident, 2: somewhat confident, 3: very confident). After each trial, the participant receives feedback which image was indeed the maximally activating one. For screenshots of each step in the task, see Appendix Fig. 7
To our knowledge, no study has yet evaluated the popular explanation method of feature visualiza- tions and how it improves human understanding of intermediate network activations. This study therefore closes an important gap: By presenting data for a forward prediction task CNN, we provide quantitative estimate of the informativeness of maximally activating images generated with the method of Olah et al.] (2017). Furthermore, our experiments are unique as they probe for the first time how well humans can predict intermediate model activations.
METHODS
We perform two human psychophysical studies" with different foci (Experiment (N = 10) and Experiment II (N = 23)). In both studies, the task is to choose the one image out of two natural query images (two-alternative forced choice paradigm) that the participant considers to also elicit strong activation given some reference images (see Fig. 2 Apart from the image choice, we record the participant's confidence level and reaction time. Specifically, responses are given by clicking on the confidence levels belonging to either query image. order to gain insights into how intuitive participants find feature visualizations, their subjective judgments are collected in separate task and a dynamic conversation after the experiment (for details, see Appendix Sec. A.I.I and Appendix Sec. A.2.6).
All design choices are made with two main goals: (1) allowing participants to achieve the best performance possible to approximate upper bound on the helpfulness of the explanation method, and (2) gaining a general impression of the helpfulness choose the natural query images from among those of lowest and highest activations (- best possible performance) and test many different feature maps across the network (-+ generality). For more details the human experiment besides the ones below, Appendix Sec. A.1
the examined method. As an example,
In Experiment I, we focus on comparing the performance of synthetic images to two baseline con- ditions: natural reference images and no reference images. In Experiment II, we compare lay vs. expert participants as well as different presentation schemes of reference images. Expert participants qualify by being familiar or having practical experience with feature visualization techniques or at least CNNs. Regarding presentation schemes, we vary whether only maximally or both maximally and minimally activating images are shown; as well as how many example images of each of these are presented (1 or 9).
Following the existing work on feature visualization OOlh et Inception V1 network Szegedy et al. 2015) trained on ImageNet (Deng et a.. 2009 Russakovsky
2017
2018
2020b a). use an
"Code and data available at httpss////egggggggggghobbbb//b/
vi visual zzt i
also known GoogLeNee

---

Published as a conference paper at ICLR 2021
A
B
Synthetic
17.
0.003
0.7
0.03
Proportion of Correct Trials
15.0
0.005
0.6
12.5
Proportion Correct
0.8
0.5
Reaction Time
0.4
0.5
Steree
0.3
0.4
5.0
0.2
0.1
0.0
Symhetic
Natural
None
Synthetic Natural
Reference Images
Confidence Rating
Reference Images
Figure 3: Participants are better, more confident and faster at judging which of two query images causes higher feature map activation with natural than with synthetic reference images. A: Perfor- mance. Given synthetic reference images, participants are well above chance (proportion correct: 82 + 44), but even better for natural reference images (92 2%). Without reference images (base- line comparison "None"), participants are close to chance. B: Confidence. Participants are much more confident (higher rating more confident) for natural than for synthetic images on correctly answered trials (x" p .001). C: Reaction time. For correctly answered trials, participants are on average faster when presented with natural than with synthetic reference images. We show addi- tional plots confidence and reaction time for incorrectly answered trials and all trials in the Ap- pendix (Fig. 16); for Experiment II, see Fig. 17 The p-values A and correspond to Wilcoxon signed-rank tests.
al. 2015). The synthetic images throughout this study are the optimization results of the feature visualization method by Olah et al. (2017) with the spatial average of a whole feature map ("channel objective"). The natural stimuli are selected from the validation set of the ImageNet ILSVRC 2012 dataset (Russakovsky et al. 2013) according to their activations for the feature map of interest. Specifically, the images of the most extreme activations or expert participant sees different query and reference images. A more detailed description of the specific sampling process for natural stimuli and the generation process of synthetic stimuli given in Sec. A.1.2
sampled, while ensuring that each lay
4 RESULTS
In this section, all figures show data from Experiment I except for Fig. 5 which show data from Experiment II. All figures for Experiment II, which replicate the findings of Experiment I, as well as additional figures for Experiment I (such as by-feature-map analysis), can be found in the Appendix Sec. A.2 Note that (unless explicitly noted otherwise), eттоr bars denote two standard errors of the mean of the participant average metric.
4.1 PARTICIPANTS ARE BETTER, MORE CONFIDENT AND FASTER WITH NATURAL IMAGES
Synthetic images can be helpful: Given synthetic reference images generated via feature visualiza- tion (Olah et al. 2017), participants are able to predict whether certain network feature map prefers over the other query image with accuracy of 82 4%, which is well above chance level (50%) (see Fig. 3A). However, performance is even higher in what we intended to the baseline condi- tion: natural reference images (92 2%). Additionally. for correct answers, participants much more frequently report being highly certain on natural relative to synthetic trials (see Fig. (33), and their average reaction time is approximately 3.7 seconds faster when seeing natural than synthetic refer- ence images (see Fig. 3C). Taken together, these findings indicate that in our setup, participants are confident and substantially faster natural images.
just better overall, but also
4.2 NATURAL IMAGES ARE MORE HELPFUL ACROSS A BROAD RANGE OF LAYERS
Next, take more fine-grained look at performance across different layers and branches of the Inception modules (see Fig. H.. Generally, feature map visualizations from lower layers show low- level features such striped patterns, color texture, whereas feature map visualizations from
5

---

Published as a conference paper at ICLR 2021
1.0
1.0
Synthelic Natural
Proportion Correct
Proportion
0.7
0.7
Charce
Chance
65
56
1xt
3e3
Layer
Branch
Inception Module
Figure 4: Performance is high across (A) broad range of layers and (B) all branches of the Inception modules. The latter differ their kernel sizes (1 1, 3 x 3, 5 x 5, pool). Again, natural images are (mostly) more helpful than synthetic images. Additional plots for the none condition as well as Experiment can be found in the Appendix respectively Fig. and Fig. 19
higher layers tend to show more high-level concepts like (parts of) objects (LeCun al. 2015 Güçlü & van Gerven 2013 Goodfellow et al. 2016). We find performance to be reasonably high across most layers and branches: participants are able to match both low-level and high-level patterns (despite not being explicitly instructed what layer a feature map belonged to). Again, natural images are mostly more helpful than synthetic images.
4.3 FOR EXPERT AND LAY PARTICIPANTS ALIKE: NATURAL IMAGES ARE MORE HELLPUUL
Explanation methods seek to explain aspects algorithmic decision-making. Importantly, an expla- nation should not just be amenable to experts but to anyone affected by an algorithm's 's decision. We here test whether the explanation method feature visualization equally applicable to expert and lay participants (see Fig. 5A). Contrary to prior expectation, we find significant differences in expert vs. lay performance (RM ANOVA, .44, for details see Appendix Sec. A.2.2). Hence, extensive experience with CNNs is not necessary perform well in this forward simulation task. In line with the previous main finding, both experts and lay participants are both better in the natural than in the synthetic condition.
4.4 EVEN FOR HAND-PICKED FEATURE VISUALIZATIONS, PERFORMANCE IS HIGHER ON NATURAL IMAGES
Often, explanation methods are presented using carefully selected network units, raising the ques- tion whether author-chosen units representative for the interpretability method as a whole. Olah et al. (017)) identify number of particularly interpretable feature maps in Inception V1 in their appendix overview. When presenting either these hand-picked visualization: or randomly selected ones, performance for hand-picke feature maps improves slightly (Fig. 5]; however this perfor- mance difference small and not significant for both natural (Wilcoxon test, p = .59) and synthetic (Wilcoxon test, p .18) reference images (see Appendix Sec. [.2.44 for further analysis). Consis- tent with the findings reported above, performance higher for natural than for synthetic reference images even carefully selected hand hand-picked feature maps.
4.5 ADDITIONAL INFORMATION BOOSTS PERFORMANCE, ESPECIALLY FOR NATURAL IMAGES
Publications on feature visualizations vary terms of how optimized images are presented: Often, a single maximally activating image is shown (e.g. Erhan et al. (2009): Carter et al. (2099 Olah al. (2018)); sometimes a few images are shown simultaneously (e.g. Yosinski et al. (2015): Nguyen al. (0166))) and occasion both maximally and minimally activating images are shown in unison (Olah et al. (017)). Naturally, the question arises as to what influence (if any) these choices have, and whether there is optimal way of presenting extremely activating images. For this reason, we systematically compare approaches along two dimensions: the number of reference images (1 vs. 9) and the availability of minimally activating images (only Max vs. Min+Max). The results can
All our hand- hand-picked feature maps are taken from the pooling branch of the Inception module. As the pendix overview in Olah al. (2017) does not contain one feature map for each of these, select interpretable feature maps for the missing layers mixed5a and mixed5h ourselves.

---

Published as a conference paper at ICLR 2021
Proportion Correct A
Proportion Correct to
Proportion Carnect
0.9-
0.3
0.9
0.8
0.8
0,)
0.7
0.7-
0.5
0.6
6.-
Cherc
Drncc
derce
Expent
Eapin
Ranbom
Hano:
Rundbin
Hand:
Max
Hax
Minstor
Max
Hax
Miv+H
Movthor
Potkes
Synthetic
Natural
Synchet
Natural
Syntheai
Natural
Expert Level
Selection Mode
Presentation Scheme
Figure 5: We found no evidence for large effects of expert level or feature map selection. However, performance does improve with additional information. A: Expert level. Both experts and lay participants perform equally well (RM ANOVA. p = .44), and consistently better on natural than on synthetic images. B: Selection mode. There is significant performance difference between hand- picked feature maps selected for interpretability and randomly selected ones (Wilcoxon test, p = .18 for synthetic and p = .59 for natural reference images). C: Presentation scheme. Presenting both maximally and minimally activating images simultancously (Min+Max)) and presenting nine instead of one single reference image tend improve performance, especially for natural reference images. "ns" highlights non- con-signifi differences.
be found in Fig. Fig.5C. When just a single maximally activating image is presented (condition Max I), natural images already outperform synthetic images (73 + 4% vs. 64 + 5%). With additional information along either dimension, performance improves both for natural as well as for synthetic images. The stronger boost in performance, however, is observed for natural reference images. In fact, performance is higher for natural than for synthetic reference images in all four conditions. In the Min+Max 9 condition, replication of the result from Experiment shown in Fig. BA, natural images now outperform synthetic images by an even larger margin (91 + 3 vs. 72 - 4%).
4.6 SUBJECTIVELY, INTERPRETABILIN OF FEATURE VISUALIZATIONS VARIES GREATLY
While our data suggests that feature visualizations are indeed helpful for humans to predict CNN activations, we want to emphasize again that our design choices aim at an upper bound on their informativeness. Another important aspect of evaluating an explanation method is the subjective impression. Besides recording confidence ratings and reaction times, we collect judgments intu- itiveness trials (see Appendix Fig. [4] and oral impressions after the experiments. The former ask for ratings of how intuitive feature visualizations appear for natural images. As Fig. 6 show, participants perceive the intuitiveness of synthetic feature visualizations for strongly activating nat- ural dataset images very differently. Further, the comparison of intuitiveness judgments before and after the main experiments reveals only a small significant average improvement for one out of three feature maps (see Fig. 6B+C, Wilcoxon test, p < .001 for mixed4b). The interactive conversations paint similar picture: Some synthetic feature visualizations are perceived as intuitive while others do not correspond to understandable concepts. Nonetheless, four participants report that their first "gut feeling" for interpreting these reference images (as one participant phrased it) is more reliable. Further, few participants point out that the synthetic visualizations are exhausting to understand. Finally, three participants additionally emphasize that the minimally activating reference images played an important role in their decision-making.
In by-feature-map analysis (see Appendix A.2.7 for details and images, as well as Supplementary Material [ for more images), we compare differences and commonalitics for feature maps of dif- ferent performance levels. According object parts or shapes. In contrast, difficult feature maps seem to have diverse reference images, features that do not correspond to human concepts, contain conflicting information commonalities between query and reference images matter more. Bluntly speaking, we are also often surprised that participants identified the correct image- the reasons for this unclear us.
observations, easy feature maps seem to contain clear
which
DISCUSSION & CONCLUSION
Feature visualizations such synthetic maximally activating images widely used explana- tion method, but it is unclear whether they indeed help humans to understand CNNs. Using well- controlled psychophysical experiments with both expert and lay participants, here conduct the

---

Published as a conference paper at ICLR 2021
A
B
Difference in Intuitivenes
200
90-
150
Intuitiveness
Intuitiveness
100
50
50
-50
100
-100
Layer
Layer
Layer
Figure 6: The subjective intuitiveness of feature visualizations varies greatly (see A for the ratings from the beginning of Experiment I and B for the ratings at the beginning and end of Experiment II). The means over all participants yield a neutral result, i.e. the visualizations are neither un- nor intuitive, and the improvement of subjective intuitiveness before and after the experiment is only significant for one feature map (mixed4b). C: On average, participants found feature visualizations slightly more intuitive after doing the experiment as the differences larger than zero show. In all three subfigures, gray dots and lines show data per participant.
very first investigation of intermediate synthetic feature visualizations by Olah et al. (2017): Can participants predict which of two query images leads strong activation in feature map, given extremely activating visualizations? Specifically, we shed light on the following questions:
(1.) How informative are synthetic feature visualizations- and how do they compare natural image baseline? We find above-chance performance given synthetic feature visualizations, but to own surprise, synthetic feature visualizations are systematically less informative than the simple baseline of strongly activating natural images. Interestingly, many synthetic feature visualizations contain regularization mechanisms introduce more "natural structure" (Olah al. 2017). some-
times even called a "natural image prior" (Mahendran & Vedaldi 2015: Offert & Bell | (020).. This
raises the question: Are natural images maybe all you need? One might posit that extremely acti- vating natural (reference) images would have an unfair advantage because we also test on extremely activating natural (query) images. However, our task design ultimately reflects that XAI mainly concerned with explaining how units behave on natural inputs. Furthermore, the fact that feature visualization are not bound the natural image manifold often claimed as an advantage because it supposedly allows them to capture more precisely which features a unit is sensitive to (Olah et al.) 2017]. Our results, though, demonstrate that this is not case if we want to understand the behav- 1or of units on natural inputs.
(2.) Do you need to be a CNN expert in order to understand feature visualizations? To the best of our knowledge, our study is the first to compare the performances of expert and lay people when eval- uating explanation methods. Previously, publications either focused on only expert groups (Hase & Bansal 2020 Kumarakulasingle
al. 2020) or only laypeople (Schmidt & Biessmann 2019|[A1- ufaisan et al. 2020). Our experiment shows no significant difference between expert and lay partic- ipants in our task both perform similarly well, and even better on natural images: replication of our main finding. While a few caveats remain when moving an experiment from the well-controlled lab a crowdsoureing platform (Haghiri et al. 2019). this suggests that future studies may not have to rely on selected expert participants, but may leverage larger lay participant pools.
(3.) Are hand-picked synthetic feature visualizations representative? ? An open question was whether the visualizations shown in publications represent the general interpretability of feature visualiza- tions (a concern voiced by e.g. KKiieeskorrte 2015). even though they are hand-picked (Olah al.) 2017). Our finding that there is no large difference in performance between hand- and randomly- picked feature visualizations suggests that this aspect is minor.
(4.) What is the best way of presenting images? Existing work suggests that more than one example OOffrtt 2017) and particularly negative examples (Kim et al., (0166) enhance human understanding of data distributions. Our systematic exploration of presentation schemes provides evidence that increasing the number of reference images as well as presenting both minimally and maximally activating reference images (as opposed to only maximally activating ones) improve human per- formance. This finding might be of interest to future studies aiming at peak performance or for developing software for understanding CNNs.
8

---

Published as a conference paper at ICLR 2021
(5.) How do humans subjectively perceive feature visualizations? Apart from the high informa- tiveness of explanations, another relevant question is how much trust humans have in them. In our experiment, find that subjective impressions of how reasonable synthetic feature visualizations are for explaining responses to natural images vary greatly. This finding is in line with Hase & (020)) who evaluated explanation methods on text and tabular data.
Bansal
Caveats. Despite our best intentions, a few caveats remain: The forward simulation paradigm is only one specific way to measure the informativeness of explanation methods, but does not allow us to make judgments about their helpfulness in other applications such as comparing different CNNs. Further, we emphasize that all experimental design choices were made with the goal to measure the best possible performance. As consequence, our finding that synthetic reference images help humans predict network's strongly activating image may not necessarily be representative of a less optimal experimental set-up with e.g. query images corresponding to less extreme feature map activations. Knobs to further de- or increase participant performance remain (e.g. parammer choices could be tuned to layers). Finally, while we explored one particular method in depth (Olah ct al. 2017); it remains an open question whether the results can be replicated for other feature visualizations methods.
Future directions. We see many promising future directions. For one, the current study uses query images from extreme opposite ends of feature map's activation spectrum. For more fine-grained measure of informativeness, we will study query images that elicit more similar activations. Addi- tionally, future participants could be provided with even more information-such as, for example, where a feature map is located in the network. Furthermore, it has been suggested that the combina- tion of synthetic and natural reference images might provide synergistic information to participants Olla et al. 2017), which could again be studied in our experimental paradigm. Finally, further studies could explore single neuron- centereeeneeeed feature visualizations, combinations of units as well as different network architectures.
Taken together, our results highlight the need for thorough human quantitative evaluations feature visualizations and suggest that example natural images provide a surprisingly challenging baseline for understanding CNN activations.
AUTHOR CONTRIBUTIONS
The initiative of investigating human predictability CNN activations came from WB. JB, WB, MB and TSAW jointly combined it with the idea of investigating human interpretability of feature visualizations. JB led the project. JB, RSZ and JS jointly designed and implemented the experiments (with advice and feedback from TSAW, RG, MB and WB). The data analysis was performed by JB and RSZ (with advice and feedback from RG, TSAW, MB and WB). JB designed, and JB and JS implemented the pilot study. JB conducted the experiments (with help from JS). RSZ performed the statistical significance tests (with advice from TSAW and feedback from JB and RG). MB helped shape the bigger picture and initiated intuitiveness trials. WB provided day-to-day supervision. JB, RSZ and RG wrote the initial version of the manuscript. All authors contributed the final version of the manuscript.
ACKNOWLEDGMENTS
We thank Felix A. Wichmann and Isabel Valera for helpful discussions. We further thank Alexander Böttcher and Stefan Sietzen for support as well as helfpul discussions on technical details. Additionally, we thank Chris Olah for clarifications via glack..dist. distiissiii iestiillppu Moreover, thank Leon Sixt for valuable feedback on the introduction and related work. From our lab, we thank Matthias Kümmerer, Matthias Tangemann, Evgenia Rusak and Ori Press for helping in piloting experiments, as well as feedback from Evgenia Rusak, Claudio Michaelis, Dylan Paiton and Matthias Kümmerer. And finally, we thank all our participants for taking part in experiments.
We thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS IS)) for supporting JB, RZ and RG. We acknowledge support from the German Federal Ministry of Education and Research (BMBF) through the Competence Center for Machine Learning ATUE.AI, FKZ OIIS 18039A) and the Bernstein Com- putational Neuroscience Program Tübingen (FKZ: 01GQ1002), the Cluster of Excellence Machine Learning: New Perspectives for Sciences (EXC2064/1), and the German Research Foundation (DFG: SFB 1233, Robust Vision: Inference Principles and Neural Mechanisms, TP3, project numher 276693517).

---

Published as a conference paper at ICLR 2021
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangging Jia, Rafal Jazefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg. Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, llya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wat- tenberg. Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learn- ing on heterogeneous systems, 2015. URL hitps: https//wwwwwensrfllo...oooo available from tensorflow.org.
Software
Sravanti Addepalli, Dipesh Tamboli, R Venkatesh Babu, and Biplab Banerjee. Saliency- class impressions for feature visualization of deep neural networks. arXiv preprint arXiv:2007.15861 2020.
Ahmed Alqaraawi, Martin Schuessler, Philipp Weiß, Enrico Costanza, and Nadia Berthouze. Eval- uating saliency map explanations for convolutional neural networks: user study. In Proceedings of the 25th International Conference on Intelligent User Interfaces, pp. 275-285, 2020.
Yasmeen Alufaisan, Laura R Marusich, Jonathan Z Bakdash, Yan Zhou, and Murat Kantar- cioglu. Does explainable artificial intelligence improve human decision-making? arXiv preprint arXiv:2006. 11144, 2020.
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Rober Müller, and Wojeiech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise -wiee relevance propagation. PloS one, 10(7)ee0130140, 2015.
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In Proceedings the IEEE conference on computer vision and pattern recognition, pp. 6541-6549, 2017.
David Bau, Jun Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, and Antonio Torralba. Understanding the role of individual units deep neural network. Proceedings of the National Academy of Sciences, 2020.
Felix Biessmann and Dionysius Irza Refiano. A psychophysics approach for quantitative comparison of interpretable computer vision models. arXiv preprini arXiv: 1912.0001 2019.
Santiago A Cadena, Marissa A Weis, Leon A Gatys, Matthias Bethge, and Alexander S Ecker. Diverse feature visualizations reveal invariances in early layers of deep neural networks. Pro- ceedings of the European Conference Computer Vision (ECCV), pp. 217-232, 2018.
Carrie J Cai, Emily Reif, Narayan Hegde, Jason Hipp, Been Kim, Daniel Smilkov, Martin Wat- tenberg, Fernanda Viegas, Greg S Corrado, Martin C Stumpe, et al. Human-centered tools for coping with imperfect algorithms during medical decision-making. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pp. 1-14, 2019.
Nick Cammarata, Gabriel Goh, Shan Carter, Ludwig Schubert, Michael Petrov, and Chris Olah. Curve detectors. Distill, 2020. doi: 223915//istill.00024.000 http:///shlllbbooo/00/c/eelellee
Shan Carter, Zan Armstrong, Ludwig Schubert, Ian Johnson, and Chris Olah. Activation atlas. wwtpllltp//////////iii/
Distill, 2019. doi: 10.2391 1/2991//dsiill.000055
Diogo V Carvalho, Eduardo M Pereira, and Jaime S Cardoso. Machine learning interpretability: A survey on methods and metrics. Electronics, 8(8):332, 2019.
Arjun Chandrasekaran, Deshraj Yadav, Prithvijit Chattopadhyay, Viraj Prabhu, and Devi Parikh. It takes two to tango: Towards theory of ai's mind. arXiv preprint / 2017.
Eric Chu, Deb Roy, and Jacob Andreas. Are visual explanations useful? a case study in model-in- the-loop prediction, arXiv preprint ar0iv:2007. 1 12248, 2020.
10

---

Published as a conference paper at ICLR 2021
Dennis Collaris and Jarke J van Wijk. Explainexplore: Visual exploration of machine learning explanations. In 2020 IEEE Pacific Visualization Symposium (Pacific pp. 26-35. IEEE, 2020.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Seake Scale Hierarchical Image Database. In CVPR09, 2009.
Jürgen Dieber and Sabrina Kirrane. Why model why? assessing the strengths and limitations of lime. arXiv preprini urXiv:2012.00093, 2020.
Jonathan Dinu, Jeffrey Bigham, and Zico Kolter. Challenging common interpretability assump- tions in feature attribution explanations. arXiv preprini arXiv:2012.02745, 2020.
Finale Doshi- Velez and Been Kim. Towards rigorous science of interpretable machine learning. arXiv preprint arXiv: / 770..06002... 2017.
Imme Ebert-Uphoff and Kyle Hilburn. Evaluation, tuning and interpretation of neural networks for working with images in meteorological applications. Bulletin of the American Meteorological Society, pp. 1-49, 2020.
Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher-layer features of a deep network. University of Montreal, 1341(3):1, 2009.
Thomas Fel and David Vigouroux. Representativity and consistency measures for deep neural net- work explanations. arXiv preprint (tt::2000900452/1/ 2020.
Ruth Fong and Andrea Vedaldi. Net2vec: Quantifying and explaining how concepts are encoded by filters in deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8730-8738, 2018.
GDPR. Regulation (EU) 2016/679 the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement such data, and repealing Directive 95/46. Official Journal of the European Union (OJ), 5944 2016.
Amirata Ghorbani, James Wexler, James Y Zou, and Been Kim. Towards automatic concept-based explanations. In Advances in Neural Information Processing Systems, pp. 9277-9288 2019.
Leilani H Gilpin, David Bau, Ben Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal. Explaining explanations: An overview of interpretability of machine learning, In 2018 IEEE 5th International Conference on data science and advanced analytics (DSAA), pp. 80-89, IEEE, 2018.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http: 7 /ww.eeepleamm..ggooooorgg
Bryce Goodman and Seth Flaxman. European Union regulations on algorithmic decision-making and "right to explanation". Al magazine, 38(3):50-555 57, 2017.
Fabio M. Graetz. How to visualize convolutional features in 40 lines of code, Jan 2019, URL
convolut ional
Umut Güiçü and Marcel AJ van Gerven. Deep neural networks reveal a gradient in the complexity of neural representations across the ventral stream. Journal of Neuroscience, 10005-010005100114 2015.
Siavash Haghiri, Patricia Rubisch, Robert Geirhos, Felix Wichmann, and Ulrike von Luxburg. Comparison- based framework for psychophysics: Lab versus crowdsourcing. arXiv preprint 1905.0/905.0720 2019.
Peter Hase and Mohit Bansal. Evaluating explainable ai: Which algorithmic explanations help users predict model behavior? arXiv preprint arXiv:2005.01831, 2020.

---

Published as a conference paper at ICLR 2021
Fred Hohman, Hackyu Park, Caleb Robinson, and Duen Horng Polo Chau. Summit: Scaling deep learning interpretability by visualizing activation and attribution summarizations, IEEE transac- fions visualization and computer graphics, 26(1):1096 106,, 2019.
Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. benchmark for interpretabil- ity methods in deep neural networks. In Advances in Neural Information Processing Systems, pp. 9737-9748, 2019.
Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. In Advances in Neural Information Processing Systems, pp. 125-136, 2019.
JASP Team. JASP (Version 0.13.1), 2020. URL
Jeya Vikranth Jeyakumar, Joseph Noor, Yu-Hsi Cheng, Luis Garcia, and Mani Srivastava. How can i explain this you? empirical study of deep neural network explanation methods. Advances in Neural Information Processing Systems, 33, 2020.
Andrej Karpathy, Justin Johnson, and Li Fei-Fei. Visualizing and understanding recurrent networks. arXiv preprint 1506. 2015.
Been Kim, Rajiv Khanna, and Oluwasanmi Koyejo. Examples are not enough, learn to criticize! criticism for interpretability. In Advances in neural information processing systems, pp. 2280- 2288, 2016.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al.
Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning, pp. 2668-2677. PMLR, 2018.
Nikolaus Kriegeskorte. Deep neural networks: a new framework for modeling biological vision and brain information processing. Annual review of vision science, 1:417-446, 2015.
Jean-Philippe KrölI, Simon B Fickhoff, Felix Hoffstaedter, and Kaustubh Patil. Evolving complex yet interpretable representations: application to alzheimer's diagnosis and prognosis. In 2020 IEEE Congress on Evolutionary Computation (CEC), pp. 1-8. IEEE, 2020.
Nesaretnam Barr Kumarakulasinghe, Tobias Blomberg, Jintai Liu, Alexandra Saraiva Leao, and Panagiotis Papapetrou. Evaluating local interpretable model-agnostic explanations on clinical ma- chine learning classification models. In 2020 IEEE 33rd International Symposium on Computer- Based Medical Systems (CBMS), pp. 7-12. IEEE, 2020.
Sebastian Lapuschkin, Stephan Wäldchen, Alexander Binder, Grégoire Montavon, Wojeiech Samek, and Klaus-Rober Müilerr. Unmasking clever hans predictors and assessing what machines really learn. Nature communications, 10(1):1-8, 2019..
Matthew L. Leavitt and Ari Morcos. Towards falsifiable interpretability research. arXiv preprint arXiv:2010. 12016, 2020.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):436 444, 2015.
Yi-Shan Lin, Wen-Chuan Lee, and Z. Berkay Celik. What do you see? evaluation of ex- plainable artificial intelligence (xai) interpretability through neural backdoors. arXiv preprint arXiv:2009. 10339, 2020.
Zachary C Lipton. The mythos of model interpretabilityy Queue, 16(3):31-57,200 2018.
Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by inverting them. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 9188-5196, 2015.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed represen- tations words and phrases and their compositionality. In Advances neural information pro- cessing systems, pp- 311 13111-1199 19, 2013.
12

---

Published as a conference paper at ICLR 2021
Tim Miller. Explanation in artificial intelligence: Insights from the social sciences. Artificial Intel- ligence, 267:1-38, 2019.
Grégoire Montavon, Wojciech Samek, and Klaus-Robert Müller. Methods for interpreting and un- derstanding deep neural networks. Digital Signal Processing, 73:1-15, 2018.
Ari Morcos, David GT Barrett, Neil C Rabinowitz, and Matthew Botvinick. On the importance of single directions for generalization. arXiv preprint arXiv: orXiv:/83300695) 2018.
2018.
Alexander Mordvintsev, Christopher Olah, and Mike Tyka. Inceptionism: Going deeper into neural
networks, 2015.
W James Murdoch, Chandan Singh. Karl Kumbier, Reza Abbasi-Asl, and Bin Yu. Interpretable machine learning: definitions, methods, and applications. arXiv preprint arXiv:1901 04592, 2019.
Karthikeyan Natesan Ramamurthy, Bhanukiran Vinzamuri, Yunfeng Zhang, and Amit Dhurandhar. Model agnostic multilevel explanations. Advances in Neural Information Processing Systems, 33, 2020.
An-phi Nguyen and María Rodríguez Martínez. On quantitative aspects of model interpretability. arXiv preprini arXiv:2007.0755: 0755444 2020.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confi- dence predictions for unrecognizable images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 427-436, 2015.
Anh Nguyen, Alexey Dosovitskiy, Jason Yosinski, Thomas Brox, and Jeff Clune. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. In Advances in neural information processing systems, pp. 3387-3395, 2016a.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Multifaceted feature visualization: Uncovering the different types of features learned by each neuron in deep neural networks. arXiv preprint 1602.0600..0300 036/6, 2016b.
Anh Nguyen, Jeff Clune, Yoshua Bengio, Alexey Dosovitskiy, and Jason Yosinski. Plug & play generative networks: Conditional iterative generation of images in latent space. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 4467-4477, 2017.
Fabian Offert. i know it when i see it".. visualization and intuitive interpretability. arXiv preprint arXiv:1711.08042, 2017.
Fabian Offert and Peter Bell. Perceptual bias and technical metapictures: critical machine vision as humanities challenge. Al & SOCIETY, pp. 1-12, 2020.
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2(11)e77, 2017.
Chris Olah, Arvind Satyanarayan, Ian Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and Alexander Mordvintsev. The building blocks of interpretability. Distill, 100, 2018.
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. An overview of early vision in inceptionvl. Distill, 2020a. doi: 14223911/iisttil0000244 https//ddsiilllpp///22eecccciaii
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter.
Zoom in: An introduction to circuits. Distill, 2020b.
doi: (0.23915/dissiil.00024..
https//siiillub22222//vrrriiiiin
OpenAl. OpenAl Microscope. cessed 09/12/202)
2020. (Ac-
Jonathan Peirce, Jeremy R Gray, Sol Simpson, Michael MacAskill, Richard Höchenberger, Hiroyuki Sogo, Erik Kastman, and Jonas Kristoffer Lindeløv. Psychopy2: Experiments in behavior made easy. Behavior research methods, 5151):195-203, 2019.
13

---

Published as a conference paper at ICLR 2021
Alec Radford, Rafal Jozefowicz, and llya Sutskever. Learning generate reviews and discovering sentiment. arXiv preprint arXiv: 2017.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should i trust you?" explaining the predictions of any classifier. Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135-1144, 2016.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Anchors: High-precision model-agnostic explanations. In Thirty-Secone AAAI Conference on Artificial Intelligence, 2018.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeey Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (ICV), 1113)2211-222, 2015. doi: 10. 10.10007/1112----5--0008
Wojciech Samek, Grégoire Montavon, Sebastian Lapuschkin, Christopher Anders, and Klaus- Robert Müller. Toward interpretable machine learning: Transparent deep neural networks and beyond. arXiv preprint wrXiv:2003.076::1 2020.
Philipp Schmidt and Felix Biessmann. Quantifying interpretability and trust machine learning systems. arXiv preprint arXiv:/901.068 008558, 2019.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruy Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE international conference on computer vision, pp. 618 618-626, 2017.
Hua Shen and Ting-Hao' Kenneth' Huang. How useful the machine- generated interpretations to general users? human evaluation on guessing the incorrectly predicted labels. arXiv preprint arXiv:2008. 1 1,21,,122 2020.
Rui Shi, Tianxing Li, and Yasushi Yamaguchi. Group visualization of clas-dissrriminative features. Neural Networks, 2020.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. arXiv preprint arXiv:1706.03822 2017.
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint 1rXiv:1412.6806 2014.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013.
Christian Szegedy, Wei Liu, Yangging Jia, Pierre Sermanet, Scott Reed. Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1-9, 2015.
Erico Tjoa and Cuntai Guan. Quantifying explainability of saliency methods in deep neural net- works. arXiv preprint arXiv:2009.02899 2020.
Julian Tritscher, Markus Ring, Daniel Schlr, Lena Hettinger, and Andreas Hotho. Evaluation of post- hoc xai approaches through synthetic tabular data. In International Symposium on Methodologies for Intelligent Systems, pp. 422-42 430. Springer, 2020.
Zijie Wang, Robert Turko, Omar Shaikh, Haekyu Park, Nilaksh Das, Fred Hohman, Minsuk Kahng, and Duen Homg Chau. Cnn explainer: Learning convolutional neural networks with interactive visualization. arXiv preprint arXiw:2004.15044 2020.
Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson. Understanding neural networks through deep visualization, arXiv preprint Ttti::/506065779 2015.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference computer vision, pp. 818-833. Springer, 2014.
14

---

Published as a conference paper at ICLR 2021
Quan-shi Zhang and of Information Technology & Electronic Engineering, 19(1):27-39, 2018.
Zhu. Visual interpretability for deep learning: survey. Frontiers
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Object detectors emerge in deep scene cnns. arXiv preprint arXiv: 1412.6856, 2014.
Luisa M Zintgraf, Taco S Cohen, Tameem Adel, and Max Welling. Visualizing deep neural network decisions: Prediction difference analysis. arXiv preprint 77X1v:/702.04595, 2017.
15

---

Published as a conference paper at ICLR 2021
A APPENDIX
A.1 DETAILS ON METHOSS
A.1.1 HUMAN EXPERIMENTS
In our two human psychophysical studies, we ask humans to predict a feature map's strongly acti- vating image ("forward simulation task", tusk",Doshii -Doshi- Velez Kim Answers the two-alternative forced choice paradigm are recorded together with the participants' confidence level (1: not con- fident, 2: somewhat confident, 3: very confident, see Fig. 77. Time per trial is unlimited and we record reaction time. After each trial, feedback is given (see Fig.7). A progress bar at the bottom of the screen indicates how many trials of a block are already completed. As reference images, either synthetic, natural izations from the method of Olah et al. (017)). Trials of different reference images are arranged in blocks. Synthetic and natural reference images are alternated, and, in the case of Experiment framed by trials without reference images (see Fig. BA, B). The order of the reference image types is boanter-balanced across subjects.
reference images
given. The synthetic images the feature visual-
The main trials in the experiments are complemented by practice, catch and intuitiveness trials. To avoid learning effects, we use different feature maps for each trial type per participant. Specifically, practice trials give participants the opportunity to familiarize themselves with the task. In order to monitor the attention of participants, catch trials appear randomly throughout blocks of main trials. Here, the query images are copy of one of the reference images, i.e., there is an obvious correct answer (see Fig. 15). This control mechanism allows us to decide whether trial blocks should be excluded from the analysis due to e.g. fatigue. To obtain the participant's subjective impression of the helpfulness of maximally activating images, the experiments are preceded (and also succeeded in the case of Experiment II) by three intuitiveness trials (see Fig, 14). Here, participants judge in slightly different task design how intuitive they consider the synthetic stimuli for the natural stimuli. For more details on the intuitiveness trials, see below.
At the end of the experiment, all expert participants in Experiment I and all lay (but not expert) participants in Experiment II are asked about their strategy and whether it changed over time. The information gained through the first group allows us to understand the variety of cues used and paves the way to identify interesting directions for follow-up experiments. The information gained through the second group allowed comparisons to experts' impressions reported in Experiment I.
The first experiment focuses on comparing performance of synthetic images to two baselines: natural reference images and no reference images (see Fig. (A). Screenshots of trials are shown in Fig. [2] In total, 45 feature maps are tested: 36 of these are uniformly sampled from the feature maps of each of the four branches for each of the nine Inception modules. The other nine feature maps are uniformly hand-picked for interpretability from the Inception modules' pooling branch based on the appendix overview selection provided by Olah al.) (2017) based on our own choices. In the spirit of a general statement about the explainability method, different participants see different natural reference and query images, and each participant sees different natural query images for the same feature maps in different reference conditions. To check the consistency of participants' responses, we repeat six randomly chosen main trials for each of the three tested reference image types at the end of the experiment.
Experiment I
Experiment II The second experiment (see Fig. 8B) is about testing expert vs. lay participants as well as comparing different presentation schemes" (Max 1, Min+Max 1, Max 9 and Min+Max 9, see Fig. KE). Screenshots of trials are shown in Fig. 13, In total, 80 feature maps are tested: They are uniformly sampled from every second layer with an Inception module of the network (hence a total of instead of 9 layers), and from all four branches of the Inception modules. Given the focus on four different presentation schemes in this experiment, we repeat the sampling method four times without overlap. In terms of reference image types, only synthetic and natural images are tested. Like in Experiment I, different participants see different natural reference and query images.
'In pilot experiments, we learned that participants preferred 9 over 4 reference images, hence this "default" choice in Experiment I.
16

---

Published as a conference paper at ICLR 2021
(a) Screen at the beginning of trial. The question is which of the natural images the center screen also strongly activates the CNN feature map given the reference images on sides.
Mirimally actvating
Masimaly actlunting
(b) Screen including participant's 'ss answer visualized by black boxes around the image and the confidence level. A participant indicates which natural image the center would also a strongly activating image by clicking the number corresponding his/her confidence level (I: not confident, 2: somewhat confident, 3: confident). The time until a participant selects an answer is recorded ("reaction time").
Maro confident.
(c) Screen including participant's answer (black boxes) and feedback which image is indeed also strongly activating image (green box).
Figure 7: Forward Simulation Task. The progress bar at the bottom of the screen indicates the progress within block of trials.
17

---

Published as a conference paper at ICLR 2021
A
Experiment (Expert Participants)
Intuitiveness Thials
Synthetic
Repeated Trials
Natural
Practice Trials
None
Main Trials
B
Experiment II (Expert & Lay Participants)
Block Type
Trials in (i))
15 (10)
12 5 (3)
E
Max 1
Min-Max
Max
Min-Max
Figure 8: Detailed structure of the two experiments with different foci. A: Experiment I. Here, the focus is comparing performance of synthetic and natural reference images to the most simple baseline: no reference images ("None"). To counter- counter-balance conditions, the order of natural and synthetic blocks is alternated across participants. For each of the three reference image types (syn- thetic, natural and none), 45 relevant trials are used plus additional catch, practice and repeated trials. B: Experiment II. Here, the focus is testing expert and lay participants well as comparing dif- ferent presentation schemes (Max 1, Min+Max 1, Max 9 and Min+Max 9, see E for illustrations), Both the order of natural and synthetic blocks well as the four presentation conditions are counter- balanced across participants. To maintain reasonable experiment length for each participant, only 20 relevant trials are used per reference image type and presentation scheme, plus additional catch and practice trials. C: Legend. D: Number of trials per block type (i.c. reference image type and main vs. practice trial) and experiment. Catch trials are not shown in the figure; there was a total of 3 (2) catch trials per each synthetic and natural main block in Experiment (II). E: Illustration of presentation schemes. In Experiment II, all four schemes are tested, in Experiment only Min+Max 9 is tested.
However, expert and lay participants see the same images. For details on the counter-balancing of all conditions, please refer Tab.
Intuitiveness Trials In order to obtain the participants' subjective impression of the helpfulness of maximally activating images, we add trials at the beginning of the experiments, and also at the end of Experiment II. The task set-up slightly different (see Fig. [4]: Only maximally activating (i.e. no minimally activating) images are shown. We ask participants to rate how intuitive they find the explanation of the entirety of the synthetic images for the entirety of the natural images. Again, all images presented in one trial are specific to one feature map. By moving slider to the right (left), participants judge the explanation method as intuitive (not intuitive). The ratings are recorded on a continuous scale from 100 (not intuitive) to +100 (intuitive). All participants trials in a randomized order. The trials are again taken from the hand picked (i.e. interpretable) feature maps of the appendix overview in Olah et al. (017)) In theory, this again allows for the highest intuitiveness ratings possible. The specific feature maps are from low, intermediate and high layer: feature map 43 of mixed3a, feature map 504 of mixed4b and feature map 17 of mixed 5b.
the same three
Participants Our two experiments are within-subjec studies, meaning that every participant an- swers trials for all conditions. This design choice allows us test fewer participants. In Experi- ment I, 10 expert participants take part (7 male, 3 female, age: 27.2 years, SD = 1.75). In Experi- ment II, 23 participants take part (of which 10 are experts; 14 male, female, age: 28.1 years, SD = 6.76). Expert participants qualify by being familiar having worked with convolutional neural networks and most of them even with feature visualization techniques. All participants are naive with respect to the aim of the study. Expert (lay) participants paid 15E (10 E), per hour for participation. Before the experiment, all participants give written informed consent for participat- ing. All participants have normal or corrected to normal vision. All procedures conform to Standard
18

---

Published as a conference paper at ICLR 2021
8 of the American Psychological 405 Association's "Ethical Principles of Psychologists and Code of Conduct" (2016). Before the experiment, the first author explains the task to each participant and ensures complete understanding. For lay participants, the explanation is simplified: Maximally (minimally) activating images are called "favorite images" ("non-favorite images") of a "computer program" and the question is explained as which of the two query images would also be "favorite" image to the computer program.
Apparatus Stimuli are displayed on a VIEWPiXx 3D LCD (VPIXX Technologies; spatial resolu- tion 1920 x 1080 px, temporal resolution 120 Hz). Outside the stimulus image, the monitor is set to mean gray. Participants view the display from em (maintained via chinrest) in a darkened chamber. At this distance, pixels subtend approximately 0.024 degrees on average (41 ps per de- gree of visual angle). Stimulus presentation and data collection is controlled via a desktop computer (Intel Core i5-4460 CPU, AMD Radeon R9 380 GPU) running Ubuntu Linux (16.04 LTS), using PsychoPy (Peirce et al. 2019
version 3.0) under Python 3.6.
A.1.2 STIMULI SELECTION
Model Following the existing work on feature visualization by Olah et al. 2011 2018 2020b a an Inception V1 n SZeeedy et al. 2015) trained on ImageNet (Deng et al. 2009 Russakovsky et al., 2013). Note that the Inception VI network used in previously mentioned work slightly deviates from the original network architecture: The 3 3 branch of Inception module mixed4a only holds 204 instead of 208 feature maps. To stay as close as possible to the aforemen- tioned work, also use their implementation and trained weights of the network We investigate feature visualizations for all branches (i.c. kernel sizes) of the Inception modules and sample from layers mixed3a to mixed5b before the ReLU non-linearity.
Synthethic Images from Feature Visualization The synthetic images throughout this study are the optimization results of the feature visualization method from Olah et al. (2017). We use the channel objective to find synthetic stimuli that maximally (minimally) activate the spatial mean of a given feature map of the network. We perform the optimization using lucid 0.3.8 and TensorFlow 1.15.0 (Abadi et al. 2015) and use the hyperparameter as specified in Olah et al. (017)) For the experimental conditions with more than one minimally/maximally activating reference image, add a diversity regulariztion across the samples. In hindsight, we realized that we generated 10 synthetic images in Experiment I, even though we only needed and used 9 per feature map.
Selection of Natural Images The natural stimuli are selected from the validation set of the Im- ageNet ILSVRC 2012 (Russakovsky et al., 2015) dataset. To choose the maximally (minimally) activating natural stimuli for given feature map. we perform three steps, which are illustrated in Fig. and explained in the following: First, we calculate the activation of said feature map for all pre-processed images (resizing to 256 x 256 pixels, cropping centrally to 224 x 224 pixels and nor- malizing) and take the spatial average to get a scalar representing the excitability of the given feature map caused by the image. Second, we order the images according to the collected activation values and select the (N + +) -Natchhes maximally (respectively minimally) activating images. Here, Natimuli corresponds to the number of reference images used (either or 9, see Fig. 8 E). the +1 comes from the query image, and Noatches 20 determines the maximum number of participants we can test with our setup. Third, we distribute the selected images into N Nstimult + blocks. Within each block, we randomly shuffle the order of the images. Lastly, we create Nhatches batches of data by selecting one image from each of the blocks for every batch B
"This network is considered very interpretable (Olah et al. 2018), yet other work also finds deeper networks more interpretable (Bau et al. 2017). More recent work, again, suggests that "analogous features [..]] form across models [...]]" Le. that interpretable feature visualizations appear "universally" for different CNNs OOlh
2020b OpenAl (2020).
github.com
After having performed Experiment and II, we realized a minor bug in our code: Instead moving every 20th image into the same batch for one participant, we moved every 10th image into the same batch for one participant. This means that we only use a total of 110 different images, instead of 200. The minimal query image is still always selected from the 20 least activating images; the maximal query image is selected from the 91" to 1 th maximally activating images and we do use the 111" 11 to 200" maximally activating images.
19

---

Published as a conference paper at ICLR 2021
A
C
images
images
Probabiity Density
Prosahityy Donety
0030
basch
20
0000
20
0000
images from each Ion randimmly
40
-0
Activiaton
20
00
35
90
10
00 05
20
Figure 9: Sampling of natural images. A: Distribution of activations. For an example channel (mixed3a, kernel size x 1, feature map 25), the smoothed distribution of activations for all 50, 000 ImageNet validation images is plotted. The natural stimuli for the experiment are taken from the tails of the distribution (shaded background). B: Zoomed-in tail of activations distribution. In the presentation schemes with 9 images, 10 bins with 20 images each are created (10 because of 9 reference plus query image). C: In order to obtain 20 batches with 10 images each, the 20 images from one bin are randomly distributed to the 20 batches. This guarantees that each batch contains fair selection of extremely activating images. The query images are always sampled from the most extreme bins in order to give the best signal possible. In the case of the presentation schemes with reference image, the number of bins in B is reduced to and the number images per batch in also reduced to 2.
Order of presentation schemes (0-3) and batch-blocks (A-D)
Batches
Order of synthetic and natural
Subject
Practice
Main
1
0 (A)
1 (B)
(C)
3 (D)
2
(B)
(D)
1 (C)
(A)
natural:
natural synthetic
3
3 (B)
1 (D)
(A)
(C)
synthetic: 2
4
3 (C)
(B)
1 (A)
(
5
6
natural: 3
see subject 1-4
synthetic natural
7
synthetic: 4
0
8
9
10
natural: 5 synthetic: 6
see subject 1-4
natural synthetic
11
12
natural: 7
13
see subject 1-4
synthetic natural
synthetic: 8
Table 1: Counter-balancing conditions in Experiment I.. In total, 13 naive and 10 lay partic- ipants are tested. Fach "batch block" contains 20 feature maps (sampled from five layers and all Inception module branches). Batches indicate which batch number the natural query (and reference images) are taken from.
The reasons for creating several batches of extremely activating natural images are two-fold: (1) We want get general impression of the interpretability method and would like reduce the depen- dence on single images, and (2) in Experiment I, a participant has to see different query images in the three different reference conditions.
downside of this design choice an increase variability. The precise allocation was done follows: In Experiment I, the natural query images of the none condition were always allocated the batch with batch_nr = subject _id, the query and reference im- ages of the natural condition were allocated the batch with batch_nr ral query images the synthetic condition were allocated the batch with batch = subject.id The allocation scheme in Experiment II can be found in Table T
subject-id +d++1 1, and the natu-
20

---

Published as a conference paper at ICLR 2021
Selection of Feature Maps
The selection of feature maps used in Experiment I is shown in Ta-
ble 2 the selection of feature maps used in Experiment II is shown in Table 3
Layer
Branch
Feature Map
Layer
Branch
Feature Map
X 1
25
x 1
95
3
189
3
342
mixed3a
mixed4d
5 x 5
197
5 x
451
Pool
227
Pool
483
Pool'
230
Pool
516
x 1 3 x
64
231
178
x
524
mixed3b
mixed4e
5 5 Pool
390
5 5
656
430
Pool
816
Pool
462
Pool
809
x
68
1
229
3 x
257
3 x
278
mixed4a
mixed5a
5
427
5x5
636
Pool
486
Pool
743
Pool
501
Pool"
720
45
119
3 x
339
3 x 3
684
mixed4b
mixed5b
5
438
844
Pool
491
Pool
1007
Pool*
465
Pool
946
x
94
3 x 3
247
mixed4c
5
432
Pool
496
Pool
449
Table 2: Feature maps analyzed in Experiment I. For each of the layers with Inception module, one randomly chosen feature map per branch (1 x 1, 3 x 3, x and pool) and one additional hand-picked feature map (highlighted with *) are used.
A.1.3 DIFFERENT ACTIVATION MAGNITUDES
We note that the elicited activations synthetic images are almost always about one magnitude larger than the activations of natural images (see Fig, [Oab. This constitutes inherent difference in the synthetic and natural reference image condition. A simple approach to make the two conditions more comparable is limit the optimization process such that the resulting feature visualizations elicit activations similar to that of natural images. This can be achieved by halting the optimization process the activations approximately match. By following that procedure finds limited synthetic images which are indistinguishable from natural images in terms their activations (see Fig. [Ob). Importantly though, these images are visually not more similar natural images. have much lower color contrast than normal feature visualizations, and above all hardly resemble mean- ingful features (see Fig.
A.1.4 DATA ANALYSIS
Significance Tests All significance tests are performed with JASP (JAPP Team 2020 version 0.13.1). For the analysis of the distribution of confidence ratings (see Fig. BB), we use contingency tables with tesss.. For testing pairwise effects in accuracy, confidence, reaction time and intuitive- ness data, we report Wilcoxon signed-rank tests with uncorrected p-values (Bonferroni-correcteed critical alpha values with family-wise alpha level of 0.05 reported in all figures where relevant). These non- parametric tests are preferred for these data because they do not make distributional as- sumptions like (main effects of one factor marginalizing over another) we report results from repeated measures ANOVA (RM ANOVA), which does assume normality.
errors, in e.g. paired t-tests. For testing marginal effeets
21

---

Published as a conference paper at ICLR 2021
Synthetic
2000
Netural
Activation
1000
-1000-
-000
mires 1x1
maeeta 3x3
mixess 5x5
mixedtaa Pool
moxedaa Pool
mixes3 1xl
moxedt 3x3
mixed 5v5
miredt Pool
mixed Pool
mixed4a 1xl
mixed4a 3x3
mixeddla 5x5
mixed4a Pool
mixed4a Poor-
mixed44 1xl
mixeddo 3v3
mixedda 5x5
mixedAo Pool
mixed4o Poot
mixed4c 1xl
mixed4c 5xd5
mixeddc Pool
mixed4: Poot
mixedkd 1xl
mixeded 3x3
mixedAd 5x5
mixeddd Pool
mixedde 1xl
mixed4e 3x3
mixedde S
mixed4e Pool
mixedile Pool
mixed 1xl
mixed5a 3x3
mixedSa 5x5
mixedSa Pool
mixed5a Poot-
moxedSo 1xl
mixed50 3x3
movedSo 5x5
mixed50 Pool
mixed Puol
mixedd:
mireded
Layer and Branch
(a) Activations of natural and synthetic images.
400
Limited Synthetic Natural
Activation
200
-200
-400
mixed 3e 1x1
mixed _a 5x5
mixed 30 Pool
mixed 3a Pool
mixed 30 1x1
mived 3x3
mixed 30 5x5
mixed 30 Pool
mixed Pool-.
mixed 4a 1x1
mixed mixed Aa 3x3
mixed da Pool
mixed 4a Pool
mixed 4b ixl
mixed 40 3x3
mixed 40 Pool
mixed 4b Pool..
mixed 4c 3x3
mixed 4e 5x5
mixed 4e Pool
mixed 4d 1x1
mixed 4d 3x3
mixed 4d 5x5
mixed Pool
mixed Ad Pool*
mixed 4e 1xl
mixed 4e 595
mixed de Pool
mixed Pool
mixed 55 lxl
mixed 50 3x3
mixed 55 SxS
mixed 5a Pool
mixed Pool
mixed 50 1x1
mixed 5b 3x3
mixed 55 Pool mixed 5b Pool*
mixed
mixed
Layer and Branch
(b) Activations of natural and limited synthetic images.
Figure 10: Mean activations and standard deviations (not two standard errors of the mean!) of the minimally (below 0) and maximally (above 0) activating synthetic and natural images used in Experiment Note that there are 10 (i.e. accidentally not 9) synthetic images and 20 10 = 200 natural images (because of 20 batches) in Experiment I for both minimally and maximally activating images. Please also note that the standard deviations for the selected natural images are invisible because they are so small. Limited synthetic images refer to feature visualizations which are the result of stopping the optimization process early with the goal of matching the activation level of natural stimuli.
A
B
Figure 11: Limited feature visualizations, which are the result of stopping the optimization process
early with the goal of matching the activation level of the chosen extreme natural stimuli. A: Feature visualizations for mixed_4 -4a pool* feature map Experiment I. B: Feature visualizations for nine pool* feature maps of Experiment I.
22

---

Published as a conference paper at ICLR 2021
Experiment I: Example trials of the three reference images conditions: synthetic refer- ence images (first row), natural reference images (second row) or no reference images (third row). The query images the center are always natural images.
Figure 12:
23

---

Published as a conference paper at ICLR 2021
Figure 13:
Experiment Example trials the four presentation schemes: Max 1, Min+max 1, Max 9, Min+Max 9. The left column contains synthetic reference images, the right column contains natural reference images.
24

---

Published as a conference paper at ICLR 2021
Ostimuaet Images
Images
explavation thece optimieed images
Images
Images
100
OBOR POTT
explamation
aad images
Hot tnu Ere
Imngges
Images
excionation images natural
Hot Intu Ene
Figure 14: Trials for intuitiveness judgment. The tested feature maps from layer mixed3a (chan- nel 43), mixed4b (channel 504) and mixed 5b (channel 17). They are the same in Experiment and Experiment II.
25

---

Published as a conference paper at ICLR 2021
Feature Map for
Layer
Branch
Batch Block (A-D)
B
1
25
14
12
53
3 x
189
97
171
106
mixed3a
5
197
203
212
204
Pool
227
238
232
247
I
33
45
3 3 3
257
355
321
200
mixed4a
x
427
425
429
423
Pool
486
497
478
506
1 X
94
53
59
95
3
247
237
357
209
mixed4c
5 5 5 Pool
432
402
400
416
496
498
473
497
231
83
6
89
3 x
524
323
401
373
mixed4e
5
656
624
642
620
Pool
816
755
724
783
1x
119
14
266
300
3 x 3
684
592
657
481
mixed5b
5
844
829
839
875
Pool
1007
913
927
903
Table 3: Feature maps analyzed in Experiment II. Four sets of feature maps (batch blocks A to D) are sampled: For every second layer with an Inception module layers total), one feature map is randomly selected per branch of the Inception module (1 1, 3 x 3, 5 x 5 and pool). For the practice, catch and intuitiveness trials additional randomly chosen feature maps are used.
Catch trials. An image from the reference images is copied as query image, which makes the answer obvious. The purpose of these trials is integrate mechanism into the experi- ment which allows us to check post-hoc whether participant was still paying attention.
Figure 15:
A.2 DETAILS ON RESULTS
A.2.1
COMPLEMENTING FIGURES FOR MAIN RESULTS
Figures [6 21 complement the results and figures presented in Section Here, all experimental conditions are shown.
A.2.2 DETAILS ON PERFORMANCE OF EXPERT AND LAY PARTICIPANTS
As reported in the main body of the paper, mixed-effects ANOVA revealed no significant main effect of expert level (F(1, 21) = 0.6, p = 0.44, netween-subject effect). Further, there is no significant interaction with the reference image type (F(1, (1, 0.4, p 0.53). and both expert and lay participants show significant main effect of the reference image type (F(1,21) 21) = 230.2, p.p < 0.001).
26

---

Published as a conference paper at ICLR 2021
0.00
0.000
Propartion Correct
0.000
06-
0.4
Systhetic
Reference Images
(a) Performance.
Proprtion of Correct Traals
Propartion of Incorest Triats
0.6
0.0
0.6
0.4
0.4
0.4
0:3-
0:3
03
02-
0.2
0.1
0.1
Syrthatic
Natural
home
Syrthatic
Naatural
home
Synthetic
Natural
hane
Confidence Ratingg
Ratlng
Confidence Ratling
(b) Confidence ratings on correctly (c) Confidence ratings on incor- (d) Confidence ratings on all trials. answered trials.
rectly answered trials.
17.5
0.001
0.001
Reaction Time [sec]
0.003
Reaction Time [sec]
20
0.003
17.5
12.3
Reaction Time
7.5
10
7.5
5.0
255
0.0
0.0
Synthetic Natural None Reference Images
Synthetic Natural None
Synchetic Natural None
Reference Images
Reference Images
(e) Reaction time on correctly (l) Reaction time on incor- (g) Reaction time on all trials. answered trials.
rectly answered trials.
Figure 16: Task performance (a), distribution of confidence ratings (b-d) and reaction times (e-g) of Experiment The p-values are calculated with Wilcoxon sign-rank tests. Note that unlike in the main paper, these figures consistently include the "None" condition. For explanations, see Sec. 4.1
27

---

Published as a conference paper at ICLR 2021
Proportion Correct
0.8
0.6
0.4
Synthetic Natural
Reference Images
(a) Performance.
Proportion Correct Trials
Proportion Incorrect Trials
0.5
Proportion Total Trials
0.4
0.4
0.3
0.2
0.2
0.1
0.1
0.1
0.0
0.0
0.0
Synthetic
Natural
Syntheric
Natural
Synthatic
Natural
Confidence Rating
Confidence Rating
Confidence Rating
(b) Confidence ratings rectly answered trials.
(c) Confidence ratings incor- (d) Confidence ratings all tri-
rectly answered trials.
als.
< 0.001
0.611
12
<
0.001
12
10
Reaction Time [sec]
Reaction Time [sec]
Reaction Time [sec]
10
2
2
Synthetic
Synthetic
Synthetic
Reference Images
Reference Images
Reference Images
(e) Reaction time on cor- Reaction time on incor- (g) rectly answered trials.
Reaction time on rectly answered trials. trials.
Figure 17: Task performance (a), distribution of confidence ratings (b-d) and reaction times (e-g) of Experiment II, averaged over expert level and presentation schemes. The p-values are calculated with Wilcoxon sign-rank tests. The results replicate our findings of Experiment I. For explanations on the latter, see Sec.
28

---

Published as a conference paper at ICLR 2021
Proportion Correct
0.4
Synthetic
Natural Layer
None
(a) Performance across layers.
1.0
Proportion Correct
0.8
0.6
0.4
0.0
1x1
3x3
5X5
pool
1x1
3x3
5x5 pool
1x1 3x3 5x5 pool
Synthetic
Natural
None
Branch in Inception Module
(b) Performance branches.
Figure 18: High performance across (a) layers and (b) branches of the Inception modules Exper- iment Note that unlike in the main paper these figures consistently include the "None" condition. For explanations, see Sec.
29

---

Published as a conference paper at ICLR 2021
Proportion Correct
0.8
0.6
0.4
3a
4a
4c
4e
5b
30
48
4c
4e
56
Synthetic
Natural
Layer
(a) Performance across layers.
1.0
Proportion Correct
0.8
0.6
0.4
0.2
0.0
1x1
3x3
5x5
pool
1x1 3x3
5x5
pool
Synthetic
Natural
Branch in Inception Module
(b) Performance branches in Inception module.
Figure 19: High performance across (a) layers and (b) branches the Inception modules in Ex- periment II. Note that only every second layer is tested here (unlike in Experiment I). The results replicate our findings of Experiment I. For explanations, see Sec, 4.2
30

---

Published as a conference paper at ICLR 2021
A.2.3 DETAILS ON PERFORMANCE OF EXPERTS SPLIT BY DIFFERENT LEVELS OF EXPERTISE
Even though Experiment II does not show significant performance difference for lay and expert participants, it is an open question whether the level of expertise or the background of experts mat- ters. For the data from experts, we hence further divide participants into subgroups according their expertise (see Fig. - and background level (see Fig. 20 Expertise level 1 means that partici- pants are familiar with CNNs, but not feature visualizations; expertise level means that participants have heard of or read about feature visualizations; and expertise level 3 means that participants have used feature visualizations themselves. We note that we also accepted feature visualizations meth- ods other than the one by Olah et al. (2017), e.g. DeepDream (Mordvintsev et al. 2015) for level 2 and 3. Regarding background, we distinguished computational neuroscientists from researchers working computer vision and or machine learning. We note that some subgroups only hold one participant and hence may not be representative.
Our data shows varying trends for the three expert levels (see Fig. 20p-f): For synthetic images, performance decreases with increasing expertise in Experiment I, but increases for Experiment II. For natural images, performance first increases for participants of expertise level 2, and then slightly decreases for participants with expertise level 3 a trend that holds for both Experiment I and II. In the none condition of Experiment I, performance is highest for the participant of expertise level 1, but decreases for participants of expertise level 2, and again slightly increases for expertise level 3.
Regarding expert's different backgrounds, our hypothesis that many of the computational neuro- scientists are very familiar with maximally exciting images for monkeys or rodents, and hence might perform better than pure computer vision / machine learning experts. Fig. 20g-h suggest that this not the case: The bars for all three reference image types are very similar.
Not finding clear trends in our data between different expertise levels or experts is not surprising as there is even no significant difference between participants whose professional backgrounds are much further apart: lay people vs. people familiar with CNNs.
A.2.4 DETAILS ON PERFORMANCE OF HAND- AND RANDOMLY-PICKE FEATURE MAPS
As described in the main body of the paper, pairwise Wilcoxon sign-rank tests reveal no significant differences between hand-picked and randomly-selected feature maps within each reference image type (9) = 27.5, p = 0.59 for natural reference images and (9)) = 41 p = 0.18 for synthetic references). However, marginalizing over reference image type using repeated measures ANOVA reveals a significant main effect of the feature map selection mode: ()1 = 6.14, = 0.035. Therefore, while there may be small effect of hand-picking feature maps, our data indicates that this effect, if present, is small.
A.2.5 REPEATED TRIALS
To check the consistency of participants' responses, we repeat six main trials for each of the three tested reference image types at the end of the experiment. Specifically, the six trials correspond the three highest and three lowest absolute confidence ratings. Results are shown in Fig. 21 We observe consistency be high for both the synthetic and natural reference image types, and moderate for no reference images (see Fig. 2TA). In absolute terms, the largest increase in performance occurs for the none condition; for natural reference images there was also a small increase; for synthetic reference images, there was a slight decrease (see Fig. 21 and C). In the question session after the experiments, many participants reported remembering the repeated trials from the first time.
A.2.6 QUALITATIVE FINDINGS
In qualitative interview conducted after completion of the experiment, participants reported to use a large variety of strategies. Colors, edges, repeated patterns, orientations, small local structures and (small) objects were commonly mentioned. Most but not all participants reported to have adapted their decision strategy throughout the experiment. Especially lay participants from Experiment II emphasized that the trial-by-trial feedback was helpful and that it helped to learn new strategies. As already described in the main text, participants reported that the task difficulty varied greatly; while some trials were simple, others were challenging. A few participants highlighted that the comparison between minimally and maximally activating images was crucial clue and allowed employing the
31

---

Published as a conference paper at ICLR 2021
0.8
0.8
Proptortion Correct
Proportion Correct
0.6
0.6
Proportion
0.2
0.89
0.20e5.0
081:0.35
0.:00.00
0
6.2
0:1100
0.91
0.4400.8
Synthetic
Synthetic
Synthetic Natural None
Reference Images
Reference Images
Reference Images
(a) Expertise level I: one par- (b) Expertise level 2: six par- (c) Expertise level 3: three ticipant.
ticipants.
participants.
0.8
0.8
Proportion Correct
Proportion Correct
Proportion Correct
0.6
0.6
0.4
0.4
0.4
0.2
0.62.0.0
0.71=0.0
0.2
0.71.0.0c
0.86:n.03
0.2
0.72:0.01
Synthetic
Natural
Synthetic
Natural
Synthetic Natural
Reference Images
Reference Images
Reference Images
(d) Expertise level I: one (e) Expertise level 2: six (f) Expertise participant.
level 3:
participants.
three participants.
0.8
0.0
Propertion Correct
Proportion Carrect
0.4
0.4
0.2
0.
0.920..0
0.Ex6.11
0.2
08.0.04
Synthetic Netural
Reference Images
Reference Images
(g) Computational Neuro- (h) Computer Vision
science background: six
Machine Learning back- ground: four participants.
participants.
Figure 20: Performance of experts split by different levels of expertise: The first (second) row shows the data of Experiment I (II) split up by different levels of familiarity with CNNs and feature visualizations. The third row shows the data Experiment split up by different backgrounds.
32

---

Published as a conference paper at ICLR 2021
1
Proportion Consistent
Proportion Correct
0.8
Proportion Correct
0.8
0.6
0.6
0.4
0.2
Synthetic Natural None
Synthetic Natural
None
Synthetic Natural None
Reference Images
Reference Images
Reference Images
(a) Proportion of trials that were an- Performance for repeated trials (c) Performance for repeated trials when first shown.
swered the same upon repetition.
upon repetition.
Figure 21: Repeated trials in Experiment I.
exclusion criterion: If the minimally activating query image was easily identifiable, the choice of the maximally activating query image was trivial. This aspect motivated us conduct an additional experiment where the presentation scheme was varied (Experiment II).
A.2.7 BY-FEATURE-MAT ANALYSIS
For Experiment
look at each feature map separately and analyze which feature maps partici- pants find easy and which they find difficult. Further, we investigate commonalities and differences between feature maps. We note that the data for this analysis relies on only 10 responses for each feature map and hence may be noisy.
In Fig. [2] we show the number of correct answers split up by reference image type. The patterns look similar to the trend in Fig. 4 Across most layers, there is no clearly identifiable trend that feature maps of certain network depth would be easier or more difficult; only the lowest (3a) and the highest layer (5b) seem slightly more difficult for both the synthetic and the natural reference images.
Easy Feature Maps When feature maps are easy (synthetic: 10/10, natural: 10/10 correct re- sponses), their features seem correspond clear object parts (e.g. dogs vs. humans, food cats), or shapes (e.g. round vs. edgy (see Supplementary Material Fig. 2 5). In Fig. 23 we show the query as well as natural and synthetic reference images for one such easy feature map for one participant. For the images shown to two more participants, see Supplementary Material Fig. [ Other relatively easy feature maps (where eight ten participants choose the correct query image for both reference image types) additionally contained other low level cues such as color or texture (see Supplementary Material Fig. 45).
Difficult Feature Maps are displayed in Fig. 24 Only four participants predicted the correct query image. Interestingly, the other reference image type was much more easily predictable for both feature maps: Nine out ten participants correctly simulated the network's 's decision. Our impression is that the reason for these feature maps being so difficult in one reference condition the diversity in the images. In the case of synthetic reference images, also consider identifying concept difficult and consequently unsure what to compare.
The most difficult feature maps for synthetic and natural reference images
From studying several feature maps, impression is that more of the following aspects make feature maps difficult to interpret:
Reference images are diverse (see Fig. 24 for synthetic reference images and for natural reference images)
33

---

Published as a conference paper at ICLR 2021
answers (out 10)
10.30 10
10
10
10 30 10
10
optimized
correct answers (our of 10) # correct answers (out of 10) # correct answers
Z
natiral
nore
2
2
545
30.
ab.
545
40.
AC. 1xt
5W5
dr 5X5
K. 11
5x5
111
SXS
46.
111
53.P
343
50.3
96.
36
M
Se
5%
Feature Map (Specified by Layer and Branch)
Figure 22: Data for Experiment split up by feature maps: For each reference image type, the number of correct answers (out of ten) is shown. There is no clear trend that certain feature maps would be easier or more difficult.
34

---

Published as a conference paper at ICLR 2021
Synthetic reference images
Reference Images
Reference Images
Query Images
Min
Max
(b) Natural reference
Reference Images
Reference Images
images
Query Images
An easy feature map (here: 5a, pool*) from Experiment I where all participants an- swered correctly for both synthetic and natural reference images. The shown stimuli were shown to participant 1, for stimuli shown to participant and see Supplementary Material Fig
Figure 23:
The common feature(s) seem to not correspond common human concepts (see Fig. 24 and c)
Conflicting information, i.e. commonalities can be found between one query image and both the minimal and maximal reference images (see Fig. 25p: eyes and extremity-like structure in synthetic min reference images eyes and earth-colors in synthetic max reference images both could be considered similar to the max query image of a frog)
Very small object parts such as eyes round, earth-colored shapes seem to be the decisive features (see Fig. 25p and b)
Low level cues such as the orientation of lines appear random in the synthetic reference images (see Fig. 2
Finally, when we speak bluntly, we are often surprised that participants identified the correct image the reasons for this are unclear to us (see for example Supplementary Material Fig. 6
A.2.8 HIGH QUALITY DATA AS SHOWN BY IIGH PERFORMANCE ON CATCH TRIALS
We integrate a mechanism to probe the quality of our data: In catch trials, the correct answer is trivial and hence incorrect answers might suggest the exclusion of specific trial blocks (for details, see Sec. A.I.I). Fortunately, very few trials are missed: participants miss one trial each (i.e. total of out of 180 catch trials were missed); in Experiment II, five participants miss one trial and four participants miss two trials (i.e. a total of 13 out of 736 catch
Experiment I, only two (out of ten)
We expected lower layers to be easier than higher layers for synthetic reference images, but our data showed that this was not the case (see Fig, 22 We can imagine that the diversity term as well as the non-custom hyper-parameters contribute to these suh-optimal images.
35

---

Published as a conference paper at ICLR 2021
(a) Synthetic reference images
Min Reference Images
Reference Images
Query Images
Min
Max
(b) Natural reference
Min Reference images
Max Reference Images
images
Whanom
Query Images
(c) Synthetic reference
Min Reference Images
Max Reference Images
images
Query Images
Max
(d) Natural reference
Min Reference Images
Max Reference Images
images
CHANGE
Query Images
Max
Figure 24: Two difficult feature maps (4d, 5x5 in a and b; 5b, 5x5 in and d) from Experiment I where only four participants answered correctly for synthetic (a and b) and natural (c and d) reference images. The displayed stimuli were shown to participant 1, for stimuli shown participant 2 (3), see Supplementary Material Fig. g
36

---

Published as a conference paper at ICLR 2021
(a) Synthetic reference images
Min Reference Images
Max Reference Images
Query Images
Max
(b) Natural reference im- ages.
Min Reference Images
Max Reference Images
Query Images
Min
Max
Figure 25; A feature map (here: 4a, Pool) from Experiment I where the feature is small (eyes) and a participant might perceive conflicting information (cyes and extremity-like structure in min reference images eyes and earth-colors in max reference images). In this specific example, eight (nine) out ten participants gave the correct answer for this feature map given synthetic (natural) reference images. The displayed stimuli were shown to participant for stimuli shown to participant 2 and 3, see Supplementary Material Fig. 10
37

---

Published as a conference paper at ICLR 2021
(a) Synthetic reference
Min Reference Images
Max Reference Images
images
Query Images
Max
(b) Natural reference
Min Reference Images
Max Reference Images
images
Query Images
Min
Max
Figure 26: A feature map from low layer (here: 3a. 3x3) from Experiment I where the feature seems to be a low level cue (horizontal vs. vertical striped) that is surprisingly clear the natural, but surprisingly unclear in the synthetie reference images. In this specific example, seven (eight) out of ten subjects gave the correct answer for this feature map given synthetic (natural) reference images. The displayed stimuli were shown to participant 1, for stimuli shown participant 2 and 3, see Supplementary Material Fig. [1
38

---

Published as a conference paper at ICLR 2021
trials were missed). As this indicates that data is of high quality, do not perform the analysis with excluded trials as we expect to find the same results.
"Baseline condition.
"Metrics of explanation quality computed without human judgment are inconclusive and do not correspond human rankings.
ToTak has an additional "I don't know" noow"-option for confidence rating.
Coomarisson is only performed between methods but absolute measure of interpretability for method obtained.
39

---

Published as a conference paper at ICLR 2021
A.3
DETAILS ON RELATED WORK
Analyzes
Explanation Methods Analyzed
Resalts
Paper
Intermediate Features?
Explanation helpful?
Confidence/Trust
Feuture Visualization
high variance in confidence ratings
Ours
yes
natural images"
yes
natural images more helpful
no explanation"
Biessmann Refiano (2019)
IRP
highest confidence for guided backprop"
no
Guided Backprop
yes
simple gradient"
prediction gradients
faulty explanations do not decrease trust
Chu et al. (2020)
prediction
information"
Extremal Perturb
GradCAM
Huan (2020)
SmoothGrad
no explanation"
LIME
Anchor
Jeyakumar ct al. (2020)
SHAP
no
Saliency Maps
unclear
Grad- Cadd-C-++++
Ex- Matchina
Alqurawwi al. (2020)
LRP
classification scores
yes
confidence similar across conditions
explanation"
Chandra- sekaran et (2017)
prediction confidence
attention maps
Grad-CAM explanation"
Schmidt Biessmann (2019)
LIME
humans trust judgement regardless explanations, except in one condition
custom method
yes
random/no explanation
LIME
high variance in helpfulness
Hase Bansal (2020)
Prototype
Anchor
partly
helpfulness cannot predict user formance
Decision Boundary
combination of all
Kumaraku- lasinghe et (2020)
fairly high trust and reliance
LIME
yes
high confidence for Anchor low LIME explanation
Riheiro al.
LIME
Anchor
yes
(2018)
explanation
Alufaisan ct al. (2020)
prediction + Anchor
explanations not increase confidence
no
prediction" no information
partly
Ramamurthy al. (2020)
MAME SP-LIME Two Step
adjust MAME which increased trust
no
unclear
Dieber Kirrane (2020)
no
LIME
partly
SHAP
Dinu
al. (2020)
ridge
partly
no statement
lasso
confidence ratings
random explanation"
40

---

Published as a conference paper at ICLR 2021
Experimental Setup
Paper
Dataset
Task
Participants
Collected Data
decision confidence
Ours
natural images (ImageNet)
CNN activation
classification
laypeople
experts
reaction time
post-hoc evaluation
Biessmann Refiano (2019)
face images (Cobn-Kanu) (obtn-Kanade)
2-way
decision confidence
laypeople
classification
reaction time
decision trust
Chu al. (2020)
face images (APPA-REAL)
age regression
laypeople
reaction time post-hoc evaluation
Shen & Huan (2020)
natural images
model error identification
laypeople
decision
(ImageNeet
natural images (CIFAR-10)
preference for one two explanation methods
Jeyakumar al. (2020)
(Sentiment14))
audio (Speech
laypeople
decision
Commands)
sensory data (MIT- BIH Arthythmia)
technical background (ncither lay nor expert)
decision
Alqaraawi (2020)
natural images (Pascal VOC)
contidence free answer on features
classification
Chandra- sekaran al.
model error identification regression
VQA
laypeople
decision
(visualqa.org
(2017)
Schmidr Biessmann (2019)
book categories Movic reviews (IMDb)
decision
9-/2-way classification
laypoople
reaction time
trust
movie reviews (Movic Review)
Hase & Bansal (2020)
decision
labular
classification
2-way
experts
helpfulness rating explanation helpfulness
(Adult)
Kumaraku- lasinghe al.
decision
tabular
2-way classification
feature ranking
(Patient data)
experts
satisfaction
(2020)
questionnaire
Ribeiro al.
2-way classification VQA
tabular (Adult, rodv)
decision
experts
reaction time confidence
(2018)
Alufaisun al.
tabular (COMPAS, Census Income)
decision contidence reaction time
2-way classification
laypeople
(2020)
Ramamurthy al. (2020)
tabular (HELOC, pump failure)
2-way classification
experts laypcople
decision
Dieber & Kitrane (2020)
tabular (Rain in Australia)
laypeople experts
how interpretahle
interview
LIME output
Dinu al.
tabular (Airbnh price listings)
which model would perform better in practice
decision:
interview
laypeople
(2020)
confidence
Table Overview of publications that evaluate explanation methods in human experiments. Note that the table already starts on the previous page and that the footnotes are displayed on page 39
41


Reviews:
[{"review_id": "kB1AwBpaKoP", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "--- Summary ---\n\nThis paper focuses on feature visualizations that generates maximally activating images for a given hidden node to understand inner workings of CNNs. They compare the informativeness of these images compared to natural images that also strongly activate the specified hidden node, and find that natural images help human better to answer which other test natural images are also maximally activating.\n\n--- Pros ---\n\n1. The writing is crisp and excellent: the flow in the intro is easy to follow, the related work is thorough, and the figures are designed very well.\n\n2. The experimental design is clear and reasonable. The statistical comparisons are rigorous and the visualizations are clear.\n\n3. I like the subjective score section: the qualitative conversations are intersting to read.\n\n--- Comments ---\n\n1. Are there some examples that natural images can not help the participant predict while synthetic images can, and vice versa? It might be interesting to see in what circumstances which methods help.\n\n2. The conclusion is not surprising given the task is designed to find other test natural images. But this paper does highlight the shortcomings of synthetic feature visualizations that are not easy for human to understand.\n\n--- Overall evaluation ---\n\nThe writing is great and clear. Although I find the conclusion is not very surprising, I still enjoy reading this paper throughout its rigorous experimental setups like experts v.s. laymen, hand-picked v.s. random images, number of images presented, and subjective scores etc. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "Q3SYezTeQkX", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper asks a simple question: do extreme-activating synthetic images for a CNN unit help a human observer to predict that unit’s response to natural images, compared with maximally/minimally activating natural images. The authors present human observers with images synthesized to maximally or minimally activate a CNN unit, and then ask observers to make a binary choice as to which of two subsequently presented natural images will yield a larger unit response. They find that the synthetic images provide useful information for prediction, but that the benefit is smaller than that provided by simply presenting people with other natural images that maximally or minimally activate a unit. \n\n--\n\nPros:\n\nThe paper presents a simple and reasonable framework for evaluating the utility of synthesized images in allowing human observers to make predictions about how a CNN will behave. \n\nThe paper is clear and to-the-point. \n\nThe paper examines the utility of synthetic images at multiple stages/layers of a CNN, finding similar results at all stages. \n\nI liked that the authors considered both experts and lay people.\n\n--\n\nLimitations and questions:\n\nFeature visualizations have many potential goals, only one of which is to allow a human observer to predict network behavior for natural images. For example, feature visualizations may provide insights about how the network’s behavior deviates from that of a human observer, by showing that the network relies on features that differ from human-intuitive features. Feature visualizations might also be used to compare the strategies used by two different networks to perform the same task, which might be hard to do using natural images alone. This multiplicity of uses is something which I think should be highlighted more prominently. -- Update: the authors have made minor text changes to note limitations. --\n\nI wondered what the results would look like for a simpler case: a linear filter? In the case of a linear filter, it’s clear that the filter captures everything necessary to predict the response of the unit. But it's not obvious to me how good human observers would be at using the filter to predict how a linear filter would behave in natural images. Would the results be the same if the authors looked at first the layer of the network? If so, does this suggest that humans are just bad at using synthetic stimuli to make predictions in natural images? -- Update: The authors note that feature visualizations are no more helpful for lower layers. This seems consistent with the idea that the limitation is with the human observer as opposed to the stimuli. --\n\nThe authors only examine a single kind of feature visualization, which in my opinion limits the overall impact of the paper. For example, I wondered if visualizations that highlight relevant pixels or regions of an image would help observers in making predictions. -- Update: The authors have done more to acknowledge this limitation. --\n\n--\n\nI don’t have a strong recommendation to accept or reject, at the moment. The paper doesn’t seem particularly ground breaking. But it’s solid, and introduces a useful framework for evaluating feature visualizations. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "3ekHdWweDJ", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##Updated Review##\n\nI'd like to thank the authors for addressing each of the points I made in my review and taking the time to include material that answers many of the questions I had.  This paper is in my mind a novel and interesting submission and a clear accept.\n\n# Main Idea\nThe main idea is to study how well extremely activating images help humans to predict CNN activations.  The authors do so by comparing extremely activating images with exemplary natural images that also strongly activate a specific feature map (and use a psychophysics test to see which helps the human better).\n\n# Some quick thoughts\nThe authors point that many visualization methods blend response maximization with human-define regularization methods which are essentially artistic choices meant to make the image less noisy is well taken and correct.  And these regularizations impute their own biases on the resulting images which may make them less informative. \n\nIt is also well taken that units may be highly activated by more than one semantic concept, or active in combination with other units (which may convey more information than selectively maximizing for the single neuron’s activation).\n\nWhile the authors points are well taken, they do sort of sweep aside one aspect of feature visualization that is not captured by natural images.  That is that feature visualization shows how and where a particular neuron’s response is not constrained to natural image statistics (and is potentially vulnerable to image manipulations).  This isn’t revealed in their psychophysics experiment because they test on the prediction of typical natural images only, which won’t contain these deviations from the natural image manifold.  The disconnect in predictability for future natural images actually reveals information about the networks (not necessarily a failure of the visualization technique!)\n\nIf the only goal of explainable AI is to help us predict what natural image a neuron in a mid level feature map will respond to strongly, then this study shows that using natural images helps with that.  However, if it is also to elucidate something about the working of the networks beyond constraint to the manifold of natural images, then the bias in this study will mislead us about which method is more useful.\n\nThe experiment in Figure 5C (showing different numbers of samples of maximally exciting images of each class) reveals something interesting here.  How does the diversity regularization utilized in this section effect variance in the feature visualization images when compared to the different samples of naturally exciting images in this set? \n\n# Weaknesses:\n\nLots of broad claims made about feature visualization vs natural images but really only compare to one specific method of creating synthetic images and at a particular setting of extreme activation.\n\nFigure 4 would be easier to digest if the data were on a single common axis and the colors of the bar separated between synthetic and natural (as such it is a bit difficult to compare layer to layer. There is interesting information in this correspondence!  For example, why do layers 4b and layer 5b show similar performance for both natural and feature visualization images?  \n\nFigure 10 in the supplementary also shows very interesting relationship between the response strength between the natural images and the synthetic images.  It seems like attempting to balance the response level between image types would be a good control for future experiments.  This also highlights a point raised early, that the manifold of natural images imposes constraints on how exactly these images can be configured and thus actually can’t show us certain response characteristics of these neurons).  This gets into what exactly you are using these methods for, the predict natural images they may respond to or a deeper analysis of their response characteristics.", "extra_scores": {}, "extra_text": {}}, {"review_id": "2s1Ix1JcVsU", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThe manuscript presents a systematic analysis comparing the effectiveness of using the output (explanations) produced by input reconstruction methods (a.k.a. feature visualization) when compared to exemplar (natural) images for the task of explaining the activation of given layer in a neural network.\n\nWhile the technical contributions of the manuscript, its observations are of significant relevance for the continuously growing literature related with model explanation/interpretation methods applied to neural networks. \n\nOverall, the presentation of the manuscript is good, and to a good extent its content is clear and easy to follow. A significant set of related literature is referred to.\n\nReading the manuscript was both enjoyable and insightful.\n\nRegarding the novelty of the analysis, I am not familiar with efforts analyzing model explanation and interpretation algorithms from the human side. To the best of my knowledge, this is one of the first I have read related to input reconstruction methods.\n\nHaving said that, my main concerns with the manuscript are the following:\n\nThe natural image baseline needs to be explicitly defined. \nMore specifically, a description on how the natural images used in the experiments are selected is not in place. This is a critical aspect of the whole study. This description is not present in the manuscript, it is hosted in the supplementary material instead. If this manuscript is to be accepted, and to ensure that it is self-contained, the description of this baseline should be in the body of the manuscript. \n \nWhen comparing experts vs. lay users, experts are considered as individuals with experience with CNNs. \nThis begs the question, what if you have experts that beyond being familiar with CNNs are also familiar with explanation methods? It would be insightful to further conduct experiments with more specialized users, i.e. users with familiarity on the explanation methods.\n\nThere is quite some redundancy in the content presented in Sections 4 and 5, I would encourage the combinations of these two sections in favour of providing more details related to how the natural images baseline is defined.\n\nAs admitted by the manuscript, one of its limitation is its limited focus to only considering the feature visualization methods from Olah et al, 2017. Including different examples from other families of model explanation methods would have strengthened the manuscript significantly.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper asks a simple question: Do extreme-activating synthetic images for a CNN unit help a human observer to predict that unit’s response to natural images, compared with maximally/minimally activating natural images. They authors conducted well-designed human studies and found that the synthetic images provide useful information for prediction, but that the benefit is smaller than that provided by simply presenting people with other natural images that maximally or minimally activate a unit.

The paper provides one reasonable metric for evaluating feature visualizations. Feature visualizations are widely used, but there are very few objective metrics for evaluating them. This methodological contribution is the main contribution of this work and could be impactful. Although the conclusion is not very surprising, the paper makes a potentially good contribution to the literature.

Decision: Accept (Poster)