Exemplary Natural Images Explain CNN Activations Better than State-of-the-Art Feature Visualization

1 INTRODUCTION . As Deep Learning methods are being deployed across society , academia and industry , the need to understand their decisions becomes ever more pressing . Under certain conditions , a “ right to explanation ” is even required by law in the European Union ( GDPR , 2016 ; Goodman & Flaxman , 2017 ) . Fortunately , the field of interpretability or explainable artificial intelligence ( XAI ) is also growing : Not only are discussions on goals and definitions of interpretability advancing ( DoshiVelez & Kim , 2017 ; Lipton , 2018 ; Gilpin et al. , 2018 ; Murdoch et al. , 2019 ; Miller , 2019 ; Samek et al. , 2020 ) but the number of explanation methods is rising , their maturity is evolving ( Zeiler & Fergus , 2014 ; Ribeiro et al. , 2016 ; Selvaraju et al. , 2017 ; Kim et al. , 2018 ) and they are tested and ∗Joint first and corresponding authors : firstname.lastname @ uni-tuebingen.de †Current affiliation : Institute of Psychology and Center for Cognitive Science , Technische Universität Darmstadt ‡Joint senior authors ... or M ax im al ly Ac tiv at in g ... Synthetic Natural ... M in im al ly Ac tiv at in g ... Choose or Queries ... which image is strongly activating ? A Given these reference images ... B Synthetic Natural Chance 0.6 0.7 0.8 0.9 1.0 Pr op or tio n Co rr ec t Synthetic images are helpful Natural even more or Figure 1 : How useful are synthetic compared to natural images for interpreting neural network activations ? A : Human experiment . Given extremely activating reference images ( either synthetic or natural ) , a human participant chooses which out of two query images is also a strongly activating image . Synthetic images were generated via feature visualization ( Olah et al. , 2017 ) . B : Core result . Participants are well above chance for synthetic images — but even better when seeing natural reference images . used in real-world scenarios like medicine ( Cai et al. , 2019 ; Kröll et al. , 2020 ) and meteorology ( Ebert-Uphoff & Hilburn , 2020 ) . We here focus on the popular post-hoc explanation method ( or interpretability method ) of feature visualizations via activation maximization1 . First introduced by Erhan et al . ( 2009 ) and subsequently improved by many others ( Mahendran & Vedaldi , 2015 ; Nguyen et al. , 2015 ; Mordvintsev et al. , 2015 ; Nguyen et al. , 2016a ; 2017 ) , these synthetic , maximally activating images seek to visualize features that a specific network unit , feature map or a combination thereof is selective for . However , feature visualizations are surrounded by a great controversy : How accurately do they represent a CNN ’ s inner workings—or in short , how useful are they ? This is the guiding question of our study . On the one hand , many researchers are convinced that feature visualizations are interpretable ( Graetz , 2019 ) and that “ features can be rigorously studied and understood ” ( Olah et al. , 2020b ) . Also other applications from Computer Vision and Natural Language Processing support the view that features are meaningful ( Mikolov et al. , 2013 ; Karpathy et al. , 2015 ; Radford et al. , 2017 ; Zhou et al. , 2014 ; Bau et al. , 2017 ; 2020 ) and might be formed in a hierarchical fashion ( LeCun et al. , 2015 ; Güçlü & van Gerven , 2015 ; Goodfellow et al. , 2016 ) . Over the past few years , extensive investigations to better understand CNNs are based on feature visualizations ( Olah et al. , 2020b ; a ; Cammarata et al. , 2020 ; Cadena et al. , 2018 ) , and the technique is being combined with other explanation methods ( Olah et al. , 2018 ; Carter et al. , 2019 ; Addepalli et al. , 2020 ; Hohman et al. , 2019 ) . On the other hand , feature visualizations can be equal parts art and engineering as they are science : vanilla methods look noisy , thus human-defined regularization mechanisms are introduced . But do the resulting beautiful visualizations accurately show what a CNN is selective for ? How representative are the seemingly well-interpretable , “ hand-picked ” ( Olah et al. , 2017 ) synthetic images in publications for the entirety of all units in a network , a concern raised by e.g . Kriegeskorte ( 2015 ) ? What if the features that a CNN is truly sensitive to are imperceptible instead , as might be suggested by the existence of adversarial examples ( Szegedy et al. , 2013 ; Ilyas et al. , 2019 ) ? Morcos et al . ( 2018 ) even suggest that units of easily understandable features play a less important role in a network . Another criticism of synthetic maximally activating images is that they only visualize extreme features , while potentially leaving other features undetected that only elicit e.g . 70 % of the maximal activation . Also , polysemantic units ( Olah et al. , 2020b ) , i.e . units that are highly activated by different semantic concepts , as well as the importance of combinations of units ( Olah et al. , 2017 ; 2018 ; Fong & Vedaldi , 2018 ) already hint at the complexity of how concepts are encoded in CNNs . One way to advance this debate is to measure the utility of feature visualizations in terms of their helpfulness for humans . In this study , we therefore design well-controlled psychophysical experiments that aim to quantify the informativeness of the popular visualization method by Olah et al . ( 2017 ) . Specifically , participants choose which of two natural images would elicit a higher activa- 1Also known as input maximization or maximally exciting images ( MEIs ) . tion in a CNN given a set of reference images that visualize the network selectivities . We use natural query images because real-world applications of XAI require understanding model decisions to natural inputs . To the best of our knowledge , our study is the first to probe how well humans can predict intermediate CNN activations . Our data shows that : • Synthetic images provide humans with helpful information about feature map activations . • Exemplary natural images are even more helpful . • The superiority of natural images mostly holds across the network and various conditions . • Subjective impressions of the interpretability of the synthetic visualizations vary greatly between participants . 2 RELATED WORK . Significant progress has been made in recent years towards understanding CNNs for image data . Here , we mention a few selected methods as examples of the plethora of approaches for understanding CNN decision-making : Saliency maps show the importance of each pixel to the classification decision ( Springenberg et al. , 2014 ; Bach et al. , 2015 ; Smilkov et al. , 2017 ; Zintgraf et al. , 2017 ) , concept activation vectors show a model ’ s sensitivity to human-defined concepts ( Kim et al. , 2018 ) , and other methods - amongst feature visualizations - focus on explaining individual units ( Bau et al. , 2020 ) . Some tools integrate interactive , software-like aspects ( Hohman et al. , 2019 ; Wang et al. , 2020 ; Carter et al. , 2019 ; Collaris & van Wijk , 2020 ; OpenAI , 2020 ) , combine more than one explanation method ( Shi et al. , 2020 ; Addepalli et al. , 2020 ) or make progress towards automated explanation methods ( Lapuschkin et al. , 2019 ; Ghorbani et al. , 2019 ) . As overviews , we recommend Gilpin et al . ( 2018 ) ; Zhang & Zhu ( 2018 ) ; Montavon et al . ( 2018 ) and Carvalho et al . ( 2019 ) . Despite their great insights , challenges for explanation methods remain . Oftentimes , these techniques are criticized as being over-engineered ; regarding feature visualizations , this concerns the loss function and techniques to make the synthetic images look interpretable ( Nguyen et al. , 2017 ) . Another critique is that interpretability research is not sufficiently tested against falsifiable hypotheses and rather relies too much on intuition ( Leavitt & Morcos , 2020 ) . In order to further advance XAI , scientists advocate different directions . Besides the focus on developing additional methods , some researchers ( e.g . Olah et al . ( 2020b ) ) promote the “ natural science ” approach , i.e . studying a neural network extensively and making empirical claims until falsification . Yet another direction is to quantitatively evaluate explanation methods . So far , only decision-level explanation methods have been studied in this regard . Quantitative evaluations can either be realized with humans directly or with mathematically-grounded models as an approximation for human perception . Many of the latter approaches show great insights ( e.g . Hooker et al . ( 2019 ) ; Nguyen & Martı́nez ( 2020 ) ; Fel & Vigouroux ( 2020 ) ; Lin et al . ( 2020 ) ; Tritscher et al . ( 2020 ) ; Tjoa & Guan ( 2020 ) ) . However , a recent study demonstrates that metrics of the explanation quality computed without human judgment are inconclusive and do not correspond to the human rankings ( Biessmann & Refiano , 2019 ) . Additionally , Miller ( 2019 ) emphasizes that XAI should build on existing research in philosophy , cognitive science and social psychology . The body of literature on human evaluations of explanation methods is growing : Various combinations of data types ( tabular , text , static images ) , task set-ups and participant pools ( experts vs. laypeople , on-site vs. crowd-sourcing ) are being explored . However , these studies all aim to investigate final model decisions and do not probe intermediate activations like our experiments do . For a detailed table of related studies , see Appendix Sec . A.3 . A commonly employed task paradigm is the “ forward simulation / prediction ” task , first introduced by Doshi-Velez & Kim ( 2017 ) : Participants guess the model ’ s computation based on an input and an explanation . As there is no absolute metric for the goodness of explanation methods ( yet ) , comparisons are always performed within studies , typically against baselines . The same holds for additional data collected for confidence or trust ratings . According to the current literature , studies reporting positive effects of explanations ( e.g . Kumarakulasinghe et al . ( 2020 ) ) slightly outweigh those reporting inconclusive ( e.g . Alufaisan et al . ( 2020 ) ; Chu et al . ( 2020 ) ) or even negative effects ( e.g . Shen & Huang ( 2020 ) ) . To our knowledge , no study has yet evaluated the popular explanation method of feature visualizations and how it improves human understanding of intermediate network activations . This study therefore closes an important gap : By presenting data for a forward prediction task of a CNN , we provide a quantitative estimate of the informativeness of maximally activating images generated with the method of Olah et al . ( 2017 ) . Furthermore , our experiments are unique as they probe for the first time how well humans can predict intermediate model activations . 3 METHODS . We perform two human psychophysical studies2 with different foci ( Experiment I ( N = 10 ) and Experiment II ( N = 23 ) ) . In both studies , the task is to choose the one image out of two natural query images ( two-alternative forced choice paradigm ) that the participant considers to also elicit a strong activation given some reference images ( see Fig . 2 ) . Apart from the image choice , we record the participant ’ s confidence level and reaction time . Specifically , responses are given by clicking on the confidence levels belonging to either query image . In order to gain insights into how intuitive participants find feature visualizations , their subjective judgments are collected in a separate task and a dynamic conversation after the experiment ( for details , see Appendix Sec . A.1.1 and Appendix Sec . A.2.6 ) . All design choices are made with two main goals : ( 1 ) allowing participants to achieve the best performance possible to approximate an upper bound on the helpfulness of the explanation method , and ( 2 ) gaining a general impression of the helpfulness of the examined method . As an example , we choose the natural query images from among those of lowest and highest activations ( → best possible performance ) and test many different feature maps across the network ( → generality ) . For more details on the human experiment besides the ones below , see Appendix Sec . A.1 . In Experiment I , we focus on comparing the performance of synthetic images to two baseline conditions : natural reference images and no reference images . In Experiment II , we compare lay vs. expert participants as well as different presentation schemes of reference images . Expert participants qualify by being familiar or having practical experience with feature visualization techniques or at least CNNs . Regarding presentation schemes , we vary whether only maximally or both maximally and minimally activating images are shown ; as well as how many example images of each of these are presented ( 1 or 9 ) . Following the existing work on feature visualization ( Olah et al. , 2017 ; 2018 ; 2020b ; a ) , we use an Inception V1 network3 ( Szegedy et al. , 2015 ) trained on ImageNet ( Deng et al. , 2009 ; Russakovsky 2Code and data is available at https : //bethgelab.github.io/testing visualizations/ 3also known as GoogLeNet Synthetic Natural None Reference Images 0 0.2 0.4 0.6 0.8 1 Pr op or tio n Co rre ct p = 0.005 p = 0.003 p = 0.003A Synthetic Natural None Reference Images 0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 Re ac tio n Ti m e [ s ec ] p = 0.002 p < 0.001 p = 0.003 CB Synthetic Pr op or tio n of C or re ct T ria ls Natural None 1 2 3 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 1 2 3 1 2 3 Confidence Rating Chance Figure 3 : Participants are better , more confident and faster at judging which of two query images causes higher feature map activation with natural than with synthetic reference images . A : Performance . Given synthetic reference images , participants are well above chance ( proportion correct : 82± 4 % ) , but even better for natural reference images ( 92± 2 % ) . Without reference images ( baseline comparison “ None ” ) , participants are close to chance . B : Confidence . Participants are much more confident ( higher rating = more confident ) for natural than for synthetic images on correctly answered trials ( χ2 , p < .001 ) . C : Reaction time . For correctly answered trials , participants are on average faster when presented with natural than with synthetic reference images . We show additional plots on confidence and reaction time for incorrectly answered trials and all trials in the Appendix ( Fig . 16 ) ; for Experiment II , see Fig . 17. ) . The p-values in A and C correspond to Wilcoxon signed-rank tests . et al. , 2015 ) . The synthetic images throughout this study are the optimization results of the feature visualization method by Olah et al . ( 2017 ) with the spatial average of a whole feature map ( “ channel objective ” ) . The natural stimuli are selected from the validation set of the ImageNet ILSVRC 2012 dataset ( Russakovsky et al. , 2015 ) according to their activations for the feature map of interest . Specifically , the images of the most extreme activations are sampled , while ensuring that each lay or expert participant sees different query and reference images . A more detailed description of the specific sampling process for natural stimuli and the generation process of synthetic stimuli is given in Sec . A.1.2 .

Summary: This paper asks a simple question: do extreme-activating synthetic images for a CNN unit help a human observer to predict that unit’s response to natural images, compared with maximally/minimally activating natural images. The authors present human observers with images synthesized to maximally or minimally activate a CNN unit, and then ask observers to make a binary choice as to which of two subsequently presented natural images will yield a larger unit response. They find that the synthetic images provide useful information for prediction, but that the benefit is smaller than that provided by simply presenting people with other natural images that maximally or minimally activate a unit.