How Well do Feature Visualizations Support Causal Understanding of CNN Activations?

1 Introduction . It is hard to trust a black-box algorithm , and it is hard to deploy an algorithm if one does not trust its output . Many of today ’ s best-performing machine learning models , deep convolutional neural networks ( CNNs ) , are also among the most mysterious ones with regards to their internal information processing . CNNs typically consist of dozens of layers with hundreds or thousands of units that distributively process and aggregate information until they reach their final decision at the topmost layer . Shedding light onto the inner workings of deep convolutional neural networks has been a long-standing quest that has so far produced more questions than answers . One of the most popular tools for explaining the behavior of individual network units is to visualize unit responses via activation maximization [ 16 , 33 , 38 , 35 , 39 , 36 , 54 , 15 ] . The idea is to start with an image ( typically random noise ) and iteratively change pixel values to maximize the activation 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Human psychophysical experiment : Given references which query image activates a unit more ? B ASynthetic feature visualizations claim to show what “ causes ” activation C Synthetic images are notmore helpful for causal task of a particular network unit via gradient ascent . The resulting synthetic images , called feature visualizations , often show interpretable structures , and are believed to isolate and highlight exactly those features that “ cause ” a unit ’ s response [ 40 , 50 ] . Some of the synthetic feature visualizations appear quite intuitive and precise . As shown in Fig . 1A , they might facilitate distinguishing whether , for example , a unit responds to just an eye or a whole dog ’ s face . However , other aspects cast a more critical light on feature visualization ’ s “ causality ” : Generating these synthetic images typically involves regularization mechanisms [ 36 , 33 , 38 , 35 ] , which may influence how faithfully they visualize what “ causes ” a network unit ’ s activation . Furthermore , to obtain a complete description of a mathematical function , one generally needs more information than just knowing its extrema . In view of this , it is an open question how well a unit can be characterized by simply visualizing the arguments of its maxima . Finally , a crucial unknown factor is whether humans are able to obtain a causal understanding of CNN activations from these synthetic visualizations . Given these points , we develop a psychophysical experiment to test whether feature visualizations by Olah et al . [ 40 ] indeed allow humans to gain a causal understanding of a unit ’ s behavior . Our task is based on the reasoning that being able to predict the effect of an intervention is at the heart of causal understanding . Understanding the causal relation between variables implies an understanding of how changes in one variable affect another one [ 45 ] . In our proposed experiment , this means that participants can predict the effect of an intervention — in form of an image manipulation — if they know the causal relation between image features and a unit ’ s activations . Our experiment tests whether synthetic feature visualizations indeed provide information about such causal relations . Specifically , we ask humans which of two manipulated images activates a CNN unit more strongly . The interventions we test are obtained by placing an occlusion patch at two different locations in an image . Taken together , this experiment probes the purported explanation method ’ s advantage of causality in a counterfactual-inspired prediction set-up [ 14 ] . Besides feature visualizations , other visualization methods have been used to gain an understanding of the inner workings of CNNs . In this experiment , we additionally test alternatives based on natural dataset examples and compare them with feature visualizations . This is particularly interesting because dataset examples are often assumed to provide less “ causal ” information about a unit ’ s response as they might contain misleading correlations [ 40 ] . To continue the example above , dog eyes usually co-occur with dog faces ; thus , separating the influence of one image feature from the other one using natural exemplars might be challenging . Our data shows that : • Synthetic feature visualizations provide humans with some helpful information about the most important patch in an image — but not much more information than no visualizations at all . • Dataset samples as well as other combinations and types of visualizations are similarly helpful . • How easily the most important patch is identifiable depends on the unit , the images as well as the relative activation strength attributed to the patch . 2 Related Work . Feature visualizations are a widely used method to understand the learned representations and decision-making mechanisms of CNNs [ 33 , 38 , 35 , 39 , 36 , 54 , 15 , 40 , 37 ] . As such , several works leverage this method to study InceptionV1 [ 42 , 41 , 8 , 43 , 50 , 9 , 58 , 59 , 46 ] and other networks [ 6 , 21 , 20 ] ; others create interactive tools [ 61 , 44 , 52 ] or introduce analysis frameworks [ 65 ] . In contrast , some researchers question whether this synthetic visualization technique , first introduced by Erhan et al . [ 16 ] , is too intuition-driven [ 27 ] , and how representative the appealing visualizations in publications are [ 26 ] . Further , as already mentioned above , the engineering of the loss function may influence their faithfulness [ 36 , 33 , 38 , 35 ] . Another challenge is generating diverse feature visualizations to represent the different aspects that one single unit may respond to [ 42 , 36 ] . Finally , our recent human evaluation study [ 5 ] found that while these synthetic images do provide humans with helpful information in a forward simulation-inspired task , simple natural dataset examples are even more helpful . Human evaluation studies are extensively used to quantify various aspects of interpretability . As an alternative to pure mathematical approximations [ 2 , 66 , 57 , 63 ] , researchers not only evaluate the understandability of explanation methods in psychophysical studies [ 7 , 34 , 5 ] , but also trust in these methods [ 28 , 64 ] ) as well as the human cognitive load necessary for parsing explanations [ 1 ] or whether humans would follow an explained model decision [ 47 , 13 , 48 ] . A recent study even demonstrates that metrics of the explanation quality computed with human judgment are more insightful than those without [ 4 ] . Counterfactuals are a popular paradigm for both creating as well as evaluating explanation methods . Intuitively , they provide answers to the question “ what should I change to achieve a different outcome ? ” — in the context of machine learning explanation methods , usually the smallest , realistic change to a data point is of interest . As examples , counterfactual explanation methods have been developed for vision- [ 22 ] and languagebased [ 62 ] models as well as for model-agnostic scenarios [ 51 ] . Further , they are set into context of the EU General Data Protection Regulation [ 60 ] . Ustun et al . [ 56 ] investigate feasible and least-cost counterfactuals , while Mahajan et al . [ 32 ] and Karimi et al . [ 25 ] take feature interactions into account . To evaluate — rather than create — explanation methods , researchers often follow the “ counterfactual simulation ” task introduced by Doshi-Velez and Kim [ 14 ] : Humans are given an input , an output , and an explanation and are then asked “ what must be changed to change the method ’ s [ model ’ s ] prediction to a desired output ? ” Doshi-Velez and Kim [ 14 ] . Based on this task , Lucic et al . [ 30 ] test their new explanation method and Hase and Bansal [ 24 ] compare different explanation methods to each other . In this project , we design a counterfactual-inspired task to evaluate how well feature visualizations support causal understanding of CNN activations . This is the first study to apply such a paradigm to understanding the causes of individual units ’ activations . In order to scale the experiments , we simplify our task by having participants choose between two intervention options , rather than having them freely determine interventions themselves . 3 Methods . We run an extensive psychophysical experiment with more than 12 , 000 trials distributed over 323 crowdsourced participants on Amazon Mechanical Turk ( MTurk ) and two experts ( the two first authors ) .1 For more details than provided below , please see Appx . Sec . A.1 . Design Principles Overall , our experimental design choices aim at ( 1 ) the best performance possible , meaning that we select images that make the signal as clear as possible ; ( 2 ) generality over the network , meaning that we randomly sample units of different layers and branches ( testing all units would be too costly ) ; and ( 3 ) easy extendability , meaning that we choose a between-participant design ( each participant sees only one reference image condition ) so that other visualizations methods can be added to the comparisons in the future . 3.1 Psychophysical Task . If feature visualizations indeed support causal understanding of CNN activations , this should enable humans to predict the effect of an intervention , such as how occluding an image region changes a unit ’ s activation . Based on this idea , we employ a two-alternative forced choice task ( chance performance : 50 % ) where human observers are presented with two different occlusions in an image , and asked to estimate which of them causes a smaller change to the given unit ’ s activation ( see Fig . 2 for an example trial ) . More specifically , participants choose the query image that they believe to also elicit a strong activation given a set of 9 reference images . Such references could for instance consist of synthetic feature visualizations of a certain unit ( purportedly “ causal ” ) , or alternative visualizations . To summarize , the task requires humans to first identify the shared aspect in the reference images and to then choose the query image in which that aspect is more visible . Since we do not make any assumptions about whether participants are familiar with machine learning , we avoid asking participants about activations of a unit in the CNN . Instead , we explain that an image would be “ favored ” by a machine , and the task is to select the image which is “ more favored ” . The complete set of instructions shown to participants can be found in Appx . Fig . 9 and 10 . In addition to each participant ’ s image choice , the subjective confidence level and reaction time are also recorded . 3.2 Stimulus Generation . To generate stimuli , we follow Olah et al . [ 40 ] and use an InceptionV1 network [ 53 ] trained on ImageNet [ 12 , 49 ] . Throughout this paper , we refer to a CNN ’ s channel as a “ unit ” and imply taking the spatial average of all neurons in one channel.2 We test units sampled from 9 layers and 2 Inception module branches ( namely 3× 3 and POOL ) . For more details on the generation procedures of the respective stimuli , see Appx . A.1.2 . We use five different types of reference images : • Synthetic references : The synthetic images are the optimization results of the feature visualization method by Olah et al . [ 40 ] with the channel objective for 9 diverse images . • Natural references : The reference images are the most strongly activating3 dataset samples from ImageNet [ 12 , 49 ] . • Mixed references : This is a combination of the previous two conditions : the 5 most strongly activating natural and 4 synthetic reference images are used . The motivation is that this condition combines the advantages of both worlds — namely precise information from feature visualizations and easily understandable natural images — and , thus , has the potential to give rise to higher performance in the task . Jointly looking at these two visualization types is common in practice [ 40 ] . 1Code and data are available at github.com/brendel-group/causal-understanding-via-visualizations . 2Other papers might refer to a channel as a “ feature map ” , e.g . [ 5 ] . 3To reduce compute requirements , we use a random subset of the training set ( ≈ 50 % ) . • Blurred references : To increase the informativeness of natural images for this task , we modify them by blurring everything but a single patch . This patch is chosen in the same way as in the maximally activating query image ( see below ) . Consequently , this method cues participants to the most important image feature . In a way , these images can be seen as an approximate inverse of the maximally activating query image and might improve performance on our task . • No references : This is a control condition in which participants do not see any reference images and have to solve the task purely based on query images . To generate query images , we place a square patch of 90× 90 pixels of the average RGB color of the occluded pixels into a most strongly activating image chosen from ImageNet . The location of the occlusion patch is chosen such that the activation of the manipulated image is either minimal or maximal among all possible occlusion locations . These images then yield the distractor and target query images respectively .

Summary: EDIT AFTER AUTHOR RESPONSE: I read the other reviews and I do not see the need to change my review.  The paper presents an experiment where users are trained, with or without the use of feature visualizations, to predict which of two images achieves higher activation on a unit from InceptionNet trained on ImageNet.  The query images are obtained by maximally activating and minimally activating occlusions of an natural image.  The general findings of the paper are that while feature visualizations help users get slightly higher accuracy than having no reference at all, there is no significant difference between synthetic visualizations and the highly activating natural references, and furthermore that user behavior is highly similar to the performance of a salience baseline that predicts user gaze on an image.