Venue: NeurIPS 2020 Workshop SVRHM

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper covers the central question of, “How accurately do feature visualizations represent a CNN’s inner workings (or in short, how useful are they)?”. The authors probe at this question using a controlled psychophysical study to quantify how informative feature visualization technique from Olah et al are compared to natural images. Their study finds that synthetic images help humans predict the maximally activating image, but natural images are even more helpful.

My main concern is that the question of how accurately a feature visualization technique represents the inner workings of a CNN is actually different from the question of how useful it is for a human experimenter. There may be tension between how interpretable an explanation is vs. how correct it is. Just as an example, the work of Geirhos et al., 2019 (https://arxiv.org/abs/1811.12231) suggests that ImageNet-trained models are biased towards texture rather than shape in order to make classifications - now we would want the maximally activating image to reflect this! Say we have an ImageNet-trained neural network and we are interested in how it classifies Golden Retrievers, and in this hypothetical example the network actually detects fur features (texture) rather than full dog features (shape). In this example, the synthetic feature visualizations might look like fur texture, but using natural images from ImageNet as stimuli for the task outlined in this paper would result in only examples of full dogs as possible reference images (because there are no fur pictures or very zoomed in dog pictures in ImageNet). While these reference images may help a person guess the right query image (which will also be a full picture of a dog, because it is taken from ImageNet) faster and more accurately, it may also lead the experimenter to believe that the network performs its classifications by looking for full-dog features (when in reality the network is looking for fur texture). Understanding the network via natural images in this hypothetical scenario is therefore perhaps useful for humans, but not necessarily accurate to the CNN’s inner workings.

Further, I find the central result of the paper - that humans are both faster and more accurate on natural images - not very surprising. Just as neural networks have trouble performing tasks on out-of-distribution data, we as humans are trained on natural images and I can imagine it is harder for us to interpret these images synthesized via gradient ascent.

Pros:
- The work is very original, to the best of my knowledge
- I think the finding that experts perform just as well as laypeople on this task is indeed very interesting
- The sub-study on how to best present images (maximally activating and minimally activating images) is also interesting!
- The design of the task from a psychology standpoint looks sound


Cons
- I personally think the main result that people are better with natural images is extremely obvious, for reasons mentioned above. - However if it is non-obvious I would love to see more of that intuition covered in the text
- I don’t understand the implications of this work for the machine learning community. There are problems with using natural images only to try and understand the inner workings of a CNN, for example the shape vs. texture example described above. Similarly, given a neural network with no knowledge of what classes it was trained on, it may be impossible to find a useful maximally activating natural image for this network. Where would you even begin the search for images?

For the reasons outlined I feel borderline on this work. Hopefully the other reviewers can also provide more insight.

review_title: paper review

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: ## Summary

This paper presents experiments testing whether, given information about which visual information maximally activates an intermediate-layer CNN feature map, people are able to infer which of two natural images would strongly activate it. In one condition, the visual information about feature preference is in the form of synthetic feature visualizations, and in the other, it is natural images. Across experiments, the authors vary whether only maximally activating, or both minimally and maximally activating image(s) are shown, and how many of each (1-9) are presented. The behavioral task is a 2AFC in which participants indicate which image would be strongly activating, along with their level of confidence in this judgment. Additionally, the authors compare the judgments of experts (experienced with CNNs) and lay participants. The main findings are that people perform better on the task given natural reference images, though they are still able to perform significantly above chance given synthetic feature visualization reference images. Experts do not show an advantage over non-experts, and show the same pattern of performance (better given natural reference images). Finally, participants perform better given many reference images, and when shown both minimally- and maximally-activating reference images rather than just maximally-activating ones. Based on these experiments, the authors conclude that feature visualizations are informative for inferring which natural images strongly activate an intermediate-layer feature map, but that synthetic feature visualizations are less informative than strongly activating natural images, and this is true for experts and non-experts alike.

## Pros

The authors make a compelling case for quantifying the utility of explainability methods like synthetic feature visualization. The work is relevant to a broad range of communities.

The experiments are well designed to test the specific question of whether synthetic feature visualizations provide enough training data for people to classify strongly activating images above chance. The inclusion of the natural reference images is a useful baseline. The authors consider a number of variables, for example how many reference images, and of what type, maximize human task performance. Overall, the work has several interesting and novel take-aways.

The paper is well written and the methods are for the most part clearly described (minor caveats below) to a standard of replicability. 

## Cons

The authors make the strong claim that "Natural images are more informative for interpreting CNN activations than synthetic feature visualizations" (the title). However, the paper largely ignores the main reason that feature visualization is thought to be useful: it provides information about *why* -- or what about -- an image drives activation. A visualization of a snout detector is more specific than an image of a dog. The current set of experiments does not address the difference in informativeness along this dimension, but there are a number of questions that could be asked. I hope that in follow-up work, the authors investigate experimental methods for probing this aspect of feature visualization.

I appreciate the general space constraints, but the authors should nevertheless include key details about the objective used for synthetic feature visualization in the main text, in particular the fact that they were optimizing a channel objective.

In Figure 1, it was somewhat difficult to parse what constituted a trial. By contrast, the format of the example trials provided in the Supplement was clear. The authors might consider using boxes to group the synthetic and natural image pairs, respectively.

The authors use the phrase "forward simulation" to describe the behavioral task. This descriptor strikes me as a misnomer: participants are trying to infer which input images would be more likely to elicit strong activation given information about feature preferences. This is not what I would take "forward simulation" to mean. I don't insist that the authors change this phrase, but wanted to provide feedback on my impression of it.

## Overall evaluation

On balance, this work serves as a valuable starting point for quantifying the utility of synthetic feature visualization as a tool for understanding CNN representations, and presents several results that will be of interest to the community. Nevertheless, the authors should be more circumspect in their conclusions since they fail to address a key raison d'etre for feature visualization as a tool, namely that feature visualization provides information about *why* --what about -- an image drives activation.
review_title: Natural images are more informative for interpreting CNN activations than synthetic feature visualizations

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Summary of contributions: 
In this study, the authors investigate how informative are the synthetic/natural maximally activating images in interpreting the activations of a deep neural network and perform multiple psychophysics experiments to identify which query image will show strong activation given some reference images. They find that both synthetic and natural images are highly informative, and surprisingly natural images seem to be much more informative than synthetic images.

Significance:
This study is relevant to interpreting the CNNs and provides an evaluation measure of different explanation methods such as synthetic maximally activating images. A similar evaluation approach could be applied to other CNN explanation methods. 

Pros:
1. The paper is well written and the experiments are clearly explained.
2. The experiment design  is sound and focuses on the key questions about the information in maximally activating images i.e.  explanation quality evaluated using  human judgment
3. The results are reported with statistical tests supporting the evidence presented in the paper

Cons:
1.  The experiments are performed only using a single network InceptionV1

Suggestion to authors:
1. Although in supplementary, the authors show that natural images outperform synthetics images over different layers, it might be interesting to validate the result on another network.  The current study only investigates Inception V1 and therefore it raises the question of whether the results generalize to other types of networks.
2. This point is related to the previous one in the context that in order to investigate multiple networks, authors might consider using a few shot learning model as a proxy to human judgment and apply it to investigate how well the result generalizes across the networks. Then, based on these results the authors may consider selecting a few interesting models for psychophysics experiments to reduce the number of experiments for a large number of networks.  
review_title: A human psychophysics based evaluation method for CNN explanation methods, clearly written with sound experiment design and result presentation

Decision: Accept (Poster)