Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: --- Summary ---

This paper focuses on feature visualizations that generates maximally activating images for a given hidden node to understand inner workings of CNNs. They compare the informativeness of these images compared to natural images that also strongly activate the specified hidden node, and find that natural images help human better to answer which other test natural images are also maximally activating.

--- Pros ---

1. The writing is crisp and excellent: the flow in the intro is easy to follow, the related work is thorough, and the figures are designed very well.

2. The experimental design is clear and reasonable. The statistical comparisons are rigorous and the visualizations are clear.

3. I like the subjective score section: the qualitative conversations are intersting to read.

--- Comments ---

1. Are there some examples that natural images can not help the participant predict while synthetic images can, and vice versa? It might be interesting to see in what circumstances which methods help.

2. The conclusion is not surprising given the task is designed to find other test natural images. But this paper does highlight the shortcomings of synthetic feature visualizations that are not easy for human to understand.

--- Overall evaluation ---

The writing is great and clear. Although I find the conclusion is not very surprising, I still enjoy reading this paper throughout its rigorous experimental setups like experts v.s. laymen, hand-picked v.s. random images, number of images presented, and subjective scores etc. 
review_title: Official Review #1

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: ##Updated Review##

I'd like to thank the authors for addressing each of the points I made in my review and taking the time to include material that answers many of the questions I had.  This paper is in my mind a novel and interesting submission and a clear accept.

# Main Idea
The main idea is to study how well extremely activating images help humans to predict CNN activations.  The authors do so by comparing extremely activating images with exemplary natural images that also strongly activate a specific feature map (and use a psychophysics test to see which helps the human better).

# Some quick thoughts
The authors point that many visualization methods blend response maximization with human-define regularization methods which are essentially artistic choices meant to make the image less noisy is well taken and correct.  And these regularizations impute their own biases on the resulting images which may make them less informative. 

It is also well taken that units may be highly activated by more than one semantic concept, or active in combination with other units (which may convey more information than selectively maximizing for the single neuron’s activation).

While the authors points are well taken, they do sort of sweep aside one aspect of feature visualization that is not captured by natural images.  That is that feature visualization shows how and where a particular neuron’s response is not constrained to natural image statistics (and is potentially vulnerable to image manipulations).  This isn’t revealed in their psychophysics experiment because they test on the prediction of typical natural images only, which won’t contain these deviations from the natural image manifold.  The disconnect in predictability for future natural images actually reveals information about the networks (not necessarily a failure of the visualization technique!)

If the only goal of explainable AI is to help us predict what natural image a neuron in a mid level feature map will respond to strongly, then this study shows that using natural images helps with that.  However, if it is also to elucidate something about the working of the networks beyond constraint to the manifold of natural images, then the bias in this study will mislead us about which method is more useful.

The experiment in Figure 5C (showing different numbers of samples of maximally exciting images of each class) reveals something interesting here.  How does the diversity regularization utilized in this section effect variance in the feature visualization images when compared to the different samples of naturally exciting images in this set? 

# Weaknesses:

Lots of broad claims made about feature visualization vs natural images but really only compare to one specific method of creating synthetic images and at a particular setting of extreme activation.

Figure 4 would be easier to digest if the data were on a single common axis and the colors of the bar separated between synthetic and natural (as such it is a bit difficult to compare layer to layer. There is interesting information in this correspondence!  For example, why do layers 4b and layer 5b show similar performance for both natural and feature visualization images?  

Figure 10 in the supplementary also shows very interesting relationship between the response strength between the natural images and the synthetic images.  It seems like attempting to balance the response level between image types would be a good control for future experiments.  This also highlights a point raised early, that the manifold of natural images imposes constraints on how exactly these images can be configured and thus actually can’t show us certain response characteristics of these neurons).  This gets into what exactly you are using these methods for, the predict natural images they may respond to or a deeper analysis of their response characteristics.
review_title: Interesting results, study has a natural bias that may constrain broadness of claims

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper asks a simple question: do extreme-activating synthetic images for a CNN unit help a human observer to predict that unit’s response to natural images, compared with maximally/minimally activating natural images. The authors present human observers with images synthesized to maximally or minimally activate a CNN unit, and then ask observers to make a binary choice as to which of two subsequently presented natural images will yield a larger unit response. They find that the synthetic images provide useful information for prediction, but that the benefit is smaller than that provided by simply presenting people with other natural images that maximally or minimally activate a unit. 

--

Pros:

The paper presents a simple and reasonable framework for evaluating the utility of synthesized images in allowing human observers to make predictions about how a CNN will behave. 

The paper is clear and to-the-point. 

The paper examines the utility of synthetic images at multiple stages/layers of a CNN, finding similar results at all stages. 

I liked that the authors considered both experts and lay people.

--

Limitations and questions:

Feature visualizations have many potential goals, only one of which is to allow a human observer to predict network behavior for natural images. For example, feature visualizations may provide insights about how the network’s behavior deviates from that of a human observer, by showing that the network relies on features that differ from human-intuitive features. Feature visualizations might also be used to compare the strategies used by two different networks to perform the same task, which might be hard to do using natural images alone. This multiplicity of uses is something which I think should be highlighted more prominently. -- Update: the authors have made minor text changes to note limitations. --

I wondered what the results would look like for a simpler case: a linear filter? In the case of a linear filter, it’s clear that the filter captures everything necessary to predict the response of the unit. But it's not obvious to me how good human observers would be at using the filter to predict how a linear filter would behave in natural images. Would the results be the same if the authors looked at first the layer of the network? If so, does this suggest that humans are just bad at using synthetic stimuli to make predictions in natural images? -- Update: The authors note that feature visualizations are no more helpful for lower layers. This seems consistent with the idea that the limitation is with the human observer as opposed to the stimuli. --

The authors only examine a single kind of feature visualization, which in my opinion limits the overall impact of the paper. For example, I wondered if visualizations that highlight relevant pixels or regions of an image would help observers in making predictions. -- Update: The authors have done more to acknowledge this limitation. --

--

I don’t have a strong recommendation to accept or reject, at the moment. The paper doesn’t seem particularly ground breaking. But it’s solid, and introduces a useful framework for evaluating feature visualizations. 

review_title: One reasonable framework for evaluating synthetic feature visualizations

final_score: rating: 6: Marginally above acceptance threshold
review_content: 
The manuscript presents a systematic analysis comparing the effectiveness of using the output (explanations) produced by input reconstruction methods (a.k.a. feature visualization) when compared to exemplar (natural) images for the task of explaining the activation of given layer in a neural network.

While the technical contributions of the manuscript, its observations are of significant relevance for the continuously growing literature related with model explanation/interpretation methods applied to neural networks. 

Overall, the presentation of the manuscript is good, and to a good extent its content is clear and easy to follow. A significant set of related literature is referred to.

Reading the manuscript was both enjoyable and insightful.

Regarding the novelty of the analysis, I am not familiar with efforts analyzing model explanation and interpretation algorithms from the human side. To the best of my knowledge, this is one of the first I have read related to input reconstruction methods.

Having said that, my main concerns with the manuscript are the following:

The natural image baseline needs to be explicitly defined. 
More specifically, a description on how the natural images used in the experiments are selected is not in place. This is a critical aspect of the whole study. This description is not present in the manuscript, it is hosted in the supplementary material instead. If this manuscript is to be accepted, and to ensure that it is self-contained, the description of this baseline should be in the body of the manuscript. 
 
When comparing experts vs. lay users, experts are considered as individuals with experience with CNNs. 
This begs the question, what if you have experts that beyond being familiar with CNNs are also familiar with explanation methods? It would be insightful to further conduct experiments with more specialized users, i.e. users with familiarity on the explanation methods.

There is quite some redundancy in the content presented in Sections 4 and 5, I would encourage the combinations of these two sections in favour of providing more details related to how the natural images baseline is defined.

As admitted by the manuscript, one of its limitation is its limited focus to only considering the feature visualization methods from Olah et al, 2017. Including different examples from other families of model explanation methods would have strengthened the manuscript significantly.

review_title: A potentially good contribution to the literature

Meta-review: This paper asks a simple question: Do extreme-activating synthetic images for a CNN unit help a human observer to predict that unit’s response to natural images, compared with maximally/minimally activating natural images. They authors conducted well-designed human studies and found that the synthetic images provide useful information for prediction, but that the benefit is smaller than that provided by simply presenting people with other natural images that maximally or minimally activate a unit.

The paper provides one reasonable metric for evaluating feature visualizations. Feature visualizations are widely used, but there are very few objective metrics for evaluating them. This methodological contribution is the main contribution of this work and could be impactful. Although the conclusion is not very surprising, the paper makes a potentially good contribution to the literature.

Decision: Accept (Poster)