Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper designs and conducts a psychophysical experiment to evaluate how effective unit visualization is in interpreting a neural network. The experiments presents synthetic and natural images obtained through methods of unit activation maximization to human subjects and evaluate how these images could guide them in answering effects of occlusion of images to a network. Through extensive experiment controls at different levels, it arrives at the conclusion that, unit visualization with synthetic images is not particularly useful in helping us understand CNN activations. 
main_review: Originality: The psychophysical experiment design is novel and to my knowledge it is the first study that answers questions about unit interpretability in CNN with actual human behavioral data. 

Quality: The amount of controls that are designed into the experiment is very impressive. It is able to rule out most of the competing hypothesis related to this problem. Analysis of the decision patterns, reaction time, as well as comparison to expert evaluator are also very informative. 

Clarity: The paper is extremely well-written and organized. 

Significance: The behavioral results in this paper has a lot of insights and is important in guiding the field for the next interpretability methods. It would also encourage more validating experiments to be done on other visualization methods in general. 
limitations_and_societal_impact: A potential limitation of the experiment is that, the occlusion based judgement design could bias human subjects to match objects or specific object parts to the occluded images. Units in a CNN, especially the ones in early layers, don't have to function as object detector or object descriptor detector. In this case unit visualization might still be helpful in identifying the general kind of feature that a unit cares about (for example, texture or the horizon) but fails to inform human subjects in the occlusion task. 

It is not surprising to me that the saliency baseline yields similar performance compared to using maximally activated image and that the saliency baseline has the highest decision consistency with other conditions. This result can also be interpreted positively since it demonstrates that, through unit visualization, we verify that the network learns to look at the "important" part of the image for downstream classification. In another word, saliency models that are trained with human fixation data could be used not as a baseline but instead as a method in evaluating how well the network learns.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper put forward an objective psychophysical task to evaluate whether feature visualization could help humans understanding CNN activations. The authors recorded the details of their objective psychophysical task and adopted lots of baselines that ensure their experiments' preciseness. They also analyze the results from different angles.
main_review: This is a brand-new idea that evaluates the performance of feature visualization totally with psychophysical experiments. However, it is more like a social science document rather than a research paper target for NeurIPS. This paper overflowed with design details, such as 3.1 Psychophysical Task, 3.3 Structure of the Psychophysical Experiment, and 3.4 Ensuring High-Quality Data in an Online Experiment, which could be paced in the supplement. What’s more, the results of the experiment also are trivial. Besides the task accuracy in experiment 1, all other experiments are hard to understand the purpose.
limitations_and_societal_impact: Besides the above mentioned, there are more concerns:

(1) More evidence should be provided to support that why this experiment is sufficient for supporting the claim. Why a similar performance between visualization and nature image can cause the uselessness of visualization explanation? Also, a unit may be sensitive to different patterns (multiclass classification), why nature image fixed in a single class?

(2) The authors do not illustrate how those reference images are assigned. Since the activation of CNN is relevant to units, the reference images should be different if units change. However, this paper doesn't show the details. 

(3) It seems maximally activating synthetic image is full of one pattern, and it is understandable if the image is enlarged. However, when the synthetic image is the same size as the natural image, it is hard to recognize the pattern in the image. As the difference in query images is the location of the square patch, it is more reasonable to use a maximum activation patch as a reference rather than the same size image as the reference images. 

(4) The authors point out that one advantage of this method is its easy extendability. However, If researchers want to investigate additional visualization methods,  they still need to generate new query images and re-run the collected trials.

(5) This paper lacks novelty, and the results are not interesting. Too many details which seem like placeholders because readers cannot get any inspiration from those paragraphs.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: EDIT AFTER AUTHOR RESPONSE: I read the other reviews and I do not see the need to change my review.

The paper presents an experiment where users are trained, with or without the use of feature visualizations, to predict which of two images achieves higher activation on a unit from InceptionNet trained on ImageNet.  The query images are obtained by maximally activating and minimally activating occlusions of an natural image.  The general findings of the paper are that while feature visualizations help users get slightly higher accuracy than having no reference at all, there is no significant difference between synthetic visualizations and the highly activating natural references, and furthermore that user behavior is highly similar to the performance of a salience baseline that predicts user gaze on an image.

main_review: Originality:  Low. This work is similar to Borowski et al. 2021, but it does a more thorough evaluation encompassing more types of feature visualizations and more baselines.  It also uses an experimental design that is likely to be more powerful, as it compares two images that are obtained by occluding the same original image.  However it is not entirely clear that this methodological change is a strict improvement on Borowski et al. 2021 because such occluded distributions lie outside of the support of the natural image distribution and hence could fail to be representative of neural network activity for natural images, unless the network has been shown to be robust to occlusions.

Quality:  High. A lot of thought was put into the experimental design and the analysis of the conclusions.  In particular, decomposing the variation in accuracy by no only visualization method but also unit and contrast in activation helps build the case that the visualizations do not differ by much.  The details of collecting data on MTurk also build confidence in data quality.

Clarity:  The paper is clear and details of the methods are well-motivated and explained.

Significance:  Moderate.  The paper adds to the findings of Borowski et al. which pose serious questions for the feature visualization field.  Why are human rankings of feature activations so close to chance, and why don't visualizations (natural or synthetic) help more?

limitations_and_societal_impact: Yes.

It is clear that this work is only a stepping stone, and that more studies are needed to investigate why human predictions of unit activations are so poor, and whether new tools can be developed to increase our ability to make predictions of neural network unit activations.
review_title: null

Meta-review: This paper investigates whether feature visualizations, such as activation maximization, provide a benefit to humans in predicting model performance over simpler approaches such as the maximally-activating data samples. By performing a set of human experiments, the authors show that a) there is a benefit to these visualizations over no visualizations, but that b) there is no difference between these approaches and the use of natural images. Reviewers all found the experiments to be very carefully designed and rigorous, and praised the overall clarity of the paper. There is certainly a need for more thorough and careful examination of the interpretability methods that are widely used, and I think this paper will prove impactful both because of its observations and as a model for how to conduct such evaluations. I recommend it be accepted as a spotlight. 
Decision: Accept (Spotlight)