Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: This paper creates artificial attention maps for 25 images, using a number of passive methods for creating attention maps crossed with four different convnet architectures, as well as two “active attention” models. These attention maps are correlated with human maps from three different experiments designed to test sensitivity or information across an image and three existing datasets of human eye tracking experiments in three different tasks using the same 25 images. The three experiments run here were 1) an experiment to measure how informative humans consider patches of an image compared to the original image, resulting in an information map; 2) a change sensitivity experiment where a dot is moved slightly, at a fixed grid over the image, resulting in a d’ map over the image; and 3) a kind of mmc with people experiment where subjects indicate where they think a dot was in an image and the next subject has to make the same judgment using the dot placed by the previous subject. 

From these six experiments, they compute the first principal component of the six maps, which they term the “human PC”. They then compute the correlation of the various combinations of attention method with architecture, and find that Smooth Grad with guided backpropagation (sgbp) applied to AlexNet trained on ImageNet provides the highest correlation with the human maps. Since this is correlational, they then test how well humans can make judgments of images using the maps, and again, AlexNet with sgbp works best compared with a low-performing model, and in general, images gated by the correct maps for the image vs. the map for another image are statistically significantly better. Similarly, they came up with a measure of how well the human maps worked for the different models. 

main_review: In general, this review is a bit brief because the paper is obviously excellent.

Originality: This is highly original, although there have been several previous attempts to do this, that are all cited. I would have preferred that the differences between what had been done in the past and the current experiments had been described, instead of just citing the three prior works. 

Quality: High. One slightly odd measurement was the way they decided how to rate the models’ performances using the human maps. They took the highest output for each network (e.g., which of the 1k outputs was highest for a network trained on imagenet) and measured how far that output fell in the ranking of outputs when given the correct human map vs. the incorrect one for the image. This was done because the masked images were out of distribution for the models. I would have preferred that they used transfer learning on the images to train a final linear layer to distinguish the 25 images, and then see which mask  changed the output the most. But I’m ok with what they did.

I would have liked it slightly better if they had completed the 2x2 table and tested humans with the human maps and the networks with the network maps, providing an upper bound to compare with. 

Clarity: The paper is clearly written.

 Significance: The results are quite interesting, as they give a new methodology for rating how human-like the information used by the neural nets is, and which method for computing an attention map gives the most human-like results
. 
Limitations and societal impact. They discuss how human biases might creep into their models and suggest that ways to ameliorate this should be investigated. They also mention the limitation that all this is based on only 25 images, but they justify this because of the large number of subjects needed for each experiment, which ran into the thousands.

Having read the authors' response, and the other reviews, I am now going to "make a case for this paper [I] scored so well." Since the only really negative review is now from reviewer tunC (marginally below acceptance threshold), I will respond to that specifically.
1) Cognitive Science has always had a place at NeurIPS. As a cognitive scientist myself, and a reviewer for over 30 years, I increasingly find that newer reviewers make this statement more and more often. It's a bit unfortunate that at a "neural information processing systems" conference, that reviewers ding papers that compare these systems to the best example we have of one, humans. Sigh.
2) This paper makes substantial contributions to the science! They tested over 6k subjects! They compared four architectures and k different saliency mechanisms, where k is some small integer! They backed up the correlational results with causal studies! What more do you want? Yes, they didn't test hand-designed non-deep model saliency mechanisms, and now they did. One of these mechanisms worked as well as AlexNet with sgbp. Ok, but the point here is that saliency is an emergent property of the model and mechanism used - AlexNet was not designed as a model of saliency. 
3) They effectively responded to all of your complaints, as far as I can tell.




limitations_and_societal_impact: yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper investigates visual selectivity (i.e. which areas in images are most informative) in humans and artificial neural networks. Human attention maps are generated using different behavioral tasks, while artificial attention maps are generated from different ANN architectures. A simple ANN architecture with a passive attention mechanism is revealed as the best predictor of human visual selectivity. Using recognition experiments, the authors show that ANN-based attention maps enhance human recognition and viceversa. 
main_review: The paper describes a comprehensive investigation of the correspondence between human and machine attention. The authors compare behavioral and eye tracking data to an array of ANN architectures with different attention mechanisms. The question of whether the areas in images that ANNs pay attention to are similar to those humans pay attention to is an interesting one, and it's great to see it addressed using a wealth of experimental data and ANN architectures.

The authors mention some previous work on this topic, but it's not very clear to me where this work sits in relation to this context (e.g. more comprehensive approach/different methods?) Furthermore, the work they cite mostly concludes that visual selectivity is different between ANNs and humans; it would be interesting to see some discussion of why the authors think their work reaches a different conclusion.

Although correlations between the different behavioral measures are reported, it would be great to also get an idea of the cross-participant reliability in the behavioral experiments. This could help understand the level of noise in the data and could even be used to estimate a noise ceiling for how well a good model would be expected to fit the data.

Furthermore, I found the combination of results from different behavioral tasks and the human PC presented in a slightly confusing way. Given that most behavioral measures do correlate fairly well, I am not sure the authors' interpretation that the human PC reflects a shared latent component is the only possible one. Would it not be possible that reducing the data in this way removes some of the variability due to noise and thus improves the results? I would also like to better understand how the human PC is calculated – is this the first component resulting from PCA and if so, how much variance does it explain? The explanation in appendix C1 seems a bit unclear on this. 

Furthermore, since the human PC is used as the 'best measure' of human behavior, I am wondering why it was not used to mask the images presented to the ANN in Section 6 (as far as I can tell this section discusses separate masks based on the different behavioral tasks).

The optimization of the smoothing parameter is another aspect that could be explained a bit more clearly (especially in lines 171-175). Since the parameter was optimized based on the target correlation, I wonder whether this introduced some circularity in the analysis. Do the ANN architectures rank the same if the correlation is calculated between the raw?

It is great to see validation experiments addressing the overlap between human and machines in terms of visual selectivity. Although these experiments show that human/ANN maps improve recognition compared to randomized maps, they don't address the usefulness of these maps relative to comparable methods (e.g. masks derived from different participants or different ANN architectures). It is hard to tell from these results whether these maps really provide an 'advantage' in recognition, and it would be great to see these aspects mentioned in the discussion. Furthermore, it is difficult to judge from Figure 4D how meaningful the difference in d' is - what would be the expected d' of the 'true' model? Such difficulties in model comparison are common in this type of work, but could be discussed a bit more.

Finally, the main finding of this paper seems to be that a relatively simple model, without an active attention module, captures human visual selectivity patterns better than more complex models. However, the discussion seems relatively limited in interpreting the results, e.g. are there any conclusions that can be drawn from this about humans/ANNs? How do the authors reconcile the fact that the type of attention explains much of the variance in performance, but on average active attention models do as well as passive attention ones? 

Minor comment: In places, some analyses could be described more clearly (eg. lines 195-198, the analysis done to quantify variance is described rather vaguely)

-----------------------------------------
After author response:

I thank the authors for addressing my comments so thoroughly. I think many of these issues just required more clarification, and my concerns have been addressed well in this rebuttal. In regards to Q8 - I did not mean to suggest that a different measure of accuracy should be used. By 'true' model, what I meant was that there is some individual variability in human behavior too, and so a 'great' result might actually be expected to sit below 100%. I think expanding the discussion, plus the additional checks performed by the authors, will improve the paper, and so I'm increasing my rating to 7. 
limitations_and_societal_impact: The authors acknowledge the main limitations of the work, as well as potential impacts.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this paper, the authors systematically compare visual saliency maps obtained from deep learning models against human visual selectivity maps obtained through a series of large-scale online experiments involving a variety of perceptual tasks. The correlation between maps greatly varies depending on neural network architecture, visualization technique and behavioral task. In some cases (e.g., SGBP technique applied to AlexNet architecture) the model’s map achieves a very good match with human’s maps, suggesting a remarkable similarity between humans and machines in terms of how they select information contained in visual stimuli. This finding is further corroborated by two recognition experiments, where performance of both humans and deep nets is shown to decrease when images are masked using “incorrect” maps.
main_review: Originality: This research work is interesting and novel: a systematic comparison between human visual selectivity and state-of-the-art machine vision models is still lacking. The methodology used to build human maps is taken from very recent experimental approaches and the methodology used to build ANNs maps is based on state-of-the-art techniques.

Quality: The methodology is overall sound, and the perceptual tasks used to probe human observers are appropriate. It is also remarkable that the authors implemented a wide range of passive and active attention techniques to derive saliency maps and tested a good range of deep learning architectures. However, the paper as it stands has several weaknesses, which I list here:

-	I am not fully convinced that NeurIPS would be the best venue to present these findings (maybe a more cognitively oriented conference / journal could be a better option). However, validation of neural network models against humans is one of the subject areas of NeurIPS, hence I should acknowledge that this is a matter of opinion.
-	All models considered have been trained on datasets (ImageNet and CIFAR) containing mostly single-object images, in contrast with the testing stimuli that represent complex visual scenes. I doubt that the visual saliency of a complex scene is simply the sum of visual saliency of single objects.
-	It would be useful to include a “non-deep learning” baseline to better establish whether neural networks really represent the state-of-the-art in modeling human visual selectively. I was surprised that the authors did not even mention classical saliency-based methods (for a review, see Borji and Itti, “State-of-the-art in visual attention modeling” IEEE TPAMI 2012).
-	I think the results presented in Fig. 4D are particularly informative, but only if the “Low” column refers to the AlexNet-Scorecam (as mentioned in Line 235). However, from the caption of Fig. 4 it seems that the “Low” column refers instead to the AlexNet-Gradcam maps, which I think is not particularly interesting to show given that such masking condition returns almost unperceivable images. It would be more informative to show that SGBP is still better than Scorecam, even if that method achieves a decent correlation with human PC.
-	Fig. 5 is never referred to in the text (it is mistakenly called Fig. 3 or 4).

Clarity: The paper is written in a very clear way and properly organized. I enjoyed reading it. The discussion has a good balance between technical details and more general topics related to human perception and cognitive psychology.

Significance: This work could promote further research and stimulate the interest of the deep learning community toward more systematic comparisons between neural network models and human observers.

---------------------------
I think the authors properly addressed my concerns. I am curious to read the final version of the paper with the new results. My scored changed to 6.

limitations_and_societal_impact: The authors explicitly discuss the limitations of their research approach.
review_title: null

Meta-review: This paper is a real tour-de-force, containing "data from 78 new experiments and 6,610 participants"  and "6 distinct behavioral tasks including visual discrimination, spatial localization, recognizability, free-viewing, cued-object search and saliency search fixations."  Though this paper is different than the majority of NeurIPS submissions, I think the breadth of the experimentation, and the relation to current trends in DL make it a great contribution and a breath of fresh air.  I think it will be of interest to NeurIPS attendees, could spark some interesting discussion and inspire future work.
Decision: Accept (Oral)