Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper re-introduces a very important topic in the ML + CV community that has not been studied enough: are computer vision models aligned to human visual perception (and if they are not, why should we care?). The authors introduce the neural harmonizer as a training procedure that aligns the human-relevant features with a CNN to train from scratch a couple of new models from classical CNN architectures (ResNet + VGG) and they show that their models improve human perceptual alignment while also maintaining ImageNet Test Accuracy and performance. Furthermore, they empirically show that networks trained with the Neural Harmonizer are more robust to a peripheral occlusion task, and most importantly -- that their change in performance rate is similar to that of a human without any additional fine-tuning on the network.
strengths_and_weaknesses: The paper’s main strengths is the paper’s theme as well as the very accessible way the paper has been written. Furthermore the paper’s experiments seem properly articulated and well-posed to support the main hypothesis and question that the authors introduce at the beginning of the paper. The psychophysics is pretty amazing.

However, the paper still has some weaknesses that should be addressed/clarified:
* **Weak Results:** Why are there only 2 reported neural networks that have been trained with the neural harmonizer (there are 2 purple dots)? I would have expected that for the claim to be “the neural harmonizer successfully increases man-machine perceptual alignment”, that there would be nearly all blue CNN dots transcribed to purple, including (potentially) the yellow ones which are the Transformers. Without these extra data points, the results seem very weak, and almost suggests cherry picking.
* **Missing References:**
    * “Emergent Properties of Foveated Perceptual Systems”. Deza & Konkle. ArXiv, 2020.
    * “Central and peripheral vision for scene recognition: A neurocomputational modeling exploration” Wang & Cottrell. JoV, 2017.
    * “Joint rotational invariance and adversarial training of a dual-stream Transformer yields state of the art Brain-Score for Area V4”. Berrios & Deza. Brain-Score Workshop 2022.
    * “Texture-like representation of objects in human visual cortex”. Jagadeesh & Gardner. PNAS 2022.
* **Experimental Results may provide evidence for alternative explanation**: The occlusion results from Figure 5 may also suggest that the neural harmonizer has enforced a central image bias in CNNs :O ! (So a partial rather than full human-perceptual alignment). It is expected that Transformers would not a priori have this bias. That being said: How would the Transformer’s curve trained with a Neural Harmonizer look like? In addition, what would the curve look like if the occlusion patterns were inverted, and one where to begin by starting to show peripheral regions of the image, and slowly showing the most important features last? Do the same patterns of results hold between humans and neural harmonizer trained networks? See for example the work of Deza & Konkle (ArXiv, 2020) and also Wang & Cottrell (JoV, 2017)

questions: See Weaknesses points (1) and (3).
limitations: None, the paper actually does a great job in introducing current limitations of modern DL systems. The paper is overall well written. Providing additional experiments and/or a good rebuttal may improve my rating. I am also generally curious to know what other reviewers have thought about the maturity of this paper. 

I'm really on the fence with this paper. It's amazingly well written, addresses an important problem in modern ML + CV -- via the proper cog-sci/psychology insights, but is still missing more results that back-up their claim with stronger evidence. There are no error bars for example, and authors could have definitely trained the other NN's with their Neural Harmonizer, to provide substantial evidence about how the Harmonizer provides better human-perceptual alignment.
review_title: null

final_score: rating: 2: Strong Reject: For instance, a paper with major technical flaws, and/or poor evaluation, limited impact, poor reproducibility and mostly unaddressed ethical considerations.
review_content: summary: The authors address the problem of evaluating the alignment of deep learning vision models with humans. To do so, they compare feature importance maps by humans, with saliency maps from DNNs. Additionally, they propose a training method (neural harmonizer) to improve such alignment. Finally authors present a psychophysic experiment using the rapid categorization paradigm to gather information of human decisions based on partial information of important features.
strengths_and_weaknesses: Strength:
- the topic is timely and relevant
- the paper is clear and easy to read
- the authors validate part of their claims with a human study
- code is provided for reproducibility

Weaknesses:
- Structure of the paper is not optimal. The neural harmonizer should probably be described introduced in the method section.
- The rapid categorization paradigm has already been used to compare humans and DNNs decisions and it could be beneficial to provide those references (the provided references are from the neurosc. field)
- Author exclusively use saliency maps from Simonyan et al. to reveal feature importance. However, studies have found severe weaknesses of such metrics that can also be drastically attacked. Authors could provide more arguments on why they chose to rely specifically and exclusively on this method, or eventually use more.
- Related to the previous point, the proposed neural harmonizer method optimises neural networks to change their feature importance (w.r.t. the selected method of saliency). I rather have a different interpretation than those provided by the authors on the effect of this approach: this might lead the network to learn more effective activation that lead to human-like feature importance (maybe leveraging spatial biases of human annotators) while not changing the underlying decision process. The improvement in accuracy might be due to the additional loss and training data (in fact, multi-tasking is performed). Which lead me to the following observation:
- Why do authors provide results for only two networks (ResNet50 and VGG16) of the 85 presented in the first experiment, and why those two? Their explanation is given in page 6, they are "selected because they are popular networks that sit on the boundary of the trade-off betweenDNN performance and alignment with humans". This seems to be not fully true from figure 4. Additionally, it might be very useful to know the average performance on the other architectures (or at least a random sample of them, if training them all is prohibitive) in order to support the claim that the method improves alignment without decreasing the performance. Especially important are the performance on DenseNet121, mentioned by the authors as the very limit of the trade-off between alignment and accuracy.
- Authors assume that feature importance maps from the two selected datasets represent where features which are important for the decisions are in a scene. However, I am not sure this aligns with locations people attend with their gaze. Authors should discuss this point in the paper. Eventually, an experiment involving saliency maps (using for example SALICON, a subset of imagenet with human visual saliency information) might be meaningful?

Minor:
- Page 4, row 2, "was was" (repetition)
- The authors refer to feature maps also as "visual strategies" often during the paper, however, I found this to be confusing since in the field this is more often used for visual scanpaths than for (static) saliency. I suggest use a different terminology.
- I am not sure wether presenting a partial image cropped is effective since, even when selecting a smaller area, people can see the rest of the image with peripheral vision. Trying to model this effect could have made the experiment more reliable.
- Section 4.1: motivate the use of spearman's rank-correlation or provide references
- In the paragraph "The neural harmonizer aligns human and DNN visual strategies.": authors mention "All of the three neural harmonizer models...", however they presented only two of them.
questions: I would be incline to change ma opinion if authors:
- provide strong motivations on the choice of the one method for saliency, or present results for at least another different approach.
- provide strong motivations on the choice of the two models to be improved with neural harmonizer, or provide average results for all initial 85 models
- clarify why using feature importance map is sufficient to explain and measure alignment, in particular w.r.t. research in saliency and scanpath that rely directly on eye-tracking data
limitations: The discussion of limitations is insufficient. The authors should point out that results are only provided for one explainability method, and only for two models. Additionally, they should point out how the notion of feature importance and what is actually captured by humans through the gaze might differ.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The authors conduct an empirical evaluation of the alignment of human and artificial neural network (ANN) saliency maps, and propose a method for improving it. The authors use saliency maps collected from ImageNet by Linsley et al. and compare them to those estimated from a variety of ANN's. In a separate experiment, the authors obscure images to varying degrees according to the human saliency maps, and measure the resulting degradation in classification performance in humans and ANN's, again comparing the two. Finally, the authors propose a method for aligning the saliency maps of humans and ANN's by jointly training the network for the original classification objective, and a measure of alignment with human saliency. The resulting models' saliency maps are indeed better aligned with human ones. 
strengths_and_weaknesses: The authors propose a sound evaluation of the alignment between human and ANN saliency maps. In particular, the experiment they propose which incrementally reveals images according to human saliency maps appears to be a good way of comparing the features used by humans and ANN's in an apples-to-apples manner. Whether this experiments reveals "how features are used" rather than "where these features are" is less clear: it instead appears to be a better controlled version of the original comparison between saliency maps. 

Secondly, the authors make good use of this evaluation to show that improving ImageNet classification accuracy doesn't lead to better alignment with human saliency maps, underscoring the limitations of ImageNet as a benchmark. It would be interesting to see if other computer vision benchmarks, such as object detection or semantic segmentation are more predictive of human saliency.

Finally, it is encouraging to see that co-training the network to reproduce saliency maps allows it to do so, without any cost to the ImageNet classification accuracy (or even with a slight improvement to it), however it is unclear what the scientific value of this result is. Aligning the training and testing objective usually improves test performance, hence improving the alignment with human saliency is hardly surprising. The fact that it improves ImageNet classification is more surprising, but is at odds with the first finding that models which improve classification accuracy do not improve the alignment with human saliency. 

A much more interesting point would be to investigate what training procedure, architecture, or data enables a model to reproduce human saliency **without resorting to training on this data**. This would indicate which of the above is most closely matches the conditions leading to the human visual system's choice of features for performing these tasks. 
questions: As discussed above, the proposed neural harmonizer is of limited scientific interest, as it simply aligns training and testing for this particular metric (alignment with human saliency maps). 

Assuming this were removed from the paper, could the authors instead re-analyze their data to study which training procedure, architecture, or data enables a model to reproduce human saliency maps without resorting to training on this data? For example 

- Given the same model architecture and training data, which training procedure (supervised, self-supervised, etc) leads to the best alignment with human saliency?

- Given the same training data and objective (e.g. supervised or self-supervised), which model architecture (ResNet, ViT, etc) leads to the best alignment with human saliency?

- Given the same architecture and training objective, which training data (ImageNet, ImageNet21k, etc) leads to the best alignment with human saliency?
limitations: There are no ethical or societal concerns associated with this work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Summary: This paper focusses on evaluating the alignment between visual strategies of DNNs and human perception. Specifically, the ask how this alignment evolves as architectures become more complex. For a fair comparison the paper focusses on only the rapid, feed-forward component of human perception, and suggests evaluating alignment on visual strategies as defined by a two step process---identifying regions with important features and how these regions are used to make a classification decision. For the first part, the paper evaluates alignment between DNNs and humans on three datasets. These include ClickMe and Clicktionary datasets with ground truth annotations for human visual importance, and a third dataset collected by the authors using a psychophysics experiment involving rapid classification of images. The paper shows that identifying important features peaks as architectures become better at classification, but then start dipping as they get more complex. To address this, the neural harmonizer methodology is proposed. Finally, the paper tests how these features are used by exposing different models to an increasing percentage of human-identified important regions, showing similar trends as the previous experiments.
strengths_and_weaknesses: Strengths:

1. Extremely interesting problem. While accuracy on standardized datasets is a good diagnostic for progress, measuring if these improvements result in better alignment between biological and artificial intelligence is a fundamental goal for a multitude of reasons. This has also been very well motivated in the paper.

2. The writing is clear and easy to follow. The work is placed well into the context of existing works, and is clear in elaborating how this work builds on top of this literature. 

3. Above all, the experimental design is very solid. Controls are sufficient, and claims are well supported with either experimental results or through existing literature. Psychophysics experiments are well designed, and good care has been taken to ensure as fair a comparison between humans and DNNs as possible.

4. The neural harmonizer module is interesting as it increases alignment significantly.


Weaknesses:

1. The code and data are propriety and will NOT be released. The findings of this paper are interesting, but the most important contribution here seems to be an interesting benchmark for future researchers to focus on. Since the code and the data will not be released, it severely diminishes the utility for the research community. 

Minor Weakness:

Some relevant recent literature is missing, and would be good to add to the paper:

A. On minimal images: 

- Srivastava, S., Ben-Yosef, G. and Boix, X., 2019. Minimal images in deep neural networks: Fragile object recognition in natural images. arXiv preprint arXiv:1902.03227.

B. On alignment in humans and machines in existing works:

- Bomatter, P., Zhang, M., Karev, D., Madan, S., Tseng, C. and Kreiman, G., 2021. When pigs fly: Contextual reasoning in synthetic and natural scenes. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 255-264).

questions: 1. On the role of scene context: Human annotations highlighted lesser context areas (Page 5), but it is widely known that human visual recognition is strongly dependent on scene context. Does this suggest that what humans "think" of as visually important, may not essentially be the most important regions? Also may be suggested by work on minimal images, as it is counter-intuitive to humans that a few pixels can cause such large drops in accuracy. Would be good to see this discussed briefly in the limitations.


2.  A broader question is that the evaluation of alignment necessarily requires a gold standard of knowing what humans rely on the first place. How confident are we in the current measured ClickMe/Clicktionary style data? Also, how do the psychophysics data collected here improve upon these? A short discussion on this would be helpful.
limitations: Yes.
review_title: null

Meta-review: metareview: The reviewers have brought up important concerns around the framing of the paper contributions, the presentation and application of the neural harmonizer method, and the use of saliency maps. The authors have addressed some of these concerns in their rebuttal.

However, I think the main contribution of this paper is the evaluation study itself, which examines 85 different neural network architectures and compares the important features for those architectures (via saliency maps) to the features humans consider important, for varying levels of performance.

This topic of human vision and computer vision inductive biases has been studied through e.g. comparisons of shape vs texture bias, but I think the study presented in this paper (Spearman correlation between feature importance maps) is an interesting addition to work in this area and has scope to be built on by the community, helped by the code release.

I do think the authors could have explored additional empirical studies around this main point, e.g. comparing saliency approaches to self-attention for vision transformers, or looking at the effect of architecture scale on salient features and comparison to human salient features. This to me is the main gap of the paper. 

The paper is thus borderline, with an interesting main study contribution, with scope to be built on and a code release, but lacking additional related empirical experiments.
confidence: Less certain
Decision: Accept