Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper proposes a computational model (eccNET) that produces visual search asymmetry patterns similar to those seen in humans. The model is pretrained on ImageNet and integrates eccentricity and top-down cues to produce a series of fixations until the target is found. Since a rotation of the training set changes the pattern of results, the authors conclude that visual search asymmetry may be a consequence of natural image statistics.
main_review: The goals of this paper appear to be twofold: first, to understand whether visual search asymmetry of the type seen in psychophysics experiments emerges in neural networks not specifically trained in visual search; and second, to understand the mechanisms through which this happens by manipulating the model architecture and training set. 

The paper takes a thorough approach both in characterizing human behavior (using data from six experiments) and machine behavior (four baselines, as well as model ablations to assess the role of model features in matching human behavior). I found that many questions that arose while reading the paper had been addressed by the end. The paper is well-written and well-structured.

The authors used data from six psychophysical experiments with different visual search tasks showing asymmetry in humans. Since these experiments relied on target-present/target-absent keypress responses, the authors collected an additional eyetracking dataset to aid in linking fixation data from eccNet to RT data in humans. Limitations of this approach are discussed in the paper; my only comment is that it would be interesting to see how the fixation data compares to eccNet in this one experiment (despite all the expected differences that the authors discuss).

In terms of pinpointing hypotheses for why this asymmetry emerges in eccNet, the paper seems to hint at a conclusion. First, it appears that biologically plausible model features are crucial (eccentricity and top-down feedback at several layers have a larger impact on asymmetry than training eccNet with MNIST), which seems to be less emphasized in the discussion than the role of the training set. Second, not all mechanisms are explored - some visual search effects are not reversed after training set rotation, suggesting some may arise from different visual features. 

The fact that the 90-degree rotation of the training set leads to an effect polarity reversal does appear to point at statistical regularities in the training set, which is in line with previous research suggesting e.g. familiarity as a potential factor in visual search asymmetry. However, the authors also mention potential limitations of ImageNet. I am curious as to whether effects from the other experiments could be disrupted through training set alterations, and what explains experiment E (it'd be interesting to see what would happen with a different training set, e.g. Places)

Minor points:

In Figure 4 there seem to be no error bars in the human data plots? Figure 5 could also have error bars (e.g. SEM across experiments). The number of subjects who took part in the psychophysics experiments doesn’t seem to be mentioned even in the appendix.

---------------------------------------

After response:

Thank you for addressing my points so thoroughly. Great to see a direct fixation comparison, as well as the new training regimes - look forward to seeing the updated paper!
limitations_and_societal_impact: The authors have addressed the limitations of the work.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: In this manuscript, the authors present a visual search model, which predicts a fixation order over visual search displays. The model is based on VGG-16 replacing the max-pooling layers with average pooling with growing pooling area towards the periphery. The priority for fixation is then computed as a combination of target-image similarity at three different feature maps in the network. The primary result reported is that model qualitatively reproduces 5 out of 6 visual search asymmetries. 
main_review: Overall, the observation that visual search asymmetry might be caused by peripheral pooling is interesting and the model the authors use seems reasonable. I have a relatively long list of questions though and in quantitative terms the fit of the model is bad. Thus, I do not find the model and manuscript convincing yet.

My first questions are on Figure 3: Why didn’t the authors choose the pooling areas such that the eccentricity dependence follows the ones of the visual areas they compare to? Also, are there any reasons why Layer 10 should be associated with V1 etc? 
Also, the illustration in A appears to be wrong as the center is not blurred at all, while the formulas give a 2 pixel radius pooling for the fovea at each location, which should correspond to noticeable blur in the center.

I find the pooling operation across attention maps strange. Why are the 3 A maps scaled as they are? The overall scaling amounts to max(A) / (max(A) - min(A)), i.e. a stronger weighting the larger the minimal value is in proportion to the maximum. How does this make any sense?

Also I find the nomenclature of calling the maps which measure target-image similarity at the different layers (A_{i \rightarrow j}) very odd as it strongly implies a modulation of the processing within the network producing the activations. If I understand the manuscript and the illustration in Figure 2 right, this does not happen though, but the convolution is only applied to create the readout maps.

On the pixelMatching model: As all search images with the target contain an exact copy of the target: Why doesn’t the pixel matching model find all targets within the first fixation it chooses?

As the model produces whole scan paths: Have the authors considered looking at more detailed aspects of them rather than just the overall duration?


----

After comments:
I have just slightly increased my evaluation of this manuscript as the authors have addressed some of my concerns. Especially they now do some comparisons on the scan-paths and show some better quantitative fits to the reaction time data.
Also reading some of the other reviews highlighted some of the advantages of this manuscript.

Unfortunately, the authors' answers didn't remove my concerns about the model though. Some choices still appear strange/unusual or not exactly representative for the described mechanisms to me. For me this casts doubt on any of the interpretations, which of course rest on the assumption that the model is a good representation of the theoretical ideas.
It is entirely possible, that my concerns can be addressed and this manuscript ends up being correct, but I honestly see some probability that the results do depend on some oddity of the model and are not interpretable as they are interpreted here. 

Thus, I won't improve my rating further here, but suggest some clarification and maybe some controls as suggested by other reviewers may help for the future.

To be concrete:
- on 2.2 / 2.3.: I still don't understand why the pooling regions were not exactly matched to the monkey data. Scaling pixels and the slope should allow any linear function and thus a perfect fit between the two figures. Little of the conclusions in this manuscript rests on the question which area corresponds to which layer, but not making them fit where you clearly could is strange.
- on 2.5: I did understand the scaling of the attention maps. My criticism is that the described scaling does not appear sensible to me. As you weigh maps by their maxima after scaling them by their range the difference in scale is restored and each map is effectively scaled by how much of its range is above 0, which seems highly unusual to me.
- on 2.6: I originally formulated my criticism here aimed at changing the formulation, thinking that the authors "obviously" cannot mean to interpret this interaction as top-down modulation. It seems that they do mean to interpret this as an implementation of top down modulation. I strongly disagree with this interpretation.   
The two interacting layers are a feature map and a blurred version of the same feature map (or RELU transformed, I am not perfectly sure due the confusing layer numbering) . Thus, the operation is a template match with a slightly blurred representation of the target (slightly, because the target image is processed at fixation if I am not mistaken). Template matching is a typical strategy used in search models, but has no connection to top-down modulation for me. If there is any top-down connection it is from working memory to the feature map not between layers of processing.

Note on the last point: If the two layers were not using closely related feature maps, convolving the feature map with a target representation in the higher layer would not make any sense and would in many cases not be possible due to a mismatch in the number of features. 

By the way: Something appears to be off about the counting of layers the authors use. They write that layer 10 corresponds to the 6th convolutional layer, but I checked and VGGs use RELUs after each convolution, such that it could be at most the 5th and in fact guessing based on the figure and a look at the pytorch hub VGG could be either the second max-pooling layer, which appears after 4 convolutions or the third, which happens after 7 convolutions though and has a layer index of 16 in pytorch. 

limitations_and_societal_impact: The authors do not comment on their limitations and societal impact. Besides the sampling biases of the used databases I don’t think there are important societal impacts associated with the content of this paper though.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposed an image-computable model to examine potential mechanisms underlying asymmetries in visual search. The proposed method can capture similar behaviour as the human user, and the proposed architecture is novel. However, I am not fully convinced why we need to investigate the asymmetries with the neural networks model, and I am not sure how much of the training data instead of the proposed method can affect the results.
main_review: **Paper strength**
- The paper is well-written. The author provides sufficient details of the implementation of the proposed method, and a good illustration of the methods as well as the experiment results. 

- The proposed method has a certain novelty. The eccentricity-dependent pooling is meaningful for such an attention-based model, and the author extracts the feature according to the number of the layer. The author also proposes a top-down attention modulation. Both two components are evaluated with the ablation study to show their effectiveness (Sec. 4.3).

- The experiment (Fig. 4) shows the proposed method successfully capture the visual search asymmetry of the human user. It is an interesting finding that the pre-trained model on ImageNet has such ability. In comparison with other baselines in Fig. 5, the proposed method shows it achieve leading performance and similar to the performance of the human user. 

- The author provides interesting discussions on where the asymmetry comes from in Sec. 4.4. The author thinks the asymmetry comes from the lighting direction within the natural images.

**Paper weakness**
- It is not clear to me why we want the trained model to have similar behaviour as the human. Especially, why we want the model to capture such asymmetry as the human being? I understand that the current machine and human have their own advantage respectively, and it is not necessary that the machine must learn from the human. 

- One experiment shows such asymmetry also depends on the training data. The model trained on the MNIST dataset does not have an asymmetry effect. It shows that such asymmetry actually depends on the data instead of the method/architecture. If we want to investigate the asymmetry, we should perform the analysis on the data distributions. 

- It will be very interesting if the author found the asymmetry is a general effect for neural network models. However, it is not the case as shown in Fig. 5. The asymmetry can be captured well with the proposed method, not other neural networks models. In this sense, it is not clear to me why the author wants to build such a model.

-The assumption that the asymmetry comes from the lighting condition is not convincing. It might be true for the outdoor environments but not for the indoor setting. It will be better if the author could provide any static numbers to support their argument. Such an assumption also cannot explain other shapes such as the curve.

- I would not agree that the classification result of 66.7% is similar to 72% (line 267).
limitations_and_societal_impact: The author provides a detailed discussion on limitations. I do not think there is nay negative societal impact of this work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors proposed a computational model to illuminate the mechanism underlying the search asymmetry phenomenon (finding a target A among distractors B can be easier than vice versa). Their model takes a search image and a target as inputs and produces a sequence of eye movements until the target is found. The model integrates eccentricity-dependent visual recognition with target-dependent top-down modulations. Without prior exposure to the stimuli or task-specific training, the model reproduces the polarity of search asymmetry in human vision. They also showed that when the statistics change in the dataset, the search asymmetry either disappeared or was altered.

main_review: Whether the contribution is original depends on the motivation of the work and the interpretation of the results. Particularly, if the authors aim at elucidating the mechanism underlying visual search asymmetry (as stated in the abstract), then the conclusion is not very new. Visual search asymmetry has been long studied in neuroscience. Various models have been proposed to explain search asymmetry. For instance, Heinke and Backhous 2011 modeled visual search with selective attention to explain search asymmetry and showed that search asymmetry is the outcome of interactions between top-down and bottom-up processing. Various factors have been identified as sources of search asymmetry. For instance, Bruce and Tsotsos (2011) argued that stimulus familiarity determines search difficulty and may help explain search asymmetry. If the work intends to address the long-standing question of the mechanisms underlying visual search asymmetry, the authors should mention the related literature in vision sciences, including the existing theories/computational models of visual search.

By contrast, if the authors aim at showing "Deep Nets and Humans Share Similar Inherent Biases" as the title states, then the conclusion is kind of new. But the authors should mention other papers that demonstrate similar inherent bias between deep nets and humans, such as "Cognitive Psychology for Deep Neural Networks: A Shape Bias Case Study" by Ritter et al., 2017. "Bias and Generalization in Deep Generative Models: An Empirical Study" by Zhao et al. 2018 or "Number detectors spontaneously emerge in a deep neural network designed for visual object recognition" by Nasr et al. 2019.

The submission is relatively sound. The authors conducted psychophysics experiments to compare the search asymmetry in the deep nets and the ones in human subjects. They also conducted thorough ablation studies to confirm the necessity of the model components to reproduce the asymmetry. I have three major concerns. First, although the author collected the eye movement data, they only compare the reaction time and number of fixations with the model prediction. I was wondering why they did not compare the eye movement trajectory with the predicted trajectory directly. If there's a huge discrepancy between the model predicted trajectory and the human behaviors, the authors should provide some explanations.

Second, the authors claim that visual search is "the polarity of search asymmetry arises from experience with the natural environment.. is a consequence of the statistical properties of the developmental diet fed to the model." And there are only two experiments supporting that claim, which is 1) replacing the naturalistic dataset ImageNet with an MNIST and 2) rotating ImageNet images by 90 degrees. These are just some special cases of how statistics in the training set can be altered. In order to draw a strong conclusion as what the authors claimed, a much comprehensive manipulation of different statistics of the training set should be done, including contrast, spatial frequency, brightness, orientation, etc. It's entirely possible that changing some statistics won't alter the search asymmetry, contrary to what the authors found using the limited set of two experiments. In a word, I think the claim that when statistics in the training set change, the polarity of the asymmetry would be altered is a bit overgeneralizing. A minor issue related to this is that rotating 90 degrees not only changes the lighting from directions but also many other things (e.g., familiar objects such as trees and dogs would appear unfamiliar), which makes the authors' argument less convincing.
 
Finally, although it's nice to see that without any task-specific training, search asymmetry naturally emerges from the network, the author did not give any intuition on why adding eccentricity-dependent visual recognition with target-dependent top-down cues enables that to happen. If the author could connect that to classic theories on the human visual search to provide some intuition on why those model components give rise to search asymmetry, that could make the paper more informative and inspiring than simply showing the model works.

The submission is clearly written and well organized. If the authors could make their reasoning more solid, then the results can be important in the sense that it reveals that deep neural networks share similar inherent biases with humans when it is equipped with eccentricity-dependent visual recognition with target-dependent top-down cues. This would be useful to build AI systems that can simulate human visual search behaviors, which is extremely useful in fields like user-interface design.

------------------------------------------

After comments: I've read the authors' responses to my review as well as the ones to the other reviewers'. I think their responses address some of my concerns, especially the intuition about eccentricity and top-down modulation in asymmetry. Given these improvements, I decided to raise my score to 7.
limitations_and_societal_impact:  There's no foreseeable negative societal impact of this work.
review_title: null

Meta-review: The paper investigates visual search asymmetry using a computational model, which integrate both bottom-up visual recognition with target-dependent top-down cues. By comparing the model against human behavior in visual tasks, the work revealed classical perceptual properties can emerge in neural network models pretrained on ImageNet, without the need for task-specific training. The findings are scientific and useful for providing computational evidence for human cognition. All the reviewers think the paper is well-written and provides a thorough view about human and machine behavior synergy in visual search. The reviewers give extensive, detailed reviews about the manuscript and made numerous points for clarification and improvements. The authors did a good job in the rebuttal that strengthens the reviewers confidence, which led to the increase of scores. These responses should be included in the revision. There are still a few points that deserve further clarification. Reviewer CMyL is not convinced about a few model choices as detailed in the post-rebuttal review. Both Reviewer F9R3 and CJM4 want further clarification about the motivation of the work. The authors should address these points in the revision.
Decision: Accept (Poster)