Visual Search Asymmetry: Deep Nets and Humans Share Similar Inherent Biases

Abstract: Visual search is a ubiquitous and often challenging daily task, exemplified by looking for the car keys at home or a friend in a crowd. An intriguing property of some classical search tasks is an asymmetry such that finding a target A among distractors B can be easier than finding B among A. To elucidate the mechanisms responsible for asymmetry in visual search, we propose a computational model that takes a target and a search image as inputs and produces a sequence of eye movements until the target is found. The model integrates eccentricity-dependent visual recognition with target-dependent top-down cues. We compared the model against human behavior in six paradigmatic search tasks that show asymmetry in humans. Without prior exposure to the stimuli or task-specific training, the model provides a plausible mechanism for search asymmetry. We hypothesized that the polarity of search asymmetry arises from experience with the natural environment. We tested this hypothesis by training the model on augmented versions of ImageNet where the biases of natural images were either removed or reversed. The polarity of search asymmetry disappeared or was altered depending on the training protocol. This study highlights how classical perceptual properties can emerge in neural network models, without the need for task-specific training, but rather as a consequence of the statistical properties of the developmental diet fed to the model. All source code and data are publicly available at https://github.com/kreimanlab/VisualSearchAsymmetry.


Reviews and author responses:

 We thank the reviewer for the feedback! Yes, we will revise the final version according to what we promised in the rebuttal period.

This paper presents a paradigm in which a deep neural network searches for a target image. This is accomplished by 2 streams, one of which represents the search object and the other representing the complete visual input. A winner-take-all mechanism along with inhibition of return is performed to elicit a search, and this is compared with human visual performance insofar as different stimuli give rise to asymmetries in difficulty of the search task.  Allow me to preface the comments that follow by a general statement that I found this work to be interesting and generally well done. However, I also have a number of concerns and these are reflected in the statements that follow:


1. References for visual search asymmetries are given in [10][23-25]. In my view these are inadequate. These should include at least references to:

Treisman, Anne, and Stephen Gormican. "Feature analysis in early vision: evidence from search asymmetries." Psychological review 95.1 (1988): 15.

Carrasco, Marisa, et al. "Feature asymmetries in visual search: Effects of display duration, target eccentricity, orientation and spatial frequency." Vision research 38.3 (1998): 347-374.

Rosenholtz, R. (2001). Search asymmetries? What search asymmetries?. Perception & Psychophysics, 63(3), 476-489.

Yamani, Y., & McCarley, J. S. (2010). Visual search asymmetries within color-coded and intensity-coded displays. Journal of Experimental Psychology: Applied, 16(2), 124.

Dosher, B. A., Han, S., & Lu, Z. L. (2004). Parallel processing in visual search asymmetry. Journal of Experimental Psychology: Human Perception and Performance, 30(1), 3.


2. Given the central hypothesis of asymmetries as a consequence of statistical properties, one might also consider:

Rosenholtz, R., Huang, J., Raj, A., Balas, B. J., & Ilie, L. (2012). A summary statistic representation in peripheral vision explains visual search. Journal of vision, 12(4), 14-14.

Shen, J., & Reingold, E. M. Visual search asymmetry: The influence of stimulus familiarity and low-level features.  Perception & Psychophysics.

Visual representation determines search difficulty: explaining visual search asymmetries
Neil D. B. Bruce and John K. Tsotsos, Front. Comput. Neurosci., 13 July 2011

Statistical templates for visual search , John F. Ackermann; Michael S. Landy, Journal of Vision. 2014; 14(3):18.


3. Eccentricity dependent layers are a crucial contribution in this type of study and it is nice to see these included.

4. In the introduction, the paper mentions that "When eccNET was trained from scratch on an augmented ImageNet with altered stimulus statistics, the polarity of search asymmetry disappeared or was modified, demonstrating that inductive biases in the developmental training set can govern complex visual behaviors." - without knowledge at this point of the "altered stimulus statistics", this statement stands out. If there were a summary indication of how they are altered in a general sense this would improve readability.

5. The 6 visual search paradigms considered are reasonable, albeit, I wonder why some alternatives in the literature mentioned above were not considered - e.g. chroma, novelty etc.

6. The overall model is sensible and it is nice to see this type of work as an alternative to traditional deep learning models for computer vision.

7. A winner-take-all mechanism is reasonable given the representation it works on, however, it might be interesting to consider alternatives that take into account spatial distribution of saccades etc., or e.g. an approach that is optimal in reducing uncertainty about the target (akin to Najemnik, J., & Geisler, W. S. (2005). Optimal eye movement strategies in visual search. Nature, 434(7031), 387-391.)

8. The eccentricity mechanism seems more explicit than implicit. What I mean by this is that the loss of resolution at early layers should propagate upward by virtue of connectivity and yield highly blurred, but nevertheless rich featured representations at the top. In this work the eccentricity seems to be forced on the model not by the sensor (the eye), but by construction of the network.

9. While the model shows general agreement with conditions that give rise to asymmetries, this is more in a rough qualitative sense than a strong quantitative sense. This is not a deal breaker, but it would be nice to see how conditions of training impact these curves. Moreover, experiment E is unconvincing. The human data for this case also looks counterintuitive to me - but I could be wrong.

10. Figure 5 reveals that the model outperforms alternatives, but some of these seem like a strawman having no real strong mechanism for producing asymmetries by construction. (or rather, not relying on features that are driven by image statistics in contrast to models that do)

11. The training of the model on MNIST, and focused on experiment 2 provides only weak evidence that the underlying representation learned by the model is what gives rise to any asymmetries observed (although from a personal point of view, that this assertion is true may not need further proof)
 I see no significant limitations or potential negative societal impact.

 The authors have gone to great length to indicate that they will include a wide array of missing references including those that are specific to a statistical basis for asymmetries that are observed.

I am satisfied with the discussion of eccentricity in the model. I also appreciate the notion that it is possible to train the model in a manner where the one case that is inconsistent shows up as consistent, but this may be counterproductive overall to the assessment of the extent to which deep learning models elicit similar asymmetries as humans.

Given all the promises made in the detailed reply, I am inclined to trust that the authors will incorporate these into the paper which at face value is ok, but was lacking in some respects.

With all of that said, I think the bottom line is that this paper can be viewed as a modern take on some of the earlier studies that focused on edge filters and at most V2 style neurons in their role as statistical drivers for observed asymmetries. I would not object to the acceptance of this paper, subject to the revisions that the authors have proposed in their detailed feedback.

 **This is the 2nd part of our response**

2.5. **[Optimal eye movements]** 

As the reviewer points out, there are other components of visual search that are not incorporated in the model proposed here. We appreciate the reference to Najemnik and Geisler 2005, which we will cite and discuss. One point worth mentioning is that the elegant work in Najemnik and Geisler does not constitute an image-computable model. That is, we cannot provide an arbitrary target and search images and produce search times. However, one aspect that is particularly interesting about Najemnik and Geisler is the use of an ideal searcher and the demonstration of optimality. It will be especially interesting to connect those ideas with the type of neural network model proposed here. For example, how can the model perform efficient use of each fixation based on the scene statistics? And does the model also show optimality? If not, what would be required of the model to show optimal eye movements as humans do? We do not know the answers to these questions and these are important points that warrant further work.

2.6. **[Eccentricity]** 

Based on neuroanatomical connectivity and neurophysiological studies characterizing  receptive fields, the two points made by the reviewer are not mutually exclusive. That is, there is eccentricity-dependence in the retina and this gets propagated throughout the visual cortex. In addition, there is further eccentricity-dependent pooling in cortex as well, accentuating the effects propagated from the retina. These effects are especially well understood in V1 through the so-called cortical magnification factor, but there are strong hints that this takes place throughout the visual hierarchy (e.g., see classical studies by Daniel and Whitteridge 1961,   Cowey and Rolls 1974, Born, Trott and Hartman 2015, VanEssen et al 1983, Tootell et al 1988). 

Based on a purely retinal effect, we would expect that the slope of the receptive field size versus eccentricity plots (Fig. 3B) should be approximately parallel. Instead, the experimental data show an *increased slope* from one area to the next. This observation is summarized in (Freeman, Jeremy, and Eero P. Simoncelli. "Metamers of the ventral stream." Nature neuroscience 14.9 (2011): 1195-1201.), a figure that we reproduce in Fig. 3B, left.


2.7. **[Qualitative versus quantitative fits]** 

We agree with the reviewer and emphasized in the text that we focused on the qualitative aspects of comparing model versus humans. It is important to note that *we did not set attempt to quantitatively fit human behavioral data*. In fact, we deliberately avoided tuning parameters in the model to fit the human data. Our main goal was to evaluate whether neural network models of the flavor proposed here could even show any asymmetric behavior during visual search. 

It is important to note that there are multiple aspects of human search behavior that are *not* accounted for by the model. The model produces eye movements, whereas humans had to press a key to indicate target found. The model had to precisely locate the target whereas humans only had to indicate target present or absent. Humans were exposed to target-present and target-absent trials whereas the model only dealt with target-present trials. Humans could change their strategies during the course of an experiment (due to learning, attentional changes, and other non-stationarities), but the model was fixed. And of course, the model might be completely wrong after all. All of these factors can contribute to quantitative differences.

That said, we have also fine-tuned the model to better capture human behavior, as is customarily done in many computer vision applications. As one might expect, fine-tuning the weights in the model can dramatically improve the quantitative fits. We will include those results in the revision. However, we still feel that as a proof-of-principle demonstration, noting that the model can reveal search asymmetry is more important than quantitative fit of reaction times.

The reviewer also mentions that the psychophysics results in experiment E (experiment 5, orientation) seem counterintuitive to him/her. Presumably, the reviewer intuits that it should be easier to find vertical among titled than tilted among vertical. In fact, the human asymmetry goes the other way. Tilted among vertical is easier to find. This has been replicated many times (e.g. Treisman and Gormican, 1988, Carrasco et al, 1998). The specific human data shown in (E) come fromWolfe et al 1992). Treisman used this finding (among others) to argue that the deviation from the standard was easier to find than the standard. So, for her, vertical was the standard and tilt the deviation. Wolfe argued that, as elsewhere, it is easier to find the presence of a feature than its absence. He argued that “tilt" was the feature and vertical is just an item without “tilt”.

The model produces an asymmetry, but it is in the wrong direction. What is counterintuitive to us is the output of the model for this experiment. We have built an alternative model where we allow the parameters to be fitted for each experimental condition where we can reproduce the direction observed in the human experiments. However, we prefer to show the same exact model across all experiments because: (i) this avoids any parameter tuning; (ii) it emphasizes that different forms of asymmetry may require different explanations (see question 2.4), and (iii) it represents a clear failure of the model.

2.8. **[Alternative models]** 

We agree that some of the alternatives are strawman hypothesis (e.g., random eye movements). However, we still find it instructive to present these alternatives for comparison purposes and to provide a calibration of what one may expect with such simple alternatives. It is fair to argue that random eye movements will not show search asymmetry by construction, as the reviewer points out. 

However, we note that some of the alternative models *do include* image statistics. For example, the IVSN model uses a VGG16 architecture pretrained on Imagenet. We are not convinced that one can trivially argue that IVSN should not show asymmetry *by construction*. In fact, IVSN does show asymmetry *in some cases*. Although the performance of IVSN is less similar to human behavior than the proposed ECCNet (Fig. 5b), IVSN does not seem to be a strawman model, or at the very least, it is much less of a strawman model than random eye movements. 

2.9. **[MNIST]** 

Training on MNIST disrupts search asymmetry. 
We will rephrase this in the revised version to indicate that this is only an initial step to assess the impact of the training set statistics. The experiments where we rotate the images in the training set provide a more compelling and nuanced demonstration of the important role of the training set.

The reviewer's question also inspired us to come up with different training regimes. One of them includes using a fisheye distortion (which reduces the proportion of straight lines and increases the proportion of curves). The other one is to introduce vertical and horizontal lines, thus increasing the proportion of straight lines. We conjecture that these changes to the training set will have opposite effects on asymmetry during visual search. Unfortunately, we have not been able to finish training on these large datasets during the rebuttal period, but we will also include those results in the revised version.

 **This is the 2nd part of our response**

3.3. **[Training from scratch]** 

We agree with the reviewer that this is a valid and interesting experiment to explore and also that it is more ecologically valid.

As a technical point, we note that we are working with relatively large images ($1,024 \times 1,024$ pixels as inputs to EccNet).
We chose to work with these large images to be able to assess the effect of eccentricity.

This requires considerable computational GPU resources and time given that the number of network parameters increases drastically given such large input image sizes. 

Therefore, our claim of ``...can be *easily* integrated...'' does not really hold. 
We will take a three-pronged approach: (i) we will revise the text to emphasize the computational complexity; (ii) we will emphasize that training from scratch is more ecologically relevant; (iii) we will attempt to train these models from scratch using large GPU clusters. 

3.4. **[Minor]** 

Thank you! We will fix the error.

 3.1. **[The target representations in layer l and l+1]** 
 
 The intuition of the reviewer is correct. The choice of the units that receive top-down modulation in layer l cannot be random. 
 
We selected the units in layer l such that the units in layer l+1 correspond to be the max-pooling of those units in layer l, which is common in convolution block designs. With this design, the features at layer l+1 are still consistent with those in layer l except that the features at layer l get up-sampled spatially using the max pooling operation at layer l+1. Mathematically, we will revise Equation 2 as follows to make this clearer:

$A_{l+1 \rightarrow l, n} = m( \phi^t_{l+1} ,\phi^s_{l,n})$ 

where $\phi^t_{l+1} = maxpool ( \phi^t_{l})$, 
$A$ refers to the top-down modulation map at the $n$th fixation location, $m(\cdot)$ is the target modulation function defined as a 2D convolution and $\phi$ refers to the extracted feature maps. We will make this clearer in the revised version.

3.2. **[Target representation as a function of fixation locations]** 

The reviewer is correct. $\phi_{l+1, n}^t$ is indeed invariant to the fixation location. We will fix it in the final version. Thanks!


 2.1. **[Missing references I]** 

We thank the reviewer for pointing us to these references. We will cite and discuss these related works in the revised version. Furthermore, it will be interesting to test the model with these other stimuli leading to asymmetries (see more on this below). 

2.2. **[Missing references II]** 

We thank the reviewer again. As the reviewer points out, these studies are particularly relevant and the conclusions are consistent with the notion that natural statistics of images and eccentricity play important roles in visual search asymmetry. We will cite and discuss these studies. Our work makes two additional contributions that complement those earlier studies: (i) we used data augmentation during training and model ablation during testing to explicitly test these hypothesis (it is hard to change the training statistics for humans or do ablations in human cognition!); (ii) we showed that deep neural networks could show similar inherent asymmetric biases as humans.

2.3. **[Introduction readability]** 

We will clarify in the introduction what we mean by ``altered stimulus statistics''. We expand upon this point in Section 4.4, but we agree that it is important to explain this better in the introduction.

2.4. **[Other asymmetries]** 

This is also an excellent point. We have indeed tested other forms of asymmetry: (i)
searching for prototype colored circle (e.g. red) among deviations colored circles (e.g. magenta) (Treisman et al. Psychological review, 1988) and vice versa; and
(ii) searching for "live" among "dead" elephants (and vice versa) (Wolfe, Perception and Psychophysics 2001).

We could not test additional experiments in the short amount of time allocated for rebuttals, but we appreciate the suggestions from the reviewer and we will test several other forms of asymmetry described in those studies, as suggested. We also note that we are making all the code available and we hope that this will encourage others in the community to test other stimuli as well. 

The point of the paper is a proof-of-principle demonstration of how a computational model can reveal visual search asymmetry similar to humans and thus provide a plausible mechanism for asymmetric behavior. 

There are a few interesting points indirectly alluded to in the reviewer's question. We would like to comment on them briefly here. 

(i) Is the mechanism for *all* forms of search asymmetry the same? We suspect that the answer is no. Thus, it is especially interesting to study other forms of asymmetry as the reviewer suggests. We certainly do *not* want to claim that *all* search asymmetries have the same cause. 

(ii) In addition, we also do *not* claim that our model produces all of the asymmetries seen in human psychophysics studies.

(iii) What constitutes adequate evidence in support of a model of search asymmetry? Given the assumption in (i), there may *not* be a single explanation or model for every form of asymmetry. While the accumulation of supporting evidence in the form of multiple different examples is instructive, the work here does *not* constitute a mathematical proof that the model can extrapolate to other forms of asymmetry, nor do we make such a claim. 

In addition to testing other forms of asymmetry as the reviewer suggested, we will also clarify these points in the revised version.

 1.1. **[Single domain leads to anecdotal support]**

We agree with the reviewer and this is an excellent point. We used the example of lighting direction as a proof-of-principle to illustrate how the training dataset can impact search asymmetry. As the reviewer suggested, we have expanded this notion to the other experiments and we will incorporate these results in the revised manuscript. Briefly, we introduce other distortions to the training set statistics, we retrain the model and we assess the impact on visual search asymmetry (similar to the experiment in Figure 6 with lighting direction). 

In the case of curves versus lines (Experiment 1), we independently introduced two changes to the training dataset: (i) fisheye distortion, which reduces the proportion of straight lines and enhances curvatures, and (ii) introduction of horizontal and vertical grids to the images, which increases the proportion of straight lines. In the case of Experiments 5-6, we introduce lines that enhance specific orientations. 
These manipulations require retraining the model and we could not finish these new calculations during the rebuttal period. However, we will include those results in the revised version. 


The authors address a fascinating phenomenon of asymmetries in visual search. Having been observed over half a century ago it has eluded many attempts at explanation.
The authors argue that such asymmetries are induced by biases in statistics of the learning experience (or the training data set). To test this hypothesis they designed a neural network architecture, eccNET, and trained on Imaginet dataset of natural images and on an augmented Imagenet that compensates for the said biases. Extensive computational experiment results support their hypothesis, and ablation studies support the importance of the eccentricity-dependent pooling.  This elegant paper addresses a much studied problem of asymmetries in visual search. The proposed eccNET succinctly captures visual search mechanisms thought to operate in primate brain, and the presented empirical study of "reaction times" corroborate the plausibility of the approach. 
The most interesting and impactful claim of the paper, that search asymmetries are introduced by biases in training datasets, is addressed by manipulating Imagenet dataset. A 90 deg rotation of the dataset reverses the search time pattern for illumination-dependent features, in agreement of the hypothesis. 
 Given the importance of the hypothesis that asymmetries are caused by biases in training datasets, it would be desirable to see results of manipulating the training datasets to affect the remaining 5 search experiments (Fig 1).  A demonstration in a single domain lends the hypothesis only anecdotal support.

Suggestions:
Extend training data manipulations to address sources of asymmetry for other 5 types of features that exhibit search  asymmetry.

Societal impact: 
1) paving the way for more efficient artificial vision systems can have sizeable impact on the many industries that are using CV;
2) better understanding of primate (including human) vision system may lead to better cures for vision pathologies.


The paper proposes a neural network model of target-based visual search. The network is a VGG-16 with two important modifications: eccentricity-dependent pooling and multi-layer target matching that guides the simulated fixations. The model qualitatively reproduces five classical findings of visual search asymmetry without task-driven parameter tuning (but fails to reproduce a sixth effect and does not quantitatively capture the human visual search slopes).  Overall, this is a solid contribution that further bridges the gap between neural network modeling and cognitive psychology. In particular, the effect of training on rotated ImageNet is quite striking. It demonstrates how neural network modeling can test evolutionary or developmental hypotheses that are otherwise extremely difficult to assess.

Major comments:
1. My main concern is equation (2). It requires considerable clarification. The target representation in layer $l+1$ is defined using a different set of channels than the search representation in layer $l$. How can $\phi^t_{l+1,n}$ serve as a valid and meaningful convolutional kernel for $\phi^s_{l,n}$? Intuitively, I would expect that $\phi^t_{l,n}$ would be used to define the attentional gain of features in $\phi^s_{l,n}$ (so the channel representation is consistent). Since the model seems to work well, I am assuming I misunderstood. This point requires both a verbal explanation and greater mathematical explicitness.

2. Why does the target representation $\phi^t_{l+1,n}$ take the fixation location as an argument? As a template used for spatial convolution, I would expect it to be invariant to the fixation location.

3. Training the network with the modified pooling layers on ImageNet from scratch is more ecologically valid than performing "network surgery" (i.e., replacing the pooling layers in the already trained model). Furthermore, such a trained-from-scratch model would be a better control for the models trained from scratch on MNIST and rotated ImageNet. Last, training from scratch will also support the claim that "these customized eccentricity-dependent pooling layers can be easily integrated into other object recognition deep neural networks".

Minor:
Line 105: VGG16 "traditionally" uses max pooling and not average pooling (Simonyan & Zisserman, 2014).

Reference:
Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556. Limitations are adequately discussed. There is no apparent negative societal impact.

Meta-review: This paper describes a new computational model of search asymmetry. The paper received two positive reviews and initially one negative review but the rebuttal satisfied this reviewer who increased their score to a marginal accept. All reviewers commented on the many strengths of this paper. Most of the limitations brought up by the reviewers were addressed in the rebuttal and the remaining ones should be stated in the discussion.

Decision: accept