Visual Search Asymmetry: Deep Nets and Humans Share Similar Inherent Biases
Authors: Shashi Kant Gupta
Mengmi Zhang
Chia-Chien Wu
Jeremy M. Wolfe
Gabriel Kreiman
Venue: NeurIPS 2021 Poster (2021)
Keywords: visual search
asymmetry
deep learning
recognition
neuroscience
eye movement
visual cortex
psychophyiscs
human behavior
visual cognition
inductive bias
natural statistics

Abstract: Visual search is a ubiquitous and often challenging daily task, exemplified by looking for the car keys at home or a friend in a crowd. An intriguing property of some classical search tasks is an asymmetry such that finding a target A among distractors B can be easier than finding B among A. To elucidate the mechanisms responsible for asymmetry in visual search, we propose a computational model that takes a target and a search image as inputs and produces a sequence of eye movements until the target is found. The model integrates eccentricity-dependent visual recognition with target-dependent top-down cues. We compared the model against human behavior in six paradigmatic search tasks that show asymmetry in humans. Without prior exposure to the stimuli or task-specific training, the model provides a plausible mechanism for search asymmetry. We hypothesized that the polarity of search asymmetry arises from experience with the natural environment. We tested this hypothesis by training the model on augmented versions of ImageNet where the biases of natural images were either removed or reversed. The polarity of search asymmetry disappeared or was altered depending on the training protocol. This study highlights how classical perceptual properties can emerge in neural network models, without the need for task-specific training, but rather as a consequence of the statistical properties of the developmental diet fed to the model. All source code and data are publicly available at https://github.com/kreimanlab/VisualSearchAsymmetry.

Visual Search Asymmetry: Deep Nets and Humans Share Similar Inherent Biases
Shashi Kant Gupta', Mengmi Zhang Chia-Chien Wu Jeremy M. Wolfe", and Gabriel Kreiman 2.3
Indian Institute of Technology Kanpur, India Children's Hospital, Harvard Medical School Centte for Brains, Minds and Machines
A Brigham and Women's 's Hospital, Harvard Medical School Address correspondence to gabriel.kreiman @tthhharvard.cdu
Abstract
Visual search is a ubiquitous and often challenging daily task, exemplified by looking for the car keys home or friend in crowd. An intriguing property of some classical search tasks is an asymmetry such that finding a target A among distractors B be easier than finding B among A. To elucidate the mechanisms responsible for asymmetry in visual search, we propose a computational model that takes a target and search image as inputs and produces a sequence of eye movements until the target is found. The model integrates eccentricity -eeetricityydeeeddeett visual recognition with target-dependent top-down cues. We compared the model against human behavior in six paradigmatic search tasks that show asymmetry humans. Without prior exposure to the stimuli or task-specific training, the model provides a plausible mechanism for search asymmetry. We hypothesized that the polarity of search asymmetry arises from experience with the natural environment. We tested this hypothesis by training the model augmented versions of ImageNet where the biases natural images were either removed or reversed. The polarity of search asymmetry disappeared or was altered depending on the training protocol. This study highlights how classical perceptual properties emerge in neural network models, without the need for task task-specific training, but rather as a consequence diet fed to the model. All source code and data are publicly available at https; //github. com imwnlalaaa/aaaaaaaaaaaaaaaaabaaa wwwpanaaaaiiiilllleeaaaaalllee wuw.lsearchhsymeet...
the statistical properties of the developmental
Introduction
Humans and other primates continuously move their eyes in search of objects, food, friends. Psychophysical studies have documented how visual search depends on the complex interplay between
the target objects, search images, and the subjects' memory and attention 44 29
37
44
[8]
30).
There has also heen progress in describing the neurophysiologica steps involved visual processing
32
39
23
and the neural circuits that orchestrate attention and eye movements 14
38
10
paradigmatic and intriguing effect visual search asymmetry: searching for an object, A, amidst other objects, B, can be substantially easier than searching for object B amongst instances of A. For example, detecting a curved line among straight lines is faster than searching for a straight line among curved lines. Search asymmetry has been observed in wide range of tasks [48] 22 46
50 43
Despite extensive phenomenological characterization 34 13 8 the mechanisms underlying how neural representations guide visual search and lead to search asymmetry remain mysterious.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

51 25 we developed an image- Inspired by prior works on visual search modelling 77 21 19 computable model (eccNET) to shed light on the fundamental inductive biases inherent to neural computations during visual search. The proposed model combines and top-down modulation through target-dependent attention. In contrast to deep convolutional neural networks (CNNs) that assume uniform resolution over all locations, we introduce eccentricity- dependent pooling layers in the visual recognition processor, mimicking the separation between fovea and periphery in the primate visual system. As the task-dependent target information is essential during visual search 14 3 49 5T, the model stores the target features and uses them in a top-down fashion to modulate unit activations in the search image, generating sequence of eye movements until the target is found.
sampling,
We examined six foundational psychophysics experiments showing visual search asymmetry and tested eccNET on the same stimuli. Importantly, the model was pre-trained for object classification on ImageNet and was not trained with the target or search images, or with human visual search data. The model spontaneously revealed visual search asymmetry and qualitatively matched the polarity of human behavior. We tested whether asymmetry arises from the natural statistics seen during object classification tasks. When eccNET was trained from scratch on an augmented ImageNet with altered stimulus statistics. e.g., rotating the images by 90 degrees, the polarity of search asymmetry disappeared or was modified. This demonstrates that, in addition to the model's 's architecture, inductive biases in the developmental training set also govern complex visual behaviors. These observations are consistent with, and build bridges between, psychophysics observations in visual search studies
35 40 7 2 g and analyses of behavioral biases of deep networks 28 53 33
2
Psychophysics Experiments in Visual Search Asymmetry
We studied six visual search asymmetry psychophysics experiments 48 Subjects searched for target intermixed with distractors in search array (see Appendix A for stimulus details). There were target-present and target-absent trials; subjects had to press one of two keys to indicate whether the target was present or not.
22
46
47
(Figure [].
Experiment (Curvature) involved two conditions (Figure IA) [48]: searching for (a) a straight line among curved lines, and (b) curved line among straight lines. The target and distractors were presented in any of the four orientations: -45, 0, 45, and 90 degrees.
Experiment 2 (Lighting Direction) involved two conditions (Figure T BB) [22]: searching for (a) left-right luminance changes among right-left luminance changes, and (b) top-down luminance changes among down-top luminance changes. There were 17 intensity levels.
Experiments 3-4 (Intersection) involved four conditions (Figure IC-D) [4] searching for (a)
cross among non-crosses, (b) a non-cross among crosses, (c) rotated L among rotated Ts, and (d) rotated T among rorated Ls. The objects were presented any of the four orientations: 0, 90, 180, and 270 degrees.
searching for (a) vertical line among 20-degrees-tileer lines, (b) a 20-degree-tilees tilte line among vertical straight lines, (c) 20-degree tilted line among tilted lines from -80 to 80 degrees, and (d) vertical straight line among tilted lines from -80 to 80 degrees.
Experiments 5-6 (Orientation) involved four conditions (Figure TE-F) 477:
3 Eccentricit-deeendeent network (eccNET) model
schematic of the proposed visual search model is shown in Figure eccNET takes two inputs: a target image (4e, object to search) and search image (4,, where the target object is embedded amidst distractors). eccNET starts fixating on the center of 4, and produces a sequence of fixations. eccNET uses pre-trained 2D-CNN as proxy for the ventral visual cortex, to extract eccentricityddependee visual features from I and I,. At each fixation n, these features are used to calculate top-down attention map (A,,). A winner-take-all mechanism selects the maximum of the attention map A, as the location for the n + 1-th fixation. This process iterates until eccNET finds the target with a total of N fixations. eccNET has infinite inhibition of return and therefore does not revisit previous locations. Humans do not have perfect memory and re-visit previously fixated locations (52). However, in the 6 experiments considered here N is small; therefore, the probability of return fixations is small

---

Curvature (Experiment 1)
Lighting direction (Experiment 2)
Line among Curves
Curve among Lines
Horizontal Luminance
Vertical Luminance
3
3
1'
3
C. Intersection (cross non-cross, Experiment 3) Cross among no Intersection, Intersection among Cross
D. Intersection (L vs T, Experiment
L Lamong Ts
T among Ls
3"
E. Orientation (homogeneous. Experiment 5)
F. Orientation (heterogeneous, Experiment 6)
O: D: 20
20
T:200 D:0. 40
T:0 D1+20.10060040
/
/
3'
T
3
3'
Figure Schematic illustration of the experiments for asymmetry in visual search. Each experiment has two conditions; in each condition, an example search image (left) and target image (right) are shown. The target image is smaller than the search image (see scale bars below each image). A. Experiment Searching for curve among lines and vice versa ((48). B. Experiment Searching for vertical luminance changes among horizontal luminance changes and vice versa ((22)). C. Experiment 3. Searching for shapes with no intersection among crosses and vice versa Searching for rotated Ts among rotated Ls and vice versa ((46)). E. Experiment 5. Searching for oblique lines with fixed angles among vertical lines and vice versa (. F. Experiment Similar Experiment 5 but using oblique lines of different orientations
). D. Experiment
[47
In all cases, subjects find the target faster in the condition on the right.
and the effect of limited working memory capacity negligible. eccNET needs verify whether the target is present at each fixation. Since we focus here on visual search (target localization instead of recognition), we simplify the problem by bypassing the target verification step and using an "oracle" recognition system (26). The oracle checks whether the selected fixation falls within the ground truth target location, defined as bounding box of the target object. We only consider target-present trials for the model and therefore eccNET will always find the target.
Compared with the invariant visual search network (IVSN, [T highlight novel components in eccNET, which we show to be critical in Section
have uniform receptive field sizes (pooling window sizes) within each layer. In stark contrast, visual cortex shows strong eccentricity- dependent receptive field sizes. Here we introduced eccentricity-depente replacing all the max-pooling layers in VGG16.
Standard 2D-CNNs, such as VGG16
pooling layers,
2. Visual search models compute an attention map to decide where to fixate next. The attention map in IVSN did not change from one fixation the next. Because the visual cortex module manner, here the
in eccNET changes with the fixation location in an attentional maps (A,) are updated at each fixation n.
3. In contrast IVSN where the top-down modulation happens only in single layer, eccNET combines top-down modulated features across multiple layers.
Receptive field sizes in visual cortex increase from one brain area to the next (Figure3B. right). This increase is captured by current visual recognition models through pooling operations. addition, receptive field sizes also increase with eccentricity within given visual area [16] (Figure 3 right). Current 2D-CNNs assume uniform sampling within a layer and do not reflect this eccentricity dependence. contrast, we introduced eccentricity deentricity-dependen pooling layers in eCcNET. Several psychophysics observations g 35 and
Recentricity-deeendeer pooling visual cortex in eccNET.

---

unit
RF (r)
Target image (1)
12
10
2'
8
Top-downmodulan modulation
6
Search image (1.)
4
2
Eccantriily-dapencent pooling
6'
Overall attention map
1.0
An, 10-9
An.14-13
18.18-17
C.8
0.4
0.2
0.0
Inhibition-of-return and Winner-lake-all at fixation n+1
Figure 2: Schematic of the computational model of visual search. The model takes input larget image (4,) and search image (I,), both of which are processed through the same 2D-CNN with shared weights. At each fixation n, the model produces a top-down modulation map (A,) that directs the next eye movement (Section The color bar on the right denotes attention values. Instead of using a typical 2D-CNN with uniform pooling window sizes, here we introduce an accentricity-dependene sampling within each pooling layer of the network by modifying the VGG16 ((4T) architecture. The upper right inset illustrates accentricity-dependeer pooling layer shows the receptive field sizes (ri,n) for each unit j with distance from the centre (the color bar denotes the size of the pooling window in pixels). See Figure S19 for example eye movement patterns from eceNet in each experiment.
works neuro- -automanatomicall connectivity 42 45
6 are consistent with the enhancement of search asymmetry by virtue of eecentrieit-dependeed sampling. We first define notations used in standard average pooling layers [1]. For simplicity, we describe the model components using pixels (Appendix D shows scaling factors used to convert pixels to degrees of visual angles (dva) to compare with human behavior in Section ( The model does not aim for perfect quantitative match with the macaque neurophysiologies receptive field sizes (see Appendix for further discussion).
data, but rather the goal is to preserve the trend of eccentricity
Traditionally, unit j in layer ++ VGG16 takes the average of all input units i the previous layer | within local receptive field size r1+1 and its activation value is given by: v F141 --41 vi
In the enrentricityddependee operation, the receptive field size r/41, of input unit j in layer + linear function of the Fuclidean distance di d'+1. between input unit j and the current fixation location (n) on layer 1.
- 2,
- 6)+ 2.5].
if df /RT41 18+1 < 6
11,
if d 1. /144
(I)
The floor function rounds down the window sizes. The positive scaling factor 7++1 for layer I ++ defines how fast the receptive field size of unit j expands with respect to its distance from the fixation layer receptive field size (Figure BB: in this figure, the fixation location is at the image center). Therefore, the resolution highest in the fixation location and decreases peripheral regions. Based on the slope of eccentricity versus receptive field size in the macaque visual cortex [16] we experimentally set 73 = 0.00, 26 0.00, 110 0.14, 214 = 0.32, and 7i8 0.64. See Figure 3 for the slopes of eccentricity versus receptive field sizes over pooling layers. We define For those units within the fovea, we set a constant receptive field size of 2 pixels. Constant +/++1 is positive scaling factor which converts the dva of the input image to the pixel units at the layer / (see Appendix for specific values of 1). As in the stride size the original pooling layers of VGG16, empirically set constant stride of 2 pixels for all eccentricity-depene pooling layers.
The further away the unit
from the current fixation, the larger the
the constant fovea size.

---

Input image
layer
Model
Macaque
Layer 10
Layer 14
15.0
Layer 18
15.0
RF size (deg)
RF size (deg)
10.0
10.0
After layer
layer
5.0
5.0
0.0
0.0
15
5
Eccentricity (deg)
Eccentricity (deg)
Figure 3: Eccentricity dependence in the model matches that in macaque monkey visual cortex. A. Visualization of one example image and its corresponding eccentricity-dependee
sampling at in to assess the small amount of blurring in the center. The figures are best viewed after zooming
layers 10, 14, and 18 (see Appendix E for implementation).
sampling leads to increasing receptive field sizes as function of eccentricity, in addition to increased receptive field sizes across layers for the model (left) and also for the macaque visual cortex ([16), see Appendix for implementation).
A visualization for ,i,, is shown in Figure where different colors denote how the receptive field sizes expand within a pooling layer. Receptive field sizes r/,n. versus eccentricity df,,, in pixels at each pooling layer are reported in Table S1. Figure 3 illustrates the change in acuity at different pooling layers. These customized eccentricity-deeenddee other object recognition deep neural networks. All the computational steps are differentiable and can be trained end-to-end with other layers. However, since our goal is to test the generalization of eccNET from object recognition to visual search, we did not retrain eccNET and instead, we used the pre-trained weights of VGG16 on the ImageNet classification task.
pooling layers can be easily integrated into
Top-down attention modulation across multiple layers of visual cortex in eccNET. Previous works have described top-down modulation in simplified models for search asymmetry [1 Here we focus on developing a top-down modulation mechanism in deep networks. Given the current th fixation location, the visual cortex of eccNET with eccentricity -eeeeddettyydeeeddeet pooling layers extracts feature maps of4 at layer + for the target image 4. Correspondingly, the model extracts of.,, in response to the search image I,. Inspired by the neural circuitry of visual search the top-down modulation map Ar+1 1-,,n as:
5I
we define
A++1-1, m( Di.,)
where m(-) is the target modulation function defined as 2D convolution with of41 as convolution kernel operating of.,, and Φ441 is the feature map after pooling operation ø1- Note that the layer + modulates the activity of layer I. Following the layer conventions in TensorFlow Keras 1. we empirically selected 1 9, 13, 17 as the layers where top-down modulation performed (see Figure S23 for exact layer numbering).
To compute the overall top-down modulation map An, we first resize A10-9.n and 13, be of the same size as A18-177,0 eccNET then takes the weighted linear combination of normalized
top-down modulation maps across all three layers: A.
Σ
Wi,n (max A141 min 1-11-t..))
(At+1-11, -min A4
1=9,13,17
where Wi,n are weight factors governing how strong top-down modulation the Ith layer contributes to the overall attention map A₃- Each of the top-down attention maps contributes different sets of unique features. Since these features are specific to the target image type, the weights are not necessarily equal and they depend on the demands of the given task. One way to obtain these weights would by parameter fitting. Instead, to avoid parameter fitting specific to the given visual search experiment, these weights Wi,n were calculated during the individual search trials using the maximum activation value from each individual top-down modulation map: wt, = (max (max ( 9,133117 max A()1-inn) ). A higher activation value will mean high similarity
5

---

between the target and search image at the corresponding feature level. Thus, a higher weight to the attention map produced using that feature layer ensures that the model assigns higher importance to the attention maps at those feature layers that are more prominent in the target image (see Figure S22 and Appendix for examples on how the attention map is computed).
Comparison with human performance and baseline models. The psychophysies experiments in Section 2 did not measure eye movements and instead report key press reaction time (RT) in milliseconds when subjects detected the target. To compare fixation sequences predicted by eccNET with reaction times measured in human experiments, we conducted a similar experiment as Experiment 4 in Section using eye tracking (Figure S1, Appendix C). Since RT results from a combination of time taken by eye movements plus finger motor response time, we performed a linear least-squares regression (Appendix B) on the eye tracking and RT data collected from this experiment. We fit RT as function of number of fixations N until the target was found: RT = a * N + B. The slope a 252.36 ms/fixation approximates the duration of a single saccade and subsequent fixation, and the intercept/ 3 376.27 can be interpreted as a finger motor response time. We assumed that and B are approximately independent of the task. We used the same fixed values of and / to convert the number of fixations predicted by eccNET to RT throughout all experiments.
We introduced two evaluation metrics. First, we evaluated the model and human performance showing key press reaction times (RT) as function of number of items on the stimulus, as commonly used in psychophysies 48] [2 46 47 (Section 4). We compute the slope of the RT versus number of items plots for the hard (H, larger search slopes) and easy (E, lower search slopes) conditions within each experiment. We define the Asymmetry Index, as (H-
(H + E)) for each experiment. If model follows the human asymmetry patterns for a given experiment, it will have a positive Asymmetry Index. The Asymmetry Index takes a value of 0 if there is no asymmetry, and negative value indicates that the model shows the opposite behavior to humans.
We included four baselines for comparison with eccNET:
Chance: sequence of fixations is generated by uniform random sampling.
pixelMatching: the attention map is generated by sliding the raw pixels of Le over I, (stride 1 x 1).
GBVS 19 : we used bottom-up saliency as the attention map.
IVSN Si
the top-down attention map is based on the features from the top layer of VGG16.
Results
eccNET predicts human fixations in visual search: The model produces a sequence eye move- ments in every trial (see Figure S19 for example eye movement patterns by eccNet in each experi- ment). The original asymmetry search experiments (FigurqT) did not measure eye movements. Thus repeated the L vs. T search experiment (Experiment 4) to measure eye movements. We also used data from three other search tasks to compare the fixation patterns (51). We compared the fixation patterns between humans and EccNet in terms of three metrics [5:: 1. number of fixations required to find the target in each trial (the cumulative probability distribution p(n)) that the subject or model finds the target within fixations); 2. scanpath similarity score, which compares the spatiotemporal similarity in fixation sequences [5) 55:: and 3. the distribution of saccade sizes. The results show that the model approximates the fixations made by humans on trial-by-triae trial basis both in terms of the number of fixations and scanpath similarity. The model also presents higher consistency with humans in the saccade distributions than any of the previous models in [51] (Figures S15-18)
eccNET qualitatively captures visual search asymmetry. Figure 4 (left, red line) shows the results of Experiment 1 (Figure IA, left), where subjects looked for a straight line amongst curved lines. Increasing the number of distractors led to longer reaction times (RTs), as commonly observed in classical visual search studies. When the target and distractors were reversed and subjects had to search for a curved line in the midst of straight lines, the RTs were lower and showed minimal dependence on the number of objects (Figure 4A, left, blue line). In other words, it is easier to search for a curved line amidst straight lines than the reverse.
The same target and search images were presented to eccNET. Example fixation sequences from the model for Experiment I are shown in Figure S19A. Figure (right) shows eceNET's RT as function of the number of objects for Experiment 1. eccNET had not seen any of these types of

---

Humans
Model
Humans
Model
Humans
Model
Curve among lnee
Vertical
No intersection among cross
among surves
Horizornal
Cross among no intersection
RT (ms)
760
750
750
760
1500
1500
1000
1000
500
200
500
500
16
Number of teems
Number of items
Number of items
Number of itemss
Number itams
Number Items.
E
F
smong
T-20 Dmp
T-O, Duvi20 +i40, +:600 +1:80
Lamong Te
T-o, D-20
D-0, 220002000000 +140, +1-00
(ms)
1300
1600
1030
1000
1000
1000
1000
1000
500
500
800
500
4
12
12
8
8
Number
Number
Number items
Number items
Number of items
Number iterss
The model shows visual search asymmetry and qualitatively captures human behav- ior. Reaction time as a function of the number of items in the display for each of the six experiments in Figure for for humans (left, solid lines) and for the model (right, dashed lines). The results for
Figure 4:
humans were obtained from [48] 22, (46 [47]. The line colors denote the different experimental conditions. Error bars for the model denote standard error (SE). Error bars were not available in the original publications for the human data.
images before and the model was never trained to search for curves or straight lines. Critically, there was no explicit training to show an asymmetry between the two experimental conditions. Remarkably, eccNET qualitatively captured the key observations from the psychophysics experiment (48). When searching for curves among straight lines (blue), the RT' were largely independent of the number of distractors. In contrast, when searching for straight lines among curves (red), the RTs increased substantially with the number of distractors. The model's absolute RTs were not identical to human RTs (more discussions in Section 5 Of note, there was no training to quantitatively fit the human data, In sum, without any explicit training, eccNET qualitatively captures this fundamental asymmetry in visual search behavior.
We considered five additional experiments (Section 2 that reveal similar asymmetries on a wide range of distinct features (Figure (Figure[B-F)), BB-F). In Experiment 2. it is easier to search for top-down luminance changes than left-right luminance changes (Figure LB: (221).
l) In Experiments 3-4, it is easier to find
non-intersections among crosses than the reverse (Figure 1 C: [46] ), and it is easier to find a a rotated
letter T among rotated letters I. than the reverse (Figure ID; 446 ). In Experiments 5-6, it is easier
to find tilted bar amongst vertical distractors than the reverse when distractors are homogencous (Figure TE; (47) but the situation reverses when the distractors are heterogenous (Figure IE-F; (7)). In all of these cases, the psychophysies results reveal lower RTs and only weak increase in RTs with increasing numbers of distractors for the easier search condition and a more pronounced increase in RT with more distractors for the harder condition (Figure 44-F. left panels).
Without any image-specific or task tasc-spe training, eccNET qualitatively captured these asymmetries (Figure 4 ADD F, right panels). As noted in Experiment 1, eccNET did not necessarily match the human RT' at quantitative level. While in some cases the RT's for eccNET were comparable to humans (e.g., Figure FC), in other cases, there were differences (e.g., Fi Despite the shifts along the y-axis in terms of the absolute RTs, eccNET qualitatively reproduced the visual search asymmetries in 5 out of 6 experiments. However, in Experiment 5 where humans showed minimal search asymmetry effect, eccNET showed the opposite behavior (more discussions in Section 55)
The slope of RT versus number of object plots is commonly used to evaluate human search efficiency. We computed the Asymmetry Index (Section 3 to compare the human and model results. Positive values for the asymmetry index for eccNET indicate that the model matches human behavior. There was general agreement the Asymmetry Indices between eccNET and humans, except for Experiment 5 (Figure 5)) Since the Asymmetry Index is calculated using the search slopes (Section 3 is independent of shifts along the y-axis in Figure Thus, the eccNET Asymmetry Indices can match the human indices regardless of the agreement in the absolute RT values.
7

---

A.
B.
133
0.6
Model Asymietry Index
Asymmetty Indox
0.4
Exp1
Exp2
Exp3
0.0
Exp4
-02
Exp5
-0.4
Exp8
-0.5
-0.8
Humants exxNET IVSN BcNETT MNNST GEZNET 0 BcKNET
chance
gbva piseiMathh ecoNET Rota0
-1.9-
-1.0
-0.5
0.0
0.5
1.0
Human Asymmetty Index
Figure 5: EccNet outperforms other baselines and ablation demonstrates the relevance of its components. A. Asymmetry Index (Section3) for eceNet (y-axis) versus humans (x-axis) for each experiment. Points in the 1st quadrant indicate that eccNet follows the human asymmetry patterns. See Figure S14 for Asymmetry Index plots of other models. B. Comparison to ablated and alternative models. Average Asymmetry Index for humans (dark green), eccNET (light green), and alternative models (light gray) defined in Section 3 experiments.
The error bars show the standard error (SH) over all 6
Baseline models fail to capture search asymmetry: We compared the performance of eccNET with several baseline models (Figure see also Figures S2-S5 for the corresponding RT versus number of objects plots). All the bascline models also had infinite inhibition of return, oracle recognition, and the same method to convert number of fixations into RT values.
As a null hypothesis, we considered a model where fixations landed on random locations. This model consistently required longer reaction times and yielded an Asymmetry Index value close to zero, showing no correlation with human behavior (Figure 5B. Figure S3, chance). A simple template- matching algorithm also failed to capture human behavior (Figure [B, Figure S5 pixelMatch), suggesting that mere pixel-level comparisons are insufficient to explain human visual search. Next, we considered purely bottom-up algorithm that relied exclusively on saliency (Figure Figure 84, GBVS); the failure of this model shows that it is important to take into account the top-down target features to compute the overall attention map.
None of these baseline models contain complex features from natural images. We reasoned that the previous IVSN architecture [51], which was exposed to natural images through ImageNet, would yield better performance. Indeed, IVSN showed higher Asymmetry Index than the other baselines, yet its performance was below that of eccNET (Figure 5B. Figure S2, /VSN). Thus, the components introduced in eccNET play a critical role; next, we investigated each of these components.
Model ablations reveal essential components contributing to search asymmetry: We systemati- cally ablated two essential components of eccNET (Figure 5B. In eccNET, top-down modulation occurs at three levels: layer 10 to layer 9 (A10-9), 14 to 13 (A14-11), and 18 to 17 (A18-17). We considered eccNET 18-17. where top-down modulation only happened at layer 18 to 17 (A18-117). eccNET 18-117 yielded lower average Asymmetry Index (0.084), suggesting that visual search benefits from the combination of layers for top-down attention modulation (see also Figure S7).
To model the distinction between foveal and peripheral vision, we introduced pooling layers. The resulting increase in receptive field size qualitatively consistent with the receptive field sizes of neurons in the macaque visual cortex (Figure 3B.. To assess the impact of this step. we replaced all serentricity-depended (cccNET,oecd) ). The lower average Asymmetry Index implics that max- ppoling layers of VGG16 an essential component in eccNET for asymmetry in visual search (see also Figure S6). Moreover, to evaluate whether asymmetry depends on the architecture of the recognition backbone, we tested ResNet152 200 on the six experiments (Figure S24). Though the ResNet152 backbone does not approximate human behaviors as well eccNet with VGG16, it still shows similar asymmetry behavior (positive asymmetry search index) in four out of the six experiments implying search asymmetry is general effect for deep networks.
pooling layers with the max
pooling is
The statistics of training data biases polarity search asymmetry. In addition to the ablation experiments, image-computable model enables to further dissect the mechanisms responsible
8

---

Vertical
Horizontal
A. Human
B. ecoNET
C. ecoNET COONET_MNIST _INIST
D. ecoNET SHINET_Rotee Ro190
750
750
2000
750
RI (ms)
500
500
1000
500
6
Number of items Figure 6: Training data biases polarity of search asymmetry. Reaction times as a function of the number of items for Experiment 2 on illumination conditions. (A-B) Humans and eccNET (same results as in Figure 43 reproduced here for comparison purposes). (C) eccNET trained on the MNIST 13 (D) eccNET trained ImageNet [2]] after 90 degree rotation.
Number items
Number items
Number items
for visual search asymmetry by examining the effect of training. Given that the model was not designed, or explicitly trained, to achieve asymmetric behavior, we were intrigued by how asymmetry could emerge. We hypothesized that asymmetry can be acquired from the features learnt through the natural statistics of the training images. To test this hypothesis, as a proof-of-princije we first focused on Experiment 2 (Figure IB).
We trained the visual cortex of eccNET from scratch using training set with completely different image statistics, Instead of using ImageNet containing millions of natural images, we trained eccNET on MNIST [3], which contains grayscale images of in Experiment 2 (Appendix G2). The asymmetry effect of eccNET MNIST disappeared and the absolute reaction times required to find the target increased substantially (Figure(6C) The average Asymmetry Index was also significantly reduced (Figure 5B, Figure S8).
digits. We tested eccNET cocNET_MNIST MNIST
To better understand which aspects of natural image statistics are critical for asymmetry emerge, we focused on the distinction between different lighting directions. We conjectured that humans are more used to lights coming from a vertical than a horizontal direction and that this bias would be present in the ImageNet dataset. To test this idea, we trained eccNET from scratch using an altered version of ImageNet where the training images were rotated by 90 degrees counter-clockwise. and tested eccNET on the same stimuli in Experiment 2 (see Appendix G1 for details). Note that random image rotation is not a data augmentation step for the VGG16 model pre-trained on ImageNet 4I] thus, the VGG16 pre-trained on ImageNet was only exposed to standard lighting conditions. Interestingly, eceNET trained on the 90-degree-rotated ImageNet (eccNET Ro190) shows a polarity reversal in asymmetry (Figure 6 Figure S9), while the absolute RT difference between the two search conditions remains roughly the same as eccNET.
We further evaluated the role of the training regime in other experiments. First, we trained the model on Imagenet images after applying a "fisheye" transform to reduce the proportion of straight lines and increase the proportion of curves. Interestingly we observed reversal in the polarity of asymmetry for Experiment 1 (curves among straight lines search) while the polarity for other experiments remained unchanged (Figure S10). Second, we introduced extra vertical and horizontal lines in the training data, thus increasing the proportion straight lines. Since reducing the proportion of straight lines reverses the polarity for "Curve vs Lines", we expected that increasing the proportion of straight lines might increase the Asymmetry Index. However, the polarity for asymmetry did not change for Experiment 1 (Figure S11). Third, to test whether dataset statistics other than Imagenet would alter the polarity, we also trained the model on the Places 365 dataset and rotated Place 365 dataset. We found this manipulation altered some of the asymmetry polarities but not others, similar to the experiment done on MNIST dataset. Unlike the results using the MNIST case, the absolute reaction times required to find the target did not increase significantly (Figures S12-13). In sum, both the architecture and the training regime play an important role in visual search behavior. In most cases, but not in all manipulations, the features learnt during object recognition useful for guiding visual search and the statistics from the training images contribute to visual search asymmetry.
5 Discussion
We examined six classical experiments demonstrating asymmetry in visual search, whereby humans find target, A, amidst distractors, B, much faster than in the reverse search condition of B among A.

---

Given the similarity between the target and distractors, it is not clear why one condition should be easier than the other. Thus, these asymmetries reflect strong priors in how the visual system guides search behavior. We propose eccNET, a visual search model with accentricttyddeeeneen sampling and top-down attention modulation. At the heart of the model is "ventral visual cortex" module pre-trained on ImageNet for classification [36]. The model had no previous exposure to the images in the current study, which are not in ImageNet. Moreover, the model was not trained in any of these tasks, did not have any tuning parameters dependent eye movement or reaction time (RT) data, and was not designed to reveal asymmetry. Strikingly, despite this lack of tuning or training, asymmetric search behavior emerged in the model. Furthermore, eccNET captures observations in five out of six human psychophysics experiments in terms of the search costs and Asymmetry Index.
image-computable models allow us to examine potential mechanisms underlying asymmetries in visual search. Even though the target and search images are different from those in ImageNet, natural images do contain edges of different size, color, and orientation. To the extent that the distribution of image statistics in ImageNet reflects the natural world, a question that has been contested and deserves further scrutiny, one might expect that the training set could capture some priors inherent to human perception. We considered this conjecture further, especially in the case of Experiment 2. The asymmetry in vertical versus horizontal illumination changes has been attributed to the fact that animals are used to seeing light coming from the top (the sun), a bias likely to be reflected in ImageNet. Consistent with this idea, changing the training diet for eccNET alters its visual search behavior. Specifically, rotating the images by 90 degrees, altering the illumination direction used to train eccNET, led to reversal in the polarity of search asymmetry. Modifications in search asymmetry in other tasks were sometimes also evident upon introducing other changes in the training data such as modifying the proportion of straight lines, using the places365 or MNIST datasets (Figure 66 S8, S10-13). However, the training regime is not the only factor that governs search asymmetry. For example, the network architecture also plays critical role (Figure|5B)
Although eccNET qualitatively captures the critical observations in the psychophysics experiments, the model does not always yield accurate estimates of the absolute RTs. Several factors might contribute towards the discrepancy between the model and humans. First and foremost, eccNET lacks multiple critical components of human visual search abilities, as also suggested by [27]. These include finite working memory, object recognition, and contextual reasoning, among others. Furthermore, the VGG16 backbone of eccNET constitutes only first-order approximation to the intricacies of ventral visual cortex. Second, to convert fixations from eccNET to key press reaction time, we assumed that the relationship between number of fixations and RT's is independent of the experimental conditions. This assumption probably constitutes an oversimplification. While the approximate frequency of saccades tends be similar across tasks, there can still be differences in the saccade frequency and fixation duration depending on the nature of the stimuli, on the difficulty of the task, on some of the implementation aspects such as monitor size and contrast, and even on the subjects themselves [52] Third, the human psychophysics tasks involved both target present and target absent trials. Deciding whether an image contains target or not (human psychophysics experiments) is not identical to generating a sequence of saccades until the target is fixated upon (model). It is likely that humans located the target when they pressed a key to indicate target presence, but this was not measured in the experiments and it is conceivable that a key was pressed while the subject was still fixating on a distractor. Fourth, humans tend to make "return fixations", whereby they fixate on the target, move the eyes away and come back to the target [52] while eccNET would stop at the first time of fixating on the target. Fifth, it is worth nothing that it is possible to obtain tighter quantitative fits to the RTs by incorporating bottom-up saliency contribution to the attention map and fitting the parameters which corresponds to the relative contribution of top-down and bottom-up up attention maps (see Appendix H and Figures S20-21 for more details).
We introduce biologically plausible visual search model that can qualitatively approximate the asym- metry of human visual search behaviors. The success of the model encourages further investigation of improved computational models of visual search, emphasizes the importance of directly comparing models with biological architectures and behavioral outputs, and demonstrates that complex human biases can be derived from model's 's architecture and the statisties of training images.
10

---

Acknowledgements
This work was supported by NIH RO1EY026025 and the Center for Brains, Minds and Machines, funded by NSF STC award CCF-1231216. Mengmi Zhang was supported by postdoctoral fellowship of the Agency for Science, Technology and Research. Shashi Kant Gupta was supported by The Department of Biotechnology (DBT), Govt of India, Indo-US Science and Technology Forum (IUSSTF) and WINStep Forward under "Khorana Program for Scholars".
References
[1] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeflrey Dean, Matthieu Devin, et al. Tensorflow: Large-scale machine learning on beterogeneous distributed systems. arXiv preprint arXiv:1603.000 1603.04467, 2016.
[2] John Ackermann and Michael $Laan Landy. Statistical templates for visual search. Journal of Vision, 14(3):18-18, 2014.
[3] Narcisse P Bichot, Matthew T Hcard, Ellen M DeGennaro, and Robert Desimone, A source for feature- based attention in the prefrontal cortex. Neuron, 88(4):832- 844, 2015.
[4] James W Bisley. The neural basis of visual attention. The Journal of Physiology, 589(1):49-55) 57, 2011.
Ali Borji and Laurent Itti. Statc-of-the-ard in visual attention modeling. IEEE transactions on pattern analysis and machine intelligence, 35(1):185-201 2012.
[6] Richard T Born, Alexander R Trott, and Till S Hartmann. Cortical magnification plus cortical plasticity equals vision? Vision research, 111:161-166 2015.
[7] Neil Bruce and John Tsotsos. Visual representation determines search difficulty: explaining visual search asymmetries. Frontiers in Computational Neuroscience, 5:33, 2011.
Marisa Carrasco, Tracy McLean, Svetlana M Katz, and Karen S Frieder. Feature asymmetries in visual search: Effects of display duration, larget eccentricity, orientation and spatial frequency. Vision research, 38(3):377-374, 1998.
[9] Leonardo Chelazzi, Farl K Miller, John Duncan, and Robert Desimone. Responses of neurons in macaque area v4 during memory-guides visual search. Cerebral cortex, 11(8):761-77, 2001.
[10] Maurizio Corbetta. Frontoparietal cortical networks for directing attention and the eye visual locations: identical, independent, or overlapping neural systems? Proceedings of National Academy of Sciences, 95(3):831-833 1998.
[11] A Cowey and ET Rolls. Human cortical magnification factor and its relation to visual acuity. Experimental Brain Research, 21(5):447 454, 1974.
[12] PM Daniel and D Whitteridge. The representation of the visual field on the cerebral cortex in monkeys. The Journal of physiology. 159(2):203-221, 1961.
[13] Li Deng. The mnist datahase of handwritten digit images for machine learning research [best of the web]. IEEE Signal Processing Magazine, 2916):1411142 2012.
[14] Robert Desimone and John Duncan. Neural mechanisms selective visual attention. Annual Review of Neuroscience, 18(1):193-222, 1995.
[15] Barbara Anne Dosher, Songmei Han, and Zhong-Lin Lu. Parallel processing in visual search asymmetry. Journal of Experimental Psychology: Human Perception and Performance, 30(1):3, 2004.
[16] Jeremy Freeman and Eero P Simoncelli. Metamers of the ventral stream. Nature Neuroscience, 14(9): 1955- 1201. 2011.
[17] lan Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http //www /tww:ddeepppe wwwpleermiingookk.coo org
[18] Marie-Hélène Grosbras. Angela R Laird, and Tomás Paus. Cortical regions involved in eye movements, shifts of attention, and gaze perception. Human brain mapping, 25(1):140-15 154, 2005.
11

---

[19] Jonathan Harel, Christof Koch, and Pietro Perona. Graph-based visual saliency. Advances in neural information processing systems, 19:545-552, 2006.
[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pottern recognition, pages 770-778, 2016.
[21] Dietmar Heinke and Andreas Backhaus. Modelling visual search with the selective attention for iden- tification model (vs-saim): a novel explanation for visual search asymmetries. Cognitive computation, 3(1):185-205, 2011.
[22] Dorothy A Kleffner and Vilayanur S Ramachandran. On the perception of shape from shading. Perception & Psychophysics, 52(1):18 36, 1992.
[23] Gabriel Kreiman and Thomas Serre. Beyond the feedforward sweep: feedhack computations in the visual cortex. Annals of the New York Acudemy of Sciences, 1464(1):222) 2020.
[24] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 25, pages 1097- 1105. Curran Associates, Inc., 2012.
[25] Dale K Lee, Laurent Itti, Christof Koch, and Jochen Braun. Attention activates winner-take-all competition among visual filters, Nature neuroscience, 2(4):375-381, 1999.
[26] Thomas Miconi, Laura Groomes, and Gabriel Kreiman. There's 's waldo! a normalization model of visual search predicts single-trial human fixations in an object search task. Cerebrad Cortex, 26(7):3064-308 2015.
[27] Jiri Najemnik and Wilson S Geisler. Optimal eye movement strategies in visual search. Nature, 434(7031):387-33 2005.
[28] Khaled Nasr, Pooja Viswanathan, and Andreas Nieder. Number detectors spontaneously emerge in a deep neural network designed for visual object recognition. Science advances, 5(5):caav7003, 2019.
[29] Jacob L Orquin and Simone Mueller Loose. Attention and choice: A review on eye movements in decision making. Acta Psychologica, 190(1):190-206, 2013.
[30] Sofia Paneri and Georgia G Gregorion. Top-down control visual attention by the prefrontal cortex. functional specialization and long-range interactions. Frontiers in Neuroscience, 11:545, 2017.
[31] John H Reynolds and Leonardo Chelazzi. Attentional modulation of visual processing. Annu. Rev. Neurosci., 27:611-644 647, 2004.
[32] Maximilian Ricsenhuber and Tomaso Poggio. Hicrarchical models of object recognition in cortex. Nature Neuroscience, 2(11):001--1025 1999.
[33] Samuel Ritter, David GT Barrett, Adam Santoro, and Matt M Botvinick. Cognitive psychology for deep neural networks: A shape bias case study. International conference on machine learning, pages 2940-2949, PMLR, 2017.
[34] Ruth Rosenholtz. Search asymmetries? what search asymmetries? Perception & Psychophysics, 63(3):766- 489, 2001.
[35] Ruth Rosenholtz. Jie Huang, Alvin Raj. Beajamin J Balas, and Livia Ilie. A summary statistic representation in peripheral vision explains visual search. Journal of Vision, 12(4):14-14 14, 2012."
[36] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 115(3):1112522 2015.
[37] Alexander C Schützz Doris I Braun, and Karl R Gegenfurtner. Eye movements and perception: A selective review. Journal of Vision, 11(5):9-9. 9, 2011.
[38] T Serre, A Oliva, and T Poggio. Feedforward theories of visual cortex account for human performance in rapid categorization. PNAS, 104(15):6424 6429, 2007.
[39] Thomas Serre. Gabriel Kreiman, Minjoon Kouh. Charles Cadieu, UIf Knoblich, and Tomaso Poggio. A quantitative theory of immediate visual recognition. Progress in Brain Revearch, 165:33-56, 2007.
[40] Jiye Shen and Eyal M Reingold. Visual search asymmetry: The influence of stimulus familiarity and level features. Perception & Psychophysies, 63(3):464 475,, 2001.
12

---

[41] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. arXiv preprint arXiv:1409. 2014.
[42] Roger B Tootell, Eugene Switkes, Martin S Silverman, and Susan L Hamilton. Functional anatomy of macaque striate cortex. ii retinotopic organization. Journal of neuroscience, 8(5):1531-1568, 1988.
[43] Anne Treisman and Stephen Gormican. Feature analysis in early vision: evidence from search asymmetries. Psychological Review, 95(1):15, 1988.
[44] Sabine Kastner Ungerleider and Leslic G. Mechanisms of visual attention in the human cortex. Annual Review of Neuroscience, 23(1):315-4) 3- 2000.
[45] David Essen and John HR Maunsell. Hierarchical organization and functional streams the visual cortex. Trends in neurosciences, 6:370-375, 1983.
[46] Jeremy M Wolfe Jennifer S DiMase. Do intersections serve as basic features visual search? Perception, 32(6):645 656, 2003.
[47] Jeremy Wolfc, Stacia R Friedman Hill, Marion Stewart, and Kathleen M O'Connell. The role of categorization visual search for orientation. Journal of Experimental Psychology: Human Perception and Performance, 18(1)334, 1992.
[48] Jeremy Wolfe, Alice Yee, Stacia Friedman-HHin Curvature basic feature for visual search tasks. Perception, 21(4):6654480 1992.
[49] JeremyM Wolfe and ToddS Horowitz. Five factors that guide attention in visual search. Nature Human Behaviour, 1(3):1-8, 2017.
[50] Yusuke Yamani and Jason S McCarley. Visual search asymmetries within color-coded and intensity-coded displays. Journal of Experimental Psychology: Applied, 16(2):124, 2010.
[51] Mengmi Zhang, Jiashi Feng, Keng Teck Ma, Joo Hwee Lim, Qi Zhao, and Gabriel Kreiman. Finding any waldo with zero-shot invariant and efficient visual search. Nature Communications, 9(1):1-15, 2018.
[52] Mengmi Zhang, Will Xiao, Olivia Rose, Katarina Bendtz, Margaret Livingstone, Carlos Ponce, and Gabriel Kreiman. Look twice: computational model of return fixations across tasks and species. arXiv preprint 1xXiv:2001.01611.1 2021.
[53] Shengjia Zhao, Hongyu Ren, Arianna Yuan, Jiaming Song, Noah Goodman, and Stefano Ermon, Bias and generalization in deep generative models: An empirical study. arXiv preprint 1rXiv:/8/1.032555 2018.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Sec. 5
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] URL included
in the paper
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix G.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] For the applicable figures, we provided SE (standard errors) errorbars.
13

---

(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix
4. If you using existing assets (e.g., code, data, models) curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either the supplemental material or as URL? [Yes]
Together with source code, we uploaded all the stimuli in zip file.
(d) Did/ you discuss whether and how consent was obtained from people whose data you'le rre're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [Yes] See Appendix C and FigS1.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [Yes] See Appendix C.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [Yes] See Appendix C.
14


Reviews:
[{"review_id": "_x35bkMm2nH", "reviewer": "Reviewer_CMyL", "summary": "In this manuscript, the authors present a visual search model, which predicts a fixation order over visual search displays. The model is based on VGG-16 replacing the max-pooling layers with average pooling with growing pooling area towards the periphery. The priority for fixation is then computed as a combination of target-image similarity at three different feature maps in the network. The primary result reported is that model qualitatively reproduces 5 out of 6 visual search asymmetries. ", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the observation that visual search asymmetry might be caused by peripheral pooling is interesting and the model the authors use seems reasonable. I have a relatively long list of questions though and in quantitative terms the fit of the model is bad. Thus, I do not find the model and manuscript convincing yet.\n\nMy first questions are on Figure 3: Why didn’t the authors choose the pooling areas such that the eccentricity dependence follows the ones of the visual areas they compare to? Also, are there any reasons why Layer 10 should be associated with V1 etc? \nAlso, the illustration in A appears to be wrong as the center is not blurred at all, while the formulas give a 2 pixel radius pooling for the fovea at each location, which should correspond to noticeable blur in the center.\n\nI find the pooling operation across attention maps strange. Why are the 3 A maps scaled as they are? The overall scaling amounts to max(A) / (max(A) - min(A)), i.e. a stronger weighting the larger the minimal value is in proportion to the maximum. How does this make any sense?\n\nAlso I find the nomenclature of calling the maps which measure target-image similarity at the different layers (A_{i \\rightarrow j}) very odd as it strongly implies a modulation of the processing within the network producing the activations. If I understand the manuscript and the illustration in Figure 2 right, this does not happen though, but the convolution is only applied to create the readout maps.\n\nOn the pixelMatching model: As all search images with the target contain an exact copy of the target: Why doesn’t the pixel matching model find all targets within the first fixation it chooses?\n\nAs the model produces whole scan paths: Have the authors considered looking at more detailed aspects of them rather than just the overall duration?\n\n\n----\n\nAfter comments:\nI have just slightly increased my evaluation of this manuscript as the authors have addressed some of my concerns. Especially they now do some comparisons on the scan-paths and show some better quantitative fits to the reaction time data.\nAlso reading some of the other reviews highlighted some of the advantages of this manuscript.\n\nUnfortunately, the authors' answers didn't remove my concerns about the model though. Some choices still appear strange/unusual or not exactly representative for the described mechanisms to me. For me this casts doubt on any of the interpretations, which of course rest on the assumption that the model is a good representation of the theoretical ideas.\nIt is entirely possible, that my concerns can be addressed and this manuscript ends up being correct, but I honestly see some probability that the results do depend on some oddity of the model and are not interpretable as they are interpreted here. \n\nThus, I won't improve my rating further here, but suggest some clarification and maybe some controls as suggested by other reviewers may help for the future.\n\nTo be concrete:\n- on 2.2 / 2.3.: I still don't understand why the pooling regions were not exactly matched to the monkey data. Scaling pixels and the slope should allow any linear function and thus a perfect fit between the two figures. Little of the conclusions in this manuscript rests on the question which area corresponds to which layer, but not making them fit where you clearly could is strange.\n- on 2.5: I did understand the scaling of the attention maps. My criticism is that the described scaling does not appear sensible to me. As you weigh maps by their maxima after scaling them by their range the difference in scale is restored and each map is effectively scaled by how much of its range is above 0, which seems highly unusual to me.\n- on 2.6: I originally formulated my criticism here aimed at changing the formulation, thinking that the authors \"obviously\" cannot mean to interpret this interaction as top-down modulation. It seems that they do mean to interpret this as an implementation of top down modulation. I strongly disagree with this interpretation.   \nThe two interacting layers are a feature map and a blurred version of the same feature map (or RELU transformed, I am not perfectly sure due the confusing layer numbering) . Thus, the operation is a template match with a slightly blurred representation of the target (slightly, because the target image is processed at fixation if I am not mistaken). Template matching is a typical strategy used in search models, but has no connection to top-down modulation for me. If there is any top-down connection it is from working memory to the feature map not between layers of processing.\n\nNote on the last point: If the two layers were not using closely related feature maps, convolving the feature map with a target representation in the higher layer would not make any sense and would in many cases not be possible due to a mismatch in the number of features. \n\nBy the way: Something appears to be off about the counting of layers the authors use. They write that layer 10 corresponds to the 6th convolutional layer, but I checked and VGGs use RELUs after each convolution, such that it could be at most the 5th and in fact guessing based on the figure and a look at the pytorch hub VGG could be either the second max-pooling layer, which appears after 4 convolutions or the third, which happens after 7 convolutions though and has a layer index of 16 in pytorch. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do not comment on their limitations and societal impact. Besides the sampling biases of the used databases I don’t think there are important societal impacts associated with the content of this paper though.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "11"}, "extra_text": {}}, {"review_id": "_AbGMoT-oRP", "reviewer": "Reviewer_h9eJ", "summary": "This paper proposes a computational model (eccNET) that produces visual search asymmetry patterns similar to those seen in humans. The model is pretrained on ImageNet and integrates eccentricity and top-down cues to produce a series of fixations until the target is found. Since a rotation of the training set changes the pattern of results, the authors conclude that visual search asymmetry may be a consequence of natural image statistics.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The goals of this paper appear to be twofold: first, to understand whether visual search asymmetry of the type seen in psychophysics experiments emerges in neural networks not specifically trained in visual search; and second, to understand the mechanisms through which this happens by manipulating the model architecture and training set. \n\nThe paper takes a thorough approach both in characterizing human behavior (using data from six experiments) and machine behavior (four baselines, as well as model ablations to assess the role of model features in matching human behavior). I found that many questions that arose while reading the paper had been addressed by the end. The paper is well-written and well-structured.\n\nThe authors used data from six psychophysical experiments with different visual search tasks showing asymmetry in humans. Since these experiments relied on target-present/target-absent keypress responses, the authors collected an additional eyetracking dataset to aid in linking fixation data from eccNet to RT data in humans. Limitations of this approach are discussed in the paper; my only comment is that it would be interesting to see how the fixation data compares to eccNet in this one experiment (despite all the expected differences that the authors discuss).\n\nIn terms of pinpointing hypotheses for why this asymmetry emerges in eccNet, the paper seems to hint at a conclusion. First, it appears that biologically plausible model features are crucial (eccentricity and top-down feedback at several layers have a larger impact on asymmetry than training eccNet with MNIST), which seems to be less emphasized in the discussion than the role of the training set. Second, not all mechanisms are explored - some visual search effects are not reversed after training set rotation, suggesting some may arise from different visual features. \n\nThe fact that the 90-degree rotation of the training set leads to an effect polarity reversal does appear to point at statistical regularities in the training set, which is in line with previous research suggesting e.g. familiarity as a potential factor in visual search asymmetry. However, the authors also mention potential limitations of ImageNet. I am curious as to whether effects from the other experiments could be disrupted through training set alterations, and what explains experiment E (it'd be interesting to see what would happen with a different training set, e.g. Places)\n\nMinor points:\n\nIn Figure 4 there seem to be no error bars in the human data plots? Figure 5 could also have error bars (e.g. SEM across experiments). The number of subjects who took part in the psychophysics experiments doesn’t seem to be mentioned even in the appendix.\n\n---------------------------------------\n\nAfter response:\n\nThank you for addressing my points so thoroughly. Great to see a direct fixation comparison, as well as the new training regimes - look forward to seeing the updated paper!", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have addressed the limitations of the work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "Ff7i_QXN2jT", "reviewer": "Reviewer_F9R3", "summary": "The authors proposed a computational model to illuminate the mechanism underlying the search asymmetry phenomenon (finding a target A among distractors B can be easier than vice versa). Their model takes a search image and a target as inputs and produces a sequence of eye movements until the target is found. The model integrates eccentricity-dependent visual recognition with target-dependent top-down modulations. Without prior exposure to the stimuli or task-specific training, the model reproduces the polarity of search asymmetry in human vision. They also showed that when the statistics change in the dataset, the search asymmetry either disappeared or was altered.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Whether the contribution is original depends on the motivation of the work and the interpretation of the results. Particularly, if the authors aim at elucidating the mechanism underlying visual search asymmetry (as stated in the abstract), then the conclusion is not very new. Visual search asymmetry has been long studied in neuroscience. Various models have been proposed to explain search asymmetry. For instance, Heinke and Backhous 2011 modeled visual search with selective attention to explain search asymmetry and showed that search asymmetry is the outcome of interactions between top-down and bottom-up processing. Various factors have been identified as sources of search asymmetry. For instance, Bruce and Tsotsos (2011) argued that stimulus familiarity determines search difficulty and may help explain search asymmetry. If the work intends to address the long-standing question of the mechanisms underlying visual search asymmetry, the authors should mention the related literature in vision sciences, including the existing theories/computational models of visual search.\n\nBy contrast, if the authors aim at showing \"Deep Nets and Humans Share Similar Inherent Biases\" as the title states, then the conclusion is kind of new. But the authors should mention other papers that demonstrate similar inherent bias between deep nets and humans, such as \"Cognitive Psychology for Deep Neural Networks: A Shape Bias Case Study\" by Ritter et al., 2017. \"Bias and Generalization in Deep Generative Models: An Empirical Study\" by Zhao et al. 2018 or \"Number detectors spontaneously emerge in a deep neural network designed for visual object recognition\" by Nasr et al. 2019.\n\nThe submission is relatively sound. The authors conducted psychophysics experiments to compare the search asymmetry in the deep nets and the ones in human subjects. They also conducted thorough ablation studies to confirm the necessity of the model components to reproduce the asymmetry. I have three major concerns. First, although the author collected the eye movement data, they only compare the reaction time and number of fixations with the model prediction. I was wondering why they did not compare the eye movement trajectory with the predicted trajectory directly. If there's a huge discrepancy between the model predicted trajectory and the human behaviors, the authors should provide some explanations.\n\nSecond, the authors claim that visual search is \"the polarity of search asymmetry arises from experience with the natural environment.. is a consequence of the statistical properties of the developmental diet fed to the model.\" And there are only two experiments supporting that claim, which is 1) replacing the naturalistic dataset ImageNet with an MNIST and 2) rotating ImageNet images by 90 degrees. These are just some special cases of how statistics in the training set can be altered. In order to draw a strong conclusion as what the authors claimed, a much comprehensive manipulation of different statistics of the training set should be done, including contrast, spatial frequency, brightness, orientation, etc. It's entirely possible that changing some statistics won't alter the search asymmetry, contrary to what the authors found using the limited set of two experiments. In a word, I think the claim that when statistics in the training set change, the polarity of the asymmetry would be altered is a bit overgeneralizing. A minor issue related to this is that rotating 90 degrees not only changes the lighting from directions but also many other things (e.g., familiar objects such as trees and dogs would appear unfamiliar), which makes the authors' argument less convincing.\n \nFinally, although it's nice to see that without any task-specific training, search asymmetry naturally emerges from the network, the author did not give any intuition on why adding eccentricity-dependent visual recognition with target-dependent top-down cues enables that to happen. If the author could connect that to classic theories on the human visual search to provide some intuition on why those model components give rise to search asymmetry, that could make the paper more informative and inspiring than simply showing the model works.\n\nThe submission is clearly written and well organized. If the authors could make their reasoning more solid, then the results can be important in the sense that it reveals that deep neural networks share similar inherent biases with humans when it is equipped with eccentricity-dependent visual recognition with target-dependent top-down cues. This would be useful to build AI systems that can simulate human visual search behaviors, which is extremely useful in fields like user-interface design.\n\n------------------------------------------\n\nAfter comments: I've read the authors' responses to my review as well as the ones to the other reviewers'. I think their responses address some of my concerns, especially the intuition about eccentricity and top-down modulation in asymmetry. Given these improvements, I decided to raise my score to 7.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": " There's no foreseeable negative societal impact of this work.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10 hours"}, "extra_text": {}}, {"review_id": "-ya-0k5XYaw", "reviewer": "Reviewer_CJM4", "summary": "This paper proposed an image-computable model to examine potential mechanisms underlying asymmetries in visual search. The proposed method can capture similar behaviour as the human user, and the proposed architecture is novel. However, I am not fully convinced why we need to investigate the asymmetries with the neural networks model, and I am not sure how much of the training data instead of the proposed method can affect the results.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Paper strength**\n- The paper is well-written. The author provides sufficient details of the implementation of the proposed method, and a good illustration of the methods as well as the experiment results. \n\n- The proposed method has a certain novelty. The eccentricity-dependent pooling is meaningful for such an attention-based model, and the author extracts the feature according to the number of the layer. The author also proposes a top-down attention modulation. Both two components are evaluated with the ablation study to show their effectiveness (Sec. 4.3).\n\n- The experiment (Fig. 4) shows the proposed method successfully capture the visual search asymmetry of the human user. It is an interesting finding that the pre-trained model on ImageNet has such ability. In comparison with other baselines in Fig. 5, the proposed method shows it achieve leading performance and similar to the performance of the human user. \n\n- The author provides interesting discussions on where the asymmetry comes from in Sec. 4.4. The author thinks the asymmetry comes from the lighting direction within the natural images.\n\n**Paper weakness**\n- It is not clear to me why we want the trained model to have similar behaviour as the human. Especially, why we want the model to capture such asymmetry as the human being? I understand that the current machine and human have their own advantage respectively, and it is not necessary that the machine must learn from the human. \n\n- One experiment shows such asymmetry also depends on the training data. The model trained on the MNIST dataset does not have an asymmetry effect. It shows that such asymmetry actually depends on the data instead of the method/architecture. If we want to investigate the asymmetry, we should perform the analysis on the data distributions. \n\n- It will be very interesting if the author found the asymmetry is a general effect for neural network models. However, it is not the case as shown in Fig. 5. The asymmetry can be captured well with the proposed method, not other neural networks models. In this sense, it is not clear to me why the author wants to build such a model.\n\n-The assumption that the asymmetry comes from the lighting condition is not convincing. It might be true for the outdoor environments but not for the indoor setting. It will be better if the author could provide any static numbers to support their argument. Such an assumption also cannot explain other shapes such as the curve.\n\n- I would not agree that the classification result of 66.7% is similar to 72% (line 267).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The author provides a detailed discussion on limitations. I do not think there is nay negative societal impact of this work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper investigates visual search asymmetry using a computational model, which integrate both bottom-up visual recognition with target-dependent top-down cues. By comparing the model against human behavior in visual tasks, the work revealed classical perceptual properties can emerge in neural network models pretrained on ImageNet, without the need for task-specific training. The findings are scientific and useful for providing computational evidence for human cognition. All the reviewers think the paper is well-written and provides a thorough view about human and machine behavior synergy in visual search. The reviewers give extensive, detailed reviews about the manuscript and made numerous points for clarification and improvements. The authors did a good job in the rebuttal that strengthens the reviewers confidence, which led to the increase of scores. These responses should be included in the revision. There are still a few points that deserve further clarification. Reviewer CMyL is not convinced about a few model choices as detailed in the post-rebuttal review. Both Reviewer F9R3 and CJM4 want further clarification about the motivation of the work. The authors should address these points in the revision.
Decision: Accept (Poster)