A Dual-Stream Neural Network Explains the Functional Segregation of Dorsal and Ventral Visual Pathways in Human Brains
Authors: Minkyu Choi
Kuan Han
Xiaokai Wang
Yizhen Zhang
Zhongming Liu
Venue: NeurIPS 2023 poster (2023)
Keywords: brain-inspired AI
retina transformation
eye movements
deep neural networks
TL;DR: A DNN with biologically plausible features (eye movements, two-streamed CNNs, retinal transformation) explains both ventral / dorsal visual stream in human brains.

Abstract: The human visual system uses two parallel pathways for spatial processing and object recognition. In contrast, computer vision systems tend to use a single feedforward pathway, rendering them less robust, adaptive, or efficient than human vision. To bridge this gap, we developed a dual-stream vision model inspired by the human eyes and brain. At the input level, the model samples two complementary visual patterns to mimic how the human eyes use magnocellular and parvocellular retinal ganglion cells to separate retinal inputs to the brain. At the backend, the model processes the separate input patterns through two branches of convolutional neural networks (CNN) to mimic how the human brain uses the dorsal and ventral cortical pathways for parallel visual processing. The first branch (WhereCNN) samples a global view to learn spatial attention and control eye movements. The second branch (WhatCNN) samples a local view to represent the object around the fixation. Over time, the two branches interact recurrently to build a scene representation from moving fixations. We compared this model with the human brains processing the same movie and evaluated their functional alignment by linear transformation. The WhereCNN and WhatCNN branches were found to differentially match the dorsal and ventral pathways of the visual cortex, respectively, primarily due to their different learning objectives, rather than their distinctions in retinal sampling or sensitivity to attention-driven eye movements. These model-based results lead us to speculate that the distinct responses and representations of the ventral and dorsal streams are more influenced by their distinct goals in visual attention and object recognition than by their specific bias or selectivity in retinal inputs. This dual-stream model takes a further step in brain-inspired computer vision, enabling parallel neural networks to actively explore and understand the visual surroundings.

A Dual-Stream Neural Network Explains the Functional Segregation of Dorsal and Ventral Visual Pathways in Human Brains
Minkyu Choi', Kuan Han', Xiaokai Wang', Yizhen Zhang!,, and Zhongming Liu 1.2
Department of Electrical Engincering and Computer Science University of Michigan, Ann Arbor, MI 48109 2 Department of Biomedical Engineering, University of Michigan, Ann Arbor, MI 48109 Department of Neurological Surgery University of California, San Francisco, San Francisco, CA 94143 {cminkyu, kuanhan, xiaokaiv, zhyz, zml omliuhhuuichh.e
Abstract
The human visual system uses two parallel pathways for spatial processing and object recognition. In contrast, computer vision systems tend to use single feed- forward pathway, rendering them less robust, adaptive, or efficient than human vision. To bridge this gap. we developed a dual-stream vision model inspired by the human eyes and brain. the input level, the model samples two complementary visual patterns mimic how the human eyes magnocellular and parvocellular retinal ganglion cells to separate retinal inputs to the brain. At the backend, the model processes the separate input patterns through two branches of convolutional neural networks (CNN) to mimic how the human brain uses the dorsal and ventral cortical pathways for parallel visual processing. The first branch (WhereCNN) samples a global view to learn spatial attention and control eye movements. The second branch (WhatCNN) samples local view represent the object around the fixation. Over time, the two branches interact recurrently to build a scene repre- sentation from moving fixations. We compared this model with the human brains processing the same movie and evaluated their functional alignment by linear trans- formation. The WhereCNN and WhatCNN branches match the dorsal and ventral pathways of the visual cortex, respectively, primarily due to their different learning objectives, rather than their distinctions in retinal sampling or sensitivity to attention driven eye movements. These model-based results lead us to speculate that the distinct responses and representations of the ventral and dorsal streams are more influenced by their distinct goals in visual attention and object recognition than by their specific bias or selectivity retinal inputs. This dual-stream model takes further step in brain-inspired computer vision, enabling parallel neural networks to actively explore and understand the visual surroundings.
found to differentially
Introduction
The human visual system comprises two parallel and segregated streams of neural networks: the "where" stream and the "what" stream [T] The "where" stream originates from magnocellular retinal ganglion cells and extends along the dorsal visual cortex. The "what" stream originates from parvocellular retinal ganglion cells and extends along the ventral visual cortex (2). The two streams
exhibit selective responses to different aspects of visual stimuli [. The "where" stream is tuned to coarse but fast information from a wide view, while the "what" stream is selective to fine but slow
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Dorsal
Oculomotor
FEF
Magno
Fixation
LGN
v1
Paryo
Ventral
Retina
Fixation
Sampling the next fixation (x.y)
Saliency
U
Where CNN
Wide view
X2
x2
X2
X2
Resize & Combine
Narrow view
What CNN
x2
x2
GRU FC
'Fox'
Retinal transformation
Figure 1: Brain-inspired dual-stream vision model. The top illustrates the subcortical (dashed arrows) and cortical (solid arrows) pathways for parallel visual processing in the brain. Given a scene (e.g.. "two foxes on the lawn"), the retina samples incoming light relative to the fixation of the eyes (shown as the cross). Magnocellular (orange) and parvocellular (blue) retinal ganglion cells encode complementary visual information into two sets of retinal inputs relayed onto separate layers in the lateral geniculate nuclei (LGN) and further onto different neurons in the primary visual cortex (V1). Within V1, the relative ratio of magnocellular vs. parvocellular projections is higher for the periphery and lower for the fovea. Beyond VI, the magnocellular pathway continues along the dorsal visual cortex towards the intraparietal areas and further onto the frontal eye field (FEF) for oculomotor control, while the parvocellular pathway continues along the ventral visual cortex towards the inferior temporal cortex and further onto the superior temporal areas for semantic cognition. The bottom illustrates our model architecture including WhereCNN and WhatCNN. The model's s frontend mimics the human retina and generates two separate input patterns relative to the fixation. One pattern is wider but coarser while the other is narrower but finer, providing the respective inputs WhereCNN and WhatCNN. With the wide-view input, WhereCNN generates probability map of saliency from which the next fixation is sampled. With narrow-vier input, WhatCNN generates an object representation per each fixation and constructs a scene representation recurrently from multiple fixations.
information from a narrow view 22 4 The two streams are thought to serve different purposes. The "where" stream zooms out for spatial analysis [S, visual attention 6 7 R), and guiding actions [9] such as eye movements [10), while the "what" stream zooms in to recognize the object around the fixation [T. While being largely parallel, the two streams interact with each other [1). In one way of their interaction, the "where" stream decides where to look next and guides the "what" stream to focus on a salient location for visual perception. As the eyes move around the visual environment, the interaction between the "where" and "what" streams builds scene representation by accumulating object representations over time and space. This dual-stream architecture allows the brain to efficiently process visual information and support dynamic visual behaviors [3].
In contrast, computer vision systems tend to use a single stream of feedforward processing, acting as passive observers that sample visual information all at once with fixed and uniform patterns
14 15 [6]. Compared to human vision, this processing is less robust, especially given adversarial
attacks 17 18 it is less efficient since it samples visual information equally regardless of salience
or nuisance [19] less adaptive, lacking spatial attention for active sensing (20 21]. These
distinctions define major gap between human and computer vision. Many visual tasks that are

---

straightforward for humans are still challenging for machines [3] Therefore, computer vision may benefit from taking further inspiration from the brain by using a dual-stream architecture to learn adaptive and robust visual behaviors.
To gain insights into the computational mechanisms of human vision, researchers have devel- oped image-copputable models by utilizing goal-driven deep neural networks that simulate hu- man perceptual behavior. In particular, convolutional neural networks (CNNs) are leading models of visual perception, capturing the hierarchical processing by the brain's 's ventral visual stream Previous models of this nature commonly utilize CNNs trained through super- 26 30). adversarial training Bi 32). unsupervised learning [33] 34], or self-supervised leamnnng [35
24
25
26
27
28
vised learning 24
27
25
29
However, models of the dorsal stream remain relatively 40 41. Existing testing of these models has primarily focused on static images presented briefly to the fovea, thus limiting their assessment to a narrow range of visual behaviors and processes 42 A more comprehensive approach is needed to develop models that incorporate both dorsal and ventral stream processing and to assess those models against brain responses when humans engage both the dorsal and ventral streams to freely explore complex and dynamic visual environments, which may be simulated in experimental settings [43]
36
37
under-explored. despite few studies 18
39
To meet this need, we have developed a dual-stream model to mimic the parallel ventral and dorsal 3 9 The model includes two branches of convolutional neural networks: WhereCNN and WhatCNN, which share the same architecture but receive distinct visual inputs and generate different outputs. WhereCNN samples a wide view learn spatial attention and where to direct the subsequent gaze, while WhatCNN samples narrow view to learn object representations. By taking multiple gazes at a given scene, the model sequentially samples the salient locations and progressively constructs scene representation over both space and time. To evaluate this dual-stream model as a model of the human visual system, we have tested its ability to reproduce human gaze behavior and predict functional brain scans from humans watching movie with unconstrained eye movements. Our hypothesis is that the model's WhereCNN and WhatCNN branches can effectively predict the brain responses along the brain's 's dorsal and ventral visual pathways, respectively. In addition, we have also conducted experiments to evaluate the underlying factors contributing to the functional segregation of the brain's 's dorsal and ventral visual streams. Of particular interest were the relative contributions of retinal sampling, spatial attention, and attention-guic guided eye movement in shaping the function of the dorsal stream and its interplay with the ventral stream during dynamic natural vision.
streams in the human brain [ 2
2 Related Works
2.1 Dorsal-stream vision
conpttcomp models of the brain's dorsal stream have been relatively limited compared to models of the ventral stream. Previous work has attempted to model the dorsal stream by training deep neural networks to detect motion (38] or classify actions [40] using video inputs. However, these models do not fully capture the neuroscientific understanding that the dorsal stream is involved in locating objects and guiding actions, leading to its designation as the "where" or "how" visual pathway. More recent work by Mineault et al. focused training dorsal-strean model to emulate human head movements during visual exploration [39]. Additionally. Bakhtiari et al. utilized predictive learning to train parallel pathways and observed the ventral-like and dorsal-like representations an emergent consequence of structural segregation [41] However, no prior work has explored neural network models that emulate how the dorsal stream learns spatial attention and guides eye movements for visual navigation.
2.2 Spatial attention and eye movement
Prior research in the field of computer vision has attempted to train models to attend to and selectively focus on salient objects within scene [21 44 45), rather than processing the entire scene as a whole. This approach aligns with the brain's mechanism of spatial attention, where the dorsal stream acts as a global navigator, and the ventral stream functions as a local perceiver. In line with this mechanism, previous studies have employed dual-stream neural networks that process global and local features in parallel, aiming to achieve enhanced computational efficiency unified system [46] 44 47 48 49
3

---

However, these models do not fully replicate the way human eyes sample visual inputs during active exploration of the scene and thus still fall short in biological relevance.
2.3 Foveated vision and retinal transformation
The human retina functions as sophisticated camera that intelligently samples and transmits visual information. It exhibits the highest visual acuity the central region of the visual field, a phenomenon referred to as foveated vision (50 51 12 In contrast, peripheral vision uses lower spatial acuity but higher temporal sensitivity, making it better suited for detecting motion. These properties of retinal sampling are potentially useful for training neural networks to enhance performance (53) 54 55 (56] or
robustness [57 58 19], augment data or synthesize images [59]. The retina also transmits information
to the brain using distinct types of cells. The magnocellular and parvocellular retinal ganglion cells have different distributions, selectivity, and relay information through largely separate pathways. Taken together, the retina transforms visual information into segregated inputs for parallel visual processing. This biological mechanism has not been systematically investigated.
Unlike the above prior works, our work combines multiple biologically inspired mechanisms into an integral learnable model. It uses a frontend inspired by the human retina and applies complementary retinal sampling and transformation. It uses two parallel pathways inspired by the human dorsal and ventral streams. uses spatial attention and object recognition as the distinct learning objectives for training the two pathways. It further uses the attention to move fixations and thus allows the two pathways to interact for active sensing. Although these mechanisms have been explored separately in prior studies, their combination is novel and motivates work to build and test such model against the human brain and behavior in a naturalistic condition of freely watching a movie.
3 Methods
In our model, WhereCNN and WhatCNN serve distinct functions in processing objects within scene. WhereCNN identifies the spatial location of an object, determining "where" it is situated, while WhatCNN focuses on recognizing the identity of the object, determining "what" it is. When multiple objects are present in scene, WhereCNN learns spatial attention and "how" to sequentially locate and fixate on each object. This allows the model to selectively attend to different objects in sequence, mirroring the dynamic nature of human eye movements during visual exploration. Fig illustrates and describes the human visual system that inspires us to design our model.
3.1 Model design and training
Akin the human eyes 600 611, our model uses retinal transformation 162 generate separate inputs to WhereCNN and WhatCNN. For both, the retinal input consists of 64 x 64 samples non- -nifoolfrrlly distributed around the fixation. When describing point in the retinal image and its corresponding point in the visual world in terms of the radial distance and the polar angle with respect to the fixation, their polar angles the same while their radial distances related by Eq. [
1 exp((((())r) r r= r=g(r')
V 1 exp(In(a)/2)
where r' and are the radial distances in the retinal and original images, respectively. b a constant that ensures Tmax/ (max/g(r''nxd) = 1, and controls the degree of center-comcenntattion. Given larger more retinal samples are closer to the fovea relatively to the periphery. We set a = 15 for WhatCNN and 2.5 for WhereCNN. In this setting, WhereCNN is more selective to global features, while WhatCNN more selective to local features, mirroring the sampling bias magnocellular and parvocellular retinal ganglion cells, as illustrated in Fig
Both WhereCNN and WhatCNN use similar backbone architectures. The backbone consists of four blocks of convolutional layers. Each block includes two Conv2D layers (kernel size 3 x 3) followed by ReLU and BatchNorm. Applying x 2 MaxPool between adjacent blocks progressively reduces the spatial dimension. The feature dimension following each block is 64, 128, 256, or 512. Atop this backbone CNN, both WhereCNN and WhatCNN use additional components to support different goals. For WhereCNN, the feature maps from the 3rd and 4th convolutional blocks are resized to

---

16 x 16 and concatenated, providing the input to an additional convolutional block. Its output feature map is subject SoftMax generate probability map of visual saliency. By random sampling by the probability of saliency, WhereCNN decides a single location for the next fixation. To avoid
future fixations to revisit previously attended areas, inhibition of return (IOR) 6 is used. 1OR keeps records of locations of prior visits as defined in Eq 2
TOR(1) = (OR(())RLLU(I1 1 [op=
where G(1,, Σ) is a 2D Gaussian function centered l, (prior fixations) with a standard deviation o at the r-th step. Its values are normalized so that its maximum equals 1. By applying the IOR to the predicted saliency map using an element-wise multiplication, the future fixations would not revisit the areas already explored.
For WhatCNN, the output feature map from the 4th convolutional block, after global average pooling, is given as the input to an additional layer of Gated Recurrent Units (GRU) [64. which recurrently update the representation from sequence of fixations to construct cumulative representation following another fully-connected layer.
We first pre-train each stream separately and then fine-tune them together through three stages.
Stage WNe We first train WhereCNN for image recognition using ILSVRR2012 [65] for object recognition and then fine-tune it for generating the saliency map to match human attention using SALICON dataset [66]. In this stage, we use random fixations to generate the retinal inputs to WhereCNN. Adam optimizer [67 (Ir=0.002, B =0,9, 3z=0.99) is used with 25 epochs for SALICON training. At this stage, WhereCNN learns spatial attention from humans.
Stage 2- WhatCNN. We first train WhatCNN for single-object recogmition using ILSVRC2012
[65] and then fine-tune it for multi-object recognition using MSCOCO [68]. In this stage, we use the pre-trained WhereCNN to generate a sequence of eight fixations and accordingly apply the retinal transformation to generate sequence of retinal inputs to WhatCNN for recurrent object recognition. Note that the training in Stage 2 is confined to WhatCNN, while leaving WhereCNN as pre-trained in Stage Adam optimizer (Ir=0.002, 3, 0,99, 32=0.99) is used with 40 epochs for MSCOCO training.
Stage 3- ChereCNN & WhatCNN. Lastly, we equally combine the two learning objectives to train both WhereCNN and WhatCNN altogether and end-to-end using eight fixations. Adam optimizer (Ir=0.0002, 31=0.9, 3y=0.99) is used with 25 epochs for training. In this stage, SALICON, which contain labels for both salieney prediction and object recognition, is used for training. More details can be found at Appendix B
3.2 Model evaluation with human gaze behavior and fMRI responses
We use two criteria to evaluate how well model matches the brain given naturalistic and dynamic visual stimuli. First, the model should generate similar human visual behaviors, such as visual perception and gaze behavior. Second, the model's internal responses to the stimuli should predict the brain's 's responses to the same stimuli through linear projection implemented as linear encoding models 69 For our dual-stream model, we hypothesize that WhereCNN better predicts dorsal-stream voxels and WhatCNN better predicts ventral-stream voxels.
For this purpose, we use publicly available fMRI dataset from a prior study [70], in which a total of 11 human subjects (4 females) were instructed to watch the movie Raiders of the Lost Ark (115 minutes) with unconstrained eye movements. The movie was displayed on an LCD projector with a visual angle of 17 x 22.7%. Whole-brain fMRI data was acquired in a 3-T MRI system with a gradient-recalled echo planar imaging sequence (TR/TE 2.5s/35ms, flip angle 90" nominal resolution 3mm x 3mm x 3mm). We preprocess the data by using the minimal preprocessing pipeline released by the Human Connectome Project (HCP) [7T]
We test how well the model can predict the voxel-wise fMRI response to the movie stimuli through learnable linear projection of artificial units in the model. To evaluate whether and how the two branches in the model differentially predict the two streams in the brain, we define two encoding
'The code is available at ://tttp://gitubb....
5

---

models for each voxel: one based on WhereCNN and the other based on WhatCNN. We train and test the encoding models with data during different segments of the movie. To avoid overfitting, we apply dimension reduction to the internal responses in either WhereCNN or WhatCNN by applying principal component analysis (PCA) first to each layer and then to all layers while retaining 99% of the variance [30, [3] We further convolve the resulting principal components with canonical hemodynamic response function (HRF) that peaks at 5 seconds, sampe them to match the sampling rate of IMRI, generating the linear regressors used in the encoding model. Using the training data (81% % of the total data), we estimate the encoding parameters using L2-regularized least squares estimation. Using the held-out testing data (19%), we test the encoding models for their ability predicting the fMRI responses observed at each voxel and measure the accuracy of prediction as the correlation between the predicted and measured fMRI responses, denoted as Twhere and Twhat for the encoding models based on WhereCNN and WhatCNN. We test the significance of the prediction using a block permutation test [72 with a block size of 20-seconds and 100, 000 permutations and apply the false discovery rate (FDR) (p < 0.05). We further differentiate the relative roles of the brain's WhereCNN vs. WhatCNN in predicting the brain's dorsal and ventral streams for single voxels as well as regions of interest. For this, we define a relative performance (Eq 3
where
Pwhere
T where
what
In the range from 0 to Pwheer > 0.5 indicates better predictive performance by WhereCNN, while Pwkeee < 0.5 indicates better predictive performance by WhatCNN.
3.3 Alternative models and control experiments
By design, the WhereCNN and WhatCNN branches within our model exhibit two key distinctions. WhereCNN is specifically trained to learn spatial attention by utilizing wider views, while WhatCNN focuses on object recognition through the use of local views. To explore the impact of input views and learning objectives on the model's capacity to predict brain responses, introduce two modified control streams: ControlCNN-a and ControlCNN-b, designed as hybrid variants encapsulating diverse input views and learning objectives. ControlCNN-a receives narrower view as its input and is trained for saliency prediction. Conversely, ControlCNN-b is curated to accommodate a broader view and primarily focuses its training object recognition. These adjustments yield versatile examination of their respective influences and functionalities. By combining WhereCNN or WhatCNN with ControlCNN-a ControICNN-b, we create four alternative dual-stream models (illustrated in Fig 4 and examine their abilities to explain the functional segregation of the brain's 's dorsal and ventral streams.
Results
4.1 WhereCNN learns attention and WhatCNN learns perception
The WhereCNN and WhatCNN branches in our model are specifically designed to fulfill different objectives: predicting human visual saliency and recognizing visual objects, respectively. In Fig 2 we present examples comparing human attention with the model's attention based on the SALICON'S s
validation set. WhereCNN can successfully identify salient locations where humans are more likely to direct gaze. Additionally, WhereCNN can mimic human saccadic eye movements by generating sequence of fixations that navigate the model's attention to those salient locations. In contrast, WhatCNN can recognize either single or multiple objects (macro Fl score on MSCOCO's 's validation set: 61.0).
4.2 WhereCNN and WhatCNN matches dorsal and ventral visual streams
By using linear encoding models, we use the WhereCNN and WhatCNN branches to predict fMRI responses during the processing of identical movie stimuli by both the model and the brain. Together, these two branches can predict responses across wide range of cortical locations involved in visual processing. However, they exhibit distinct predictive power in relation to the dorsal and ventral streams. Generally, the WhereCNN branch exhibits superior predictive performance for the

---

WhereCNN learns human attention to mimic human gazes
Inputs
High
Human
Attention
Sallency
Model
Attention
Low
Model
Fixations
Figure 2: Saliency prediction. Given image (Ist row), WhereCNN generates saliency map (3rd row) similar to map of human attention (2nd row). Sampling this saliency map generates sequence of fixations (as red/orange/blue circles in the order of time in the 4th row) similar to human saccadic eye movements (not shown).
(a) Relative contributions of WhereCNN and WhatCNN to prediction of brain responses
Dorsal ROIs
LiPyld
MP
IPS1
VSA
VB
VEA
V2
Posterior
AIP
TPOr
PHA
AT
1,2.3
122
What CNN
Pumere
Where CNN
Lateral
Ventral ROIs
0
0.5
Lateral view
(b) Brain response predictability by WhereCNN WhatCNN
Voxel-Level
ROI-Level
Ventral
Vertral Derai
Prediction WhereCnNN
Prediction WhereCNN
0.2
ISPO
0.2
0.1
AAAPIT FFC
LO2
0.1
0.2
0.3
0.D
0.1
0.2
0.3
Prediction by WhatCNN
Prediction by WhatCNN
Figure 3: Differential encoding of the dorsal and ventral streams. (a) Relative contributions of WhereCNN and WhatCNN the prediction of the fMRI response observed at each cortical location. Color highlights the locations significantly predictable by the model (FDR=00005, block permutation test). The color itself indicates the degree by which WhereCNN is more predictive than WhatCNN (warm-tone) the opposite (cool-tone). Visual areas are delineated and labeled based on brain altas
73]. Panel (b) plots the predictive performance by WhereCNN (y-axis) against that by WhatCNN (X-axis) and shows clear separation of voxels (left panel) or ROIs (right panel) along the dorsal stream (red) vs. ventral stream (bluc) relative to the dashed line equal predictability. See Appendix A for the full ROI labels.

---

(a) Alternative inputs and objectives for WhereCNN and WhatCNN
Combination 1
Combination 2
3
Combination 4
Combination
Proppsse Modell
Logatign
Perserdion
Locattion
Percession
Location
Percaption
Lopution
Perception
Nitwern Cles
What
CMi
Chi
cln
Wide
Wide
Wide
Narrow
(c) Effect Summary
(b) Brain response predictability by WhereCNN vs. WhatCNN
Meocomn in
g.00
Combination:
3
Objectives:
Cestroccebby) y)
WhatchN 11)
Impues:
X
0 0
Figure 4: Contributing factors of the dorsal-ventral functional segmentation. (a) Alternative designs of the two-stream model for investigating the contributing factors of the functional seg- regation of two streams in the proposed model. In addition to WhereCNN and WhatCNN in the proposed model, included ControlCNN-a (narrow input field of view for predicting location) and ControlCNN-b (wide input field of view for predicting perception) for an ablation study. (b) The predictive performances and functional segregations by the two streams plotted for the dorsal (red squares) vs. ventral (blue triangles) ROIs for each of the alternative models in (a), correspondingly. The dashed diagonal lines represent equal predictive abilities of ventral and dorsal ROIs. (c) Quanti- tative evaluation of the functional segregation of the dorsal and ventral ROIs relative to the dashed line of equal predictability. Separability measures how far the predictions are away from the dashed line. Assuming the coordinates of certain ROI is (z, y), separability is calculated as the average of xx y for all ROIs
dorsal stream, while the WhatCNN branch performs better in predicting responses within the ventral stream 3. For the early visual areas (V1, V2. V3). WhereCNN better predicts the peripheral representations, while WhatCNN better predicts the foveal representations.
4.3 Factors underlying the functional segregation of the dorsal and ventral streams
We further investigate the underlying factors contributing to the model's ability to explain the func- tional segregation of the brain's 's dorsal and ventral visual streams (as depicted in 3). Specifically, we examine the input sampling pattern and output learning objective, both of which are distinct for the WhereCNN and WhatCNN branches in our dual-stream model.
To investigate the contributing factors of the functional segregation in predicting human ventral and dorsal streams, we introduce four variations of the proposed model, where the two branches either share their inputs or have the same learning objectives, and compare their abilities to account for the functional segregation of the dorsal and ventral visual streams (as shown in Fig, 4 When the two branches solely differ in their input sampling, they are unable to explain the dorsal-ventral segregation (combination 1 and 2). However, when the branches exclusively differ in their learning objectives, the functional segregation is better explained (combination 3 and 4). Moreover, when the two branches differ in both input sampling and learning objectives (combination 5), as utilized in our proposed model, the functional segregation is even more pronounced. These ablation experiments suggest that the distinct learning objectives of the brain's dorsal and ventral streams is the primary factor underlying their functional segregation.
4.4 Dual-stream: better brain model than single-stream
We also compare our dual-stream model with single-stream alternatives. One of these alternatives a baseline CNN that shares the same backbone architecture as a single branch in our dual-stream model. However, this baseline CNN is trained with original (224x224) images to recognize objects
in ImageNet (65) and MS-COCO 68 Thus, serves as direct comparison with either the WhereCNN WhatCNN branch in model. In addition, also include AlexNet [4) ResNet18,
8

---

Comparisons to feed CNNs
Response predictability ventral
Response predictability
0.3
0.3
RDI ful-eeerged R
0.2
RCK- Tok-Aurrageeh
0.2
WhatCNN
BaseCMN
ResNet118
ResNet55
ResNet18
Figure 5: WhatCNN (left) or WhereCNN (right) vs. alternative single-stream CNNs. The boxplot shows the encoding performance of different models for ventral dorsal visual areas. Each dot within the box plot signifies the average prediction accuracy r within a respective ROI in the ventral dorsal region. Asterisk (*) represents significant difference by the Wilcoxon signed-rank test. (a = 0.05).
Effect of Fixations (Learned vs Random)
Prediction By WhereCNN
Prediction By WhatCNN
VBA
UPuld
V2A
VIA
AI
TPl
S15
515
PIT 706
1.23
P11 7MA
1233
Arwhere
1,2.3
Arwhat
-0.207
0.225
Figure 6: Effects of attention-driven eye movements. The use of attention to determine fixations vs. the use of random fixations is evaluated in terms of the resulting difference in the encoding performance by WhereCNN (left) and WhatCNN (right), denoted and color-coded as Arwbere and Arwhat respectively. Voxels displayed in warm colors indicate that the predictions are more accurate with learned fixations, whereas those in cool colors signify better predictions with random fixations.
and ResNet34 [15 as additional alternatives, which have been previously evaluated in relation to brain responses 30 74. We compare these single-stream alternatives with either branch in model in terms of their ability to predict brain responses within dorsal or ventral visual areas (Fig 5 Despite their of the same backbone architecture, the baseline under-performs WhatCNN in predicting responses in ventral visual areas, and under-performs WhereCNN in predicting responses in dorsal visual areas. This result suggests that the interactive and parallel nature of the dual- model renders each stream more akin to the functioning of the human brain, surpassing the performance of isolating single stream. Moreover, WhatCNN WhereCNN also performs better than AlexNet and comparably with ResNet18 and ResNet34, which are deeper than the architecture our model.
4.5 Attention-driven eye movements improve encoding
Similar to human gaze behavior towards salient objects, our model learns spatial attention to guide fixations for parallel visual processing, In this study, we investigate whether and how the model's ability predict brain responses depends on its utilization of attention-driven fixations. To examine this, we conduct experiments where the model is allowed to use either attention-die random fixations to collect retinal samples, and we evaluate how this choice impacts the model's capability to predict brain responses by calculating Ar plavnedd random for all voxels, where learned and random are encoding performances of each voxel using learned or random fixations respectively. As depicted in Fig employing attention-driven fixations leads to higher encoding accuracy by both WhereCNN and WhatCNN compared the use of random fixations for a majority of visual cortical locations within both the dorsal and ventral streams.
fixations or

---

5 Discussion
In summary, we introduce new dual-stream neural network that incorporates the brain's mecha- nisms for parallel visual processing. The defining features of our model include 1) using retinal transformation to separate complementary inputs to each stream, 2) using different learning objec- tives to train each stream to learn either spatial attention or object recognition, and 3) controlling sequential fixations for active and interactive visual sensing and processing. We demonstrate that the combination of these features renders the model more akin to the human brain and better predictive of brain responses in humans freely engaged in naturalistic visual environments. Importantly, the two streams in our model differentially explain the two streams in the brain, contributing to the computational understanding as to how and why the brain exhibits and organizes distinct responses and processes along the structurally segregated dorsal and ventral visual pathways. Our findings suggest that the primary factor contributing to the dersal-ventral functional segregation is the different goals of the dorsal and ventral pathways. That is, the dorsal pathway learns spatial attention to control eye movements [75 7 76 777, while the ventral stream learns object recognition.
Although our model demonstrates initial steps to model parallel visual processing in the brain, it has limitations that remain be addressed in future studies. For one limitation, the model uses different spatial sampling to generate the retinal inputs to the two streams but does not consider different temporal sampling that makes the dorsal stream more sensitive to motion than the ventral stream For another limitation, the interaction between the two streams is limited to the common fixation that determines the complementary retinal input to each stream. Although attention- driven eye movement is an important aspect of human visual behavior shaping brain responses for both dorsal and ventral streams, the two streams also interact and exchange information at higher levels. The precise mechanisms for dorsal-ventral interactions remain unclear but may be important to understanding human vision or improving brain-inspired computer vision.
[38 40
39 41
Acknowledgements
This research is supported by the Collaborative Research in Computational Neuroscience (CRCNS) program from National Science Foundation (Award#:: IIS 21 2112773) and the University of Michigan.
References
[1] Mortimer Mishkin, Leslie G Ungerleider, and Kathleen A Macko. Object vision and spatial vision: two cortical pathways. Trends in neurosciences, 6:414-417, 1983.
[2] William H Merigan and John HR Maunsell. How parallel are the primate visual pathways? Anmual review of neuroscience, (66)),3694402, 1993.
[3] Jonathan J Nassi and Edward M Callaway. Parallel processing strategies of the primate visual system. Nature reviews neuroscience, 10(5):360- 372000333222200 2009.
[4] Margaret Livingstone and David Hubel. Segregation of form, color, movement, and depth: anatomy, physiology, and perception. Science 240(4853);440-749 1988.
[5] James V Haxby, Cheryl L Grady, Barry Horwitz, Leslie G Ungerleider, Mortimer Mishkín, Richard E Carson, Peter Herscovitch, Mark B Schapiro, and Stanley I Rapoport. Dissociation of object and spatial visual processing pathways in human extrastriate cortex. Proceedings of the National Academy of Sciences, 88(5): 1621-1155,, 1991.
[6] Giacomo Rizzolatti and Massimo Matelli. Two different streams form the dorsal visual system: anatomy and functions. Experimental brain research, 153: 14::157,, 2003.
[7] Maurizio Corbetta and Gordon L Shulman. Control of goal-directed and stimulus-driven attention in the brain. Nature reviews neuroscience. 3(3):201-215, 2002.
[8] Gustavo Deco and Edmund T Rolls. A neurodynamical cortical model of visual attention and invariant object recognition. Vision research, 44(6):621-642 2004.
99] Melvyn Goodale and A David Milner. Separate visual pathways for perception and action. Trends in neurosciences, 15(1):20-25, 1992.
10

---

[10] Carol L Colby and Michael E Goldberg. Space and attention in parietal cortex. Annual review of neuroscience, 22(1):319-349) 1999.
[11] Keiji Tanaka. Inferotemporal cortex and object vision. Annual review of neuroscience, 19(1):109- 139, 1996.
[12] David Milner. How do the two visual streams interact with each other? Experimental brain research, 235(5):1277-1308 2017.
[13] Laurent ltti and Christof Koch. saliency- based search mechanism for overt and covert shifts of visual attention. Vision research, 40(10- 100(0-122) 12): 119-489900 2000.
[14] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84-90, 2017.
[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[16] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 2020.
[17] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv: 131.619,,,, 2013.
[18] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversar- ial examples. arXiv preprint #xXiv:1412.6572 2014.
[19] Minkyu Choi, Yizhen Zhang. Kuan Han, Xiaokai Wang, and Zhongming Liu. Human eyes inspired recurrent neural networks are more robust against adversarial noises. arXiv preprint WxXiv:2206007822. 2022.
[20] Laurent Itti, Christof Koch, and Ernst Niebur. A model of saliency-based visual attention for rapid scene analysis. IEEE Transactions on pattern analysis and machine intelligence, 20(11):1254-1259, 1998.
[21] Volodymyr Mnih, Nicolas Heess, Alex Graves, al. Recurrent models of visual attention. Advances in neural information processing systems, 27, 2014.
[22] Brenden M Lake, Tomer D) Ullman, Joshua B Tenenbaum, and Samuel J Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40:e253, 2017.
[23] Ernest Davis and Gary Marcus. Commonsense reasoning and commonsense knowledge in artificial intelligence. Communications of the ACM, 589)):22-03,, 2015.
[24] Daniel LK Yamins, Ha Hong, Charles F Cadieu, Ethan A Solomon, Darren Seibert, and James J DiCarlo. cerformance-optimizze hierarchical models predict neural responses higher visual cortex. Proceedings of the national academy of sciences, 11 111(23:8611-86224 2014.
[25] Umut Güşlü and Marcel AJ van Gerven. Deep neural networks reveal gradient in the complexity of neural representations across the ventral stream. Journal of Neuroscience, 35(27):10005-10014 2015.
[26] Michael Eickenberg, Alexandre Gramfort, Gaöl Varoquaux, and Bertrand Thirion. Seeing all: Convolutional network layers map the function of the human visual system. Neurolmage, 152:184-194, 2017.
[27] Seyed-Mahdi Chaligh-Razav and Nikolaus Kriegeskorte. supervised, models may explain cortical representation. PLOS computational biology, 10(11)c1003315, 2014.
Deep supervised, but not un-

---

[28] Daniel LK Yamins and James DiCarlo. Using goal-driven deep learning models to understand sensory cortex. Nature neuroscience, 19(3):35--365 2016.
[29] Radoslaw Martin Cichy, Aditya Khosla, Dimitrios Pantazis, Antonio Torralba, and Aude Oliva. Comparison of deep neural networks to spatio-temporal cortical dynamies of human visual object recognition reveals hierarchical correspondence. Scientific reports, 6(1):27755, 2016.
[30] Haiguang Wen, Junxing Shi, Yizhen Zhang, Kun-Han Lu, Jiayue Cao, and Zhongming Liu. Neural encoding and decoding with deep learning for dynamic natural vision. Cerebral cortex, 28(12):4133) (3(12):4136-4161) 13134660000
2018.
[31] Joel Dapello, Tiago Marques, Martin Schrimpf, Franziska Geiger, David Cox, and James J DiCarlo. Simulating a primary visual cortex at the front of cnns improves robustness to image perturbations. Advances in Neural Information Processing Systems, 13:1333--3087 2020.
[32] William Berrios and Arturo Deza. Joint rotational invariance and adversarial training of dual-stream transformer yields state of the art brain-score for area v4. arXiv preprint wrXiv2223.066-44 2022.
[33] Kuan Han, Haiguang Wen, Junxing Shi, Kun-Han Lu, Yizhen Zhang, Di Fu, and Zhongming Liu. Variational autoencoder: An unsupervised model for encoding and decoding fimr activity in visual cortex. Neurolmage, 198: 198:125-14 1336, 2019.
[34] Minkyu Choi and Jun Tani. Predictive coding for dynamic visual processing: development of functional hierarchy in multiple spatiotemporal scales model. Neural computation, 30(1):237-273) 2018.
[35] Chengxu Zhuang, Siming Yan, Aran Nayebi, Martin Schrimpf, Michael C Frank, James J DiCarlo, and Daniel LK Yamins. Unsupervised neural network models the ventral visual stream. Proceedings of the National Academy of Sciences, 118(3):220111) 41961111119188 18,118, 2021.
[36] Aria Yuan Wang, Kendrick Kay, Thomas Naselaris, Michael Tarr, and Leila Wehbe. Incor- porating natural language into vision models improves prediction and understanding of higher visual cortex. BioRxiv, pages 2022-09,, 2022.
[37] Talia Konkle and George Alvarez. Deepnets do not need category supervision to predict visual system responses to objects. Journal of Vision, (0(11))498449) 2020.
[38] Reuben Rideaux and Andrew E Welchman. But still it moves: static image statistics underlie how we see motion. Journal of Neuroscience, 40( 40(12):2538-2552 2020.
[39] Patrick Mineault, Shahab Bakhtiari, Blake Richards, and Christopher Pack. Your head is there to move you around: Goal-driven models of the primate dorsal pathway. Advances in Neural Information Processing Systems, 34:28757-28771, 2021.
[40] Umut Güçlü and Marcel AJ van Gerven. Increasingly complex representations of natural movies across the dorsal stream are shared between subjects. Neurolmage, 145:329-336, 2017.
[41] Shahab Bakhtiari, Patrick Mineault, Timothy Lillicrap. Christopher Pack, and Blake Richards. The functional specialization of visual cortex emerges from training parallel pathways with self-supervised predictive learning. Advances in Neural Information Processing Systems, 34:25164 25148. 2021.
[42] Thomas Serre. Deep learning: the good, the bad, and the ugly. Annual review of vision science, 4:39 2019.
[43] Hyun-Chul Kim, Sangsoo Jin, Sungman Jo, and Jong-Hwan Lee. A naturalistic viewing paradigm using 360 panoramic video clips and real-time field-of-view changes with eye-gaze tracking. Neurolmage, 216:1 16617, 2020.
[44] Pierre Sermanet, Andrea Frome, and Esteban Real. Attention for fine-grained categorization. arXiv preprint arXiv:1412.7054 2014.
12

---

[45] Jianlong Fu, Heliang Zheng, and Tao Mei. Look closer to see better: Recurrent attention convolutional neural network for fine- fine-grained image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages
2017.
[46] Carlos Esteves, Christine Blen-Blaacchette, Xiaowei Zhou, and Kostas Daniilidis. Polar transformer networks. arXiv preprint arXiv:1709.01889 2017.
[47] Yulin Wang, Kangchen Lv, Rui Huang, Shiji Song, Le Yang, and Gao Huang. Glance and focus: a dynamic approach to reducing spatial redundancy in image classification. Advances in Neural Information Processing Systems, 33:2432 2222222-444, 2020.
[48] Yiyou Guo, Jinsheng Ji, Xiankai Lu, Hong Huo, Tao Fang, and Deren Li. Global-local attention network for aerial scene classification. IEEE Access, 1667000-67212,200
[49] Kevin Wu, Eric Wu, and Gabriel Kreiman. Learning scene gist with convolutional neural networks to improve object recognition. In 2018 52nd Annual Conference on Information Sciences and Systems (CISS), pages I 1-6. IEEE, 2018.
[50] AM Derrington and P Lennie. Spatial and temporal contrast sensitivities of neurones in lateral geniculate nucleus of macaque. The Journal of physiology, 357(1):219-240, 1984.
[51] Michael Connolly and David Van Essen. The representation of the visual field in parvicellular and magnocellular layers of the lateral geniculate nucleus in the macaque monkey. Journal of Comparative Neurology, 226(4):544 564, 1984.
[52] Christine Curcio, Kenneth Sloan, Robert E. Kalina, and Anita Hendrickson. Human photoreceptor topography. Journal of comparative neurology, 292(4):497-523, 1990.
[53] Juhong Min, Yucheng Zhao, Chong Luo, and Minsu Cho. Peripheral vision transformer. arXiv preprint arXiv:2206. 06801, 2022.
[54] Chittesh Thavamani, Mengtian Li, Nicolas Cebron, and Deva Ramanan. Fovea: Foveated image magnification for autonomous navigation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 15539-15548, 2021.
[55] Aditya Jonnalagadda, William Yang Wang, BS Manjunath, and Miguel P Eckstein. Foveater: Foveated transformer for image classification. arXiv preprint arXiv:2 122.1055.4422 /4/73, 2021.
(56) Emre Akbas and Miguel P Eckstein. Object detection through search with a foveated visual system. PLoS computational biology, 13(10)e 1005743, 2017.
[57] Anne Harrington and Arturo Deza. Finding biological plausibility for adversarially robust features via metameric tasks. In SVRIMM 2021 Workshop@ @ NeurIPS, 2021.
[58) Manish Reddy Vuyyuru, Andrzej Banburski, Nishka Pant, and Tomaso Poggio. Biologically inspired mechanisms for adversarial robustness. Advances Neural Information Processing Systems, 33:2135-211 224611166666...11 2020.
[59] Binxu Wang, David Mayo, Arturo Deza, Andrei Barbu, and Colin Conwell. On the use of cortical magnification and saccades as biological proxies for data augmentation. arXiv preprint 12Xiv:21 2021.
[60] Alyssa A Brewer, William A Press, Nikos K Logothetis, and Brian A Wandell. Visual ar- eas in macaque cortex measured using functional magnetic resonance imaging. Journal of Neuroscience, 22(23): 1041100111122 2002.
[61] Ricardo Gattass and Charles G Gross, Visual topography of striate projection zone (mt) in posterior superior temporal sulcus of the macaque. Journal of neurophysiology. 46(3):621-638, 1981.
[62] Pouya Bashivan, Kohitij Kar, and James J DiCarlo. Neural population control via deep image synthesis. Science, 36+(6449): 2019.
[63] Laurent Itti and Christof Koch. Computational modelling of visual attention. Nature reviews neuroscience, 194-:194-003, 2001.
13

---

[64] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.355., 2014.
[65] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211-253 2015.
[66] Ming Jiang, Shengsheng Huang, Juanyong Duan, and Qi Zhao. Salicon: Saliency in context. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1072-1080, 2015.
[67] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6988) 2014.
[68] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzelannd, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[69] Thomas Naselaris, Kendrick Kay, Shinji Nishimoto, and Jack Gallant. Encoding and decoding in fmri. Neuriiaaee, 56(2):400 2011.
[70] James V Haxby, J Swaroop Guntupalli, Andrew C Connolly, Yaroslav O Halchenko, Bryan R Conroy, M Ida Gobbini, Michael Hanke, and Peter J Ramadge. A common, high-dimensional model of the representational space human ventral temporal cortex. Neuron, 72(2):404-416. 2011.
[71] Matthew F Glasser, Stamatios N Sotiropoulos, J Anthony Wilson, Timothy S Coalson, Bruce Fischl, Jesper Andersson, Junqian Xu, Saad Jbabdi, Matthew Webster, Jonathan Polimeni, et al. The minimal preprocessing pipelines for the human connectome project. Neuroimage, 80:105-124, 2013.
[72] Daniela Adolf, Snezhana Weston, Sebastian Baecke, Michael Luchtmann, Johannes Bernarding, and Siegfried Kropf. Increasing the reliability of data analysis of functional magnetic resonance imaging by applying a new blockwise permutation method. Frontiers in neuroinformatiis, 8:72, 2014.
[73] Matthew F Glasser, Timothy S Coalson, Emma C Robinson, Carl D Hacker, John Harwell, Essa Yacoub, Kamil Ugurbil, Jesper Andersson, Christian F Beckmann, Mark Jenkinson, et al. A multi-modal parcellation human cerebral cortex. Nature, 536(7615):111-17) 2016.
[74] Haiguang Wen, Junxing Shi, Wei Chen, and Zhongming Liu. Deep residual network predicts cortical representation and organization of visual features for rapid categorization. Scientific reports, 8(1):3752, 2018.
[75] James Bisley and Michael Goldberg, Attention, intention, and priority the parietal lobe. Annual review of neuroscience, 33:1 2010.
[76] Steven Yantis and John T Serences. Cortical mechanisms of basee and object-based attentional control. Current opinion in neurobiolgyy, 13(2):187-193, 2003.
[77] John HR Maunsell and Stefan Treue Feature-based attention in visual cortex. Trends in neurosciences. 296633117332,, 2006.
14

---

Appendix: A Dual-Stream Neural Network Explains the Functional Segregation of Dorsal and Ventral Visual Pathways in Human Brains
Minkyu Choi', Kuan Han', Xiaokai Wang', Yizhen Zhang!,, and Zhongming Liu 1.2
Department of Electrical Engineering and Computer Science, University Michigan, Ann Arhor, 48109 Department of Biomedical Engineering, University of Michigan, Ann Arhor, MI 48109 Department of Neurological Surgery University of California. San Francisco, San Francisco, CA 94143 {cminkyu, kuanhan, xiaokaiv, zhyz, zmliu omliu)@umich.edi edu
A Regions of Interests
In our study, we delineated our regions of interest (ROIs) into two primary segments: 1) the ventral visual stream and object recognition-related regions and 2) the dorsal visual stream and overt attention- related regions. This approach followed the parcellations proposed by [I] For the dorsal visual stream, the ROIs includes V3A, V3B, V6, V6A, and V7. Within the parietal cortex, visuo-spatial information and overt attention are processed by the intraparietal suleus (IPS) and the superior parietal lobule (SPL) [2] 4 5, [6. The IPS encompasses V7, IPS1, IPO, IPI, and IP2; whereas the SPL consists of lateral intraparictal cortex (LIPv, LIPd), ventral intraparictal complex (VIP), anterior intraparietal (AIP), medial intraparictal area (MIP), 7PC, 7AL, 7Am, 7PL, and 7Pm. We also included the frontal eye field (FEF), which is acknowledged for controlling eye movements 7 contrast, the ROIs associated with object recognition and the ventral visual stream encompassed V8, the posterior inferotemporal (PIT) complex, the fusiform face complex (FFC), and ventromedial visual (VMV) areas 1, 2. 3, along with the lateral occipital area (LO). In addition, we included the superior temporal suleus (STS), which is recognized for processing multimodal signals, including auditory and visual cues [1] 12 13. Fig. [ST displays the full set region labels, corresponding to
10 In
3().3(a) from the main text. Among the parcellations by M regions including significantly predicted voxels either by the WhereCNN or WhatCNN are presented in Fig. SI
Region Labels
VIP
AIP
UPvdd VIP
UPvid
IP2 100/1
IPO/1
UPuld
LIPWd AiP
TPOJ
TPO
VMV
STS
PHA 1,2.3
PIT
Lateral view:
Lateral view
Figure S1: Region labels. Regions including significant voxels from Fig.3(a) in the main text are presented.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

B Training Details
The backbone Convolutional Neural Networks (CNNs) of both the WhereCNN and WhatCNN share the same architecture, consisting of four blocks of convolutional operations. Situated atop the backbone CNN, the WhereCNN and WhatCNN possess additional layers tailored to their specific objectives: the WhereCNN features two convolutional layers that produce 2D saliency maps, whereas the WhatCNN includes Gated Recurrent Unit (GRU) layer followed by fully connected layer for object classification.
During the pre-training of the backbone CNN, global average pooling and fully connected layer are integrated atop the backbone CNN, serving as a classifier. Upon completion of the pre-training process, the classifier is detached, allowing the pre-trained backbone CNN to be incorporated as component of the WhereCNN or WhatCNN.
As detailed in Section 3.1 of the main text, our model underwent three-stage training process. In this section, we will elaborate on the specifics of the pre-training phase.
Stage CNereCNN The backbone architecture of the WhereCNN was pre-trained on ILSVRC2012 [14] for image classification task over 120 epochs. A batch size of 1, 024 was employed, along with the Adam optimizer [15] (Ir=0.001, 3,=0,9, 82=0.99). During pre-training, fixations for the retinal transformation were randomly generated across the image area. Once the backbone architecture had been pre-trained, we detached the classifier and initialized the WhereCNN using the model parameters obtained from the pre-training stage. We then performed SALICON training, as described in Section 3.1 of the main text.
Stage 2- WhatCNN In a process mirrrring Stage 1, the backbone of the WhatCNN was also pre-trained on VRC2222 [14] for an image classification task over 120 epochs, utilizing random fixations and the Adam optimizer (Ir=0.001, 81 =0,9, (2=0.99). After pre-training the backbone CNN, we initialized the WhatCNN using the weights of the pre-trained backbone CNN.
Subsequently, the WhatCNN, initialized with the pre-trained weights whole, was trained on ILSVRC2012 [4] for object recognition using four fixations. Four randomly generated fixations were employed for training the WhatCNN for 55 epochs, again utilizing the Adam optimizer (Ir=0.001 000,, 31-0.9, 3,=0.9 After this stage, we conducted a fine-tune fine-tuning process using the learned fixations from the WhereCNN. In this stage, the WhereCNN, after the pre-training in Stage 1, was incor- porated to guide the WhatCNN's ' fixations. However, only the WhaCNNN was optimized. while the WhereCNN remained unchanged. This fine-tuning with learned fixations deployed four gazes, utilizing the Adam optimizer (Ir=0 $1=0.9, 32=0.99) over 25 epochs. Finally, the WhatCNN underwent further training on MSCOCO, as described in Section 3.1 of the main text.
Stage3 - WhereCNN & WhatCNN During this stage, both WhereCNN and WhatCNN, trained in the previous stages, were used to initialize model weights, followed by further end-to-end training, leveraging the stream-specific objectives (object recognition and saliency prediction, respectively). As the training requires labels for both tasks, the model was trained using images in the SALICON dataset, which contain labels for both saliency prediction and object recognition.
The model samples fixations from the predicted saliency maps from WhereCNN. As this sampling process is non-differentiable, the gradients from object recognition cannot optimize the weights of WhereCNN. To tackle this issue, we utilized REINFORCE [16] to approximate the gradient for WhereCNN. At the time f, a fixation L, is generated by WhereCNN, based on which WhatCNN predicts class prediction Pt- Then, in the context of REINFORCE, the reward r, of choosing L, as the fixation is calculated as the reduced classification loss relative to the previous time step re C'((i-1. label.) - Cp(,, label.), where CE is the cross-entropy loss, label, is class labels.
The goal of REINFORCE is to maximize the discounted sum of rewards, R ΣT ΣT Tr, where E (0, 1) is the discount factor and set as 0.8.
"Irr,
In this stage, we strived to minimize the object recognition and saliency prediction losses while maximizing the discounted sum of rewards. As indicated in Section 3.1 of the main text, we utilized the Adam optimizer (Ir=0.0002, 009,, 32=0.9 for 25 epochs for this training stage.
For All Stages All training stages were conducted using four NVIDIA A40 GPUs. All codes are written in Pytorch 1.9.1.
16

---

C Saliency Maps and Inhibition of Returns
Once the saliency maps were generated by WhereCNN, inhibition of return (IOR) was used prohibit future fixations to re-visit image areas that had been already explored. This process is illustrated in Fig. S2
Saliency Map
WhereCNN
Saliency Map with IOR
X
Current Fiaaion Point
IOR
Sample Next Fixation Point
X- Next Fisetion Point
Figure S2: Process of determining the next fixation point given the current fixation. A saliency map. generated by WhereCNN, is multiplied element-wise (indicated by *) with the inhibition return (IOR) to prevent future fixations from reverting previous positions. In the IOR, white and black colors correspond values of and 0, respectively.
In the process of determining the next fixation, the WhereCNN generate saliency map based on the current fixation. The location of this subsequent fixation is guided by the saliency map's probabilistic distribution. However, it's important note that if the current fixation point possesses high probability, subsequent fixations are likely to occur in proximity to the present fixation.
To ensure more dynamic and comprehensive exploration of the visual field, employed the principle of Inhibition of Return (IOR), detailed in Eq.2 of the main text, and presented again here in Eq 4
IOR(t) = ReLU Σ -(u=,,,-= r=1
where G(j,, Σ) is 2D Gaussian function centered at L, (prior fixations) with a standard deviation o the 7-th step. The Inhibition of Return (IOR) is initially created at resolution of 224 x 224 with 25, and subsequently resized to align with the dimensions of the saliency map. IOR serves to decrease the saliency of previously attended areas, thereby preventing the model from repetitively focusing these regions. This mechanism is informed by the model's all prior fixation history. The IOR map is designed such that assigns lower values (approaching 0.0) in the vicinity of prior fixation points, and higher values (up to 1.0) in regions further away. Thus, when the IOR map is element- multiplied with the saliency map. it effectively reduces the saliency values in areas already explored.
Following the application of IOR, the subsequent fixation point is decided upon by considering the adjusted saliency map. It then chosen based on the probabilistic distribution within this updated map. This strategy encourages diverse fixations and facilitates broader and more comprehensive understanding of the scene.
D WhereCNN's Saliency Maps and Fixation Points
The original images are presented in Cartesian coordinates. Once the retinal transformation is applied to these images, the resultant retinal images adopt retinal coordinates, as detailed in Eq. of the main
17

---

text. Since the inputs to the WhereCNN operate in retinal coordinates, it naturally follows that the output saliency maps mirror this coordinate system. To visualize these within this paper, we utilize the inverse function of Eq thereby transforming the saliency maps from retinal back to Cartesian coordinates.
In preparation for our model's processing of the movie Raiders of the Lost Ark, we reduce the frame rate 6 frames per second (fps). This adjustment helps mitigate computational and memory costs associated with the handling of the extracted features. As the model engages with the movie, solitary fixation point is established for each frame. Importantly, the Inhibition Return (IOR) mechanism is not invoked during the model's 's interaction with the movie. Fig. S3 showcases saliency maps and fixation points derived from segments of the movie Raiders of the Lost Ark. Frames situated on the same horizontal axis are selected at rate of 1 fps.
Inputs
Saliency
Maps
Fixations
Inputs
Saliency Maps
Inputs
Saliency Maps
Fixations
Inputs
Saliency Maps
Fixations
18

---

Inputs
Saliency Maps
Focatoons
Figure S3: Given the movie frames (Ist row), the WhereCNN generates saliency maps (2nd row) and fixations (3rd row). The red marker in the 3rd presents the fixation point.
E
Investigating Layer-wise Correspondence to Visual Cortex
In the main text, the whole features from the all layers of each stream used for predicting voxel activities (noted as Stream- wise encoding). In an alternative way, the features from each layer can be used to predict voxel activities, instead of concatenating all the layers, (noted as Layer-wist wise encoding). In this way, the hierarchical correspondence between each layer the model to the ROIs of the visual system be observed.
With the layer-wise encoding scheme, we predicted fMRI responses using features from each layer in the WhereCNN and WhatCNN. Fig. SS4 associates each voxel to one (color-coded) layer most predictive that voxel for either (a) WhatCNN or (b) WhereCNN. Fig. S4 (a) shows that the lower layers of WhatCNN better predict earlier visual areas such as V1/V2, whereas the higher layers of WhatCNN better predict higher-order visual areas such as LO and PIT, consistent with prior studies The results with the WhereCNN show different patterns, shown in Fig. (b). Within early visual areas, the lower layers of WhereCNN better predict foveal representations, whereas the higher layers better predict peripheral representations.
[1. I8]
Layer-wise Voxel Predictions
(a) WhatCNN
Visual Cortex Hierarchy
(b) WhereCNN
Higher
Earlier
Earlier
Higher
Peripheral Visual
Central Visian
Periphoral
Vision
1st 2nd
3rd
4th
5th
1st
2nd 3rd
4th
5th.
WhatCNN
Layer WhereCNN
Figure $4: Each voxel is predicted by the features from a single layer from (a) WhatCNN and (b) WhereCNN. Layer indexes are coder so that the layer best predicting each voxel presented.
Implications to the Computer Visions
In the current study, demonstrated that the biologically plausible components (two stream, retinal sampling and eye movements) can be used to build a better model for the human visual cortex in a naturalistic viewing condition. At the same time, those components considered in this study may also bring benefits to the computer vision applications.
1) Efficiency. Unlike conventional CNNs that process entire images, our dual-stream model allows serial processing. It concentrates processing power on key image regions through attention directed fixations. This serial processing may significantly lower memory and computational overhead,
19

---

because resources are allocated only to the crucial image regions. It is plausible that such efficiency underpins the brain's adoption of dual stream processing due biological constraints on energy use.
2) Adaptability. The dual streams of our model offer complementary lenses for visual exploration and perception in real-world environments. One stream provides broad yet rough overview of the environment. The other gathers detailed observations with precision. Their synergistic interaction may facilitate adaptive behaviors for tasks like visual search, object detection in complex and cluttered scenes. Moreover, the distinct functions of each of the parallel streams present a combinatorial flexibility when leveraged together, potentially enhancing the model's ' overall capability to adapt to diverse visual challenges, including potential applications in robotics.
However, leveraging such potential benefits within the scope of current study face challenges. First, mainstream datasets like ImageNet and MS-COCO offer a narrow view and lack the high-resolution detail our model thrives on. Moreover, these datasets often focus on large, central objects, limiting our model's ss adaptability that benefits object recognition. better benchmark to our model would be high-resolution panoramic images synthetic virtual reality environments to accommodate unlimited fixation variances. In such settings, the efficiency and adaptability of our model should be more appealing.
References
[1] Matthew F Glasser, Timothy S Coalson, Emma C Robinson, Carl D Hacker, John Harwell, Essa Yacoub, Kamil Ugurbil, Jesper Andersson, Christian F Beckmann, Mark Jenkinson, et al. A multi-modal parcellation human cerebral cortex. Nature, 536(7615):111-178, 2016.
[2] Jacqueline P Gottlieb, Makoto Kusunoki, and Michael E Goldberg. The representation of visual salience in monkey parietal cortex. Nature, (6666)6)) 1998.
[3] Anna E Ipata, Angela L Gee, Jacqueline Gottlieb, James W Bisley, and Michael E Goldberg. Lip responses to popout stimulus are reduced if it is overtly ignored. Nature neuroscience, 1(7):007 2006.
[4] Makoto Kusunoki, Jacqueline Gottlieb, and Michael Goldberg. The lateral intraparietal area as a salience map: the representation of abrupt onset, stimulus motion, and task relevance. Vision research, 40(100-1)) 1459-1444414 2000.
[5] James W Bisley, Koorosh Mirpour, Fabrice Arcizet, and Wei S Ong. The role of the lateral intraparictal area in orienting attention and its implications for visual search. European Journal of Neuroscience, 33(11):1982-199 2011.
[6] James Bisley and Michael Goldberg, Attention, intention, and priority the parietal lobe. Annual review of neuroscience, 33:1-21, 2010.
[7] Hugo L. Fernandes, Ian H Stevenson, Adam N Phillips, Mark Segraves, and Konrad Kording. Saliency and saccade encoding in the frontal eye field during natural scene search. Cerebral Cortex, 24(12):3232-3245 2014.
[8] Kirk G Thompson and Narcisse P Bichot. A visual salience map in the primate frontal eye field. Progress in brain research, 147:249-262. 262, 2005.
[9] Charles J Bruce, Michael E Goldberg, M Catherine Bushnell, and Gregory B Stanton. Pri- mate frontal eye fields. ii physiological and anatomical correlates of electrically evoked eye movements. Journal of neurophysiology, 54(3):714-734 1985.
[10] David Robinson and Albert F Fuchs. Eye movements evoked by stimulation of frontal eye fields. Journal of neurophysiology, 32(5):637-648, 1969.
[11] Jon Driver and Toemme Noesselt. Multisensory interplay reveals crossmodal influences on 'spnsory-specifie' brain regions, neural responses, and judgments. Newron, 57(1):11-23, 2008. 2008.
[12] Gemma Calvert. Crossmodal processing in the human brain: insights from functional neuroimaging studies. Cerebral cortex, 11(12):1110-1123 2001.
20

---

[13] Michael S Beauchamp. Statistical criteria in fmrr studies of multisensory integration. Neuroin- formatics, 3:93-113, 2005.
[14] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211-2555 2015.
[15] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXtv:1412.6980, 2014.
[16] Ronald J Williams. Simple statistical gralient-following algorithms for connectionist reinforce- ment learning. Machine learning, 8(3-4):229-256, 1992.
[17] Santiago A Cadena, George H Denfield, Edgar Y Walker, Leon A Gatys, Andreas S Tolias, Matthias Bethge, and Alexander S Ecker. Deep convolutional models improve predictions of macaque v1 responses to natural images. PLoS computational biology, 1006897, 2019.
[18] Haiguang Wen, Junxing Shi, Yizhen Zhang, Kun-Han Lu, Jiayue Cao, and Zhongming Liu. Neural encoding and decoding with deep learning for dynamic natural vision. Cerebral cortex, 2018.
21


Reviews:
[{"review_id": "IdAs3C2Sbk", "reviewer": "Reviewer_ZoBi", "summary": "The author introduces a dual-path parallel neural network designed to simulate the hypothesized functions of the ventral and dorsal streams in the human visual system. This network consists of a WhereCNN, which has a wide view to learn spatial attention and direct gaze, and a WhatCNN, which has a local view to learn object recognition. The network is trained using two learning objectives: the WhereCNN is trained with human gaze attention data, while the WhatCNN is trained for object recognition based on local views. The study reveals that the WhereCNN demonstrates better prediction of the dorsal stream voxels, while the WhatCNN performs better in predicting the ventral stream voxels.", "questions": "What new insights that this work provides for our understanding of the visual system? \nThe ideas are not exactly novel, though it is nicely put together, and the alignment with fMRI data is encouraging. \nAt the end of the day, what does this work teach us about the visual system that we don't already know -- good is supposed to illuminate and elucidate something we don't fully understand.\n", "limitations": "The paper discusses the limitations of the model and highlights areas for future research.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The alignment of the WhereCNN and WhatCNN with the dorsal and ventral streams provides promising evidence supporting their assumed functions.", "weaknesses": "Although the individual networks (WhereCNN and WhatCNN) have been previously proposed and explored for the dorsal and ventral streams, this study is the first to combine them. The paper's assumption of the dorsal stream function may be narrow, as the dorsal stream is known to process spatial information, motion, and action rather than solely focusing on spatial attention and eye fixation decisions.\n\nGiven the specific training methodology used for the WhereCNN and WhatCNN, it is not particularly surprising that they show better prediction results for the dorsal and ventral streams, as suggested by earlier studies. Therefore, the results, while valuable in establishing them within a coherent and integrated system, may not be particularly surprising.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your summary and constructive feedback. \n\n***\n1\\. __Novelty__. While we concur that the ventral and dorsal visual streams have been individually studied before, our work is novel in its integrative approach. We model and link the ventral and dorsal streams as two integral parts of a whole system and incorporate a retina-like model to account for the different inputs to the two streams. Our approach is inspired by the neuroscience knowledge that parallel visual processing begins at the retina and progresses through cortical pathways. This holistic view uncovers how retinal inputs (both global and local) alongside behavioral outputs (perception and attention) influence the internal computations and representations emerging from neural processes. While it has limitations, our model is a valuable starting point for modeling parallel visual processing, aiming to deepen our understanding of the functional segregation and integration of the human visual system. \n\n***\n2\\. __Assumptions on the dorsal stream.__ We recognize the various roles attributed to the dorsal stream, including spatial attention, eye movement, motion processing, recognition, action and more. Despite this diversity, the core functionalities of the dorsal stream that shape its internal representations in distinction from the ventral stream remain a topic of debate. Historically, the understanding of the dorsal stream has shifted between the “where” pathway and the “how” pathway. With this study, we propose that spatial attention and eye movement might represent the core functionalities of the dorsal stream due to their evident correlations with human visual behavior and the relevance to both spatial analysis (“where”) and action-oriented tasks (“how”). Our focus on spatial attention and eye movement is also motivated by the dataset we use - free viewing of natural videos. Given this dataset, evaluating other potential functions would be cumbersome. However, our study does not downplay the potential relevance of other plausible dorsal-stream functions. In future studies, we plan to delve deeper into other functional aspects towards a holistic understanding of the dorsal stream and its interaction with the ventral stream. \n\n***\n3\\. __Significance.__ Our primary aim is not necessarily the proposition of a groundbreaking theory in neuroscience. Our aim is to demonstrate that a novel integration of interacting models of the eyes, dorsal and ventral streams can explain brain activity during dynamic naturalistic vision at various levels from cortical voxels to regions and streams, extending prior studies which often evaluate different regions in isolation using simple visual tasks. Uniquely, our models are brain inspired and hypothesis-driven, bridging the gap between retinal inputs and behavioral outputs through parallel and hierarchical neural information processing. Our findings both corroborate and falsify existing hypotheses. Notably, we have shown that the distinct retinal inputs to the dorsal and ventral pathways are not sufficient to differentiate their responses during movie viewing. Instead, the distinct learning objectives - namely perception vs. attention - are more significant factors for the functional segregation between these streams. Although our findings may affirm some prior knowledge while challenging others, they are not necessarily predictable from existing literature. Given our model’s adaptability to diverse architectures and objectives, we are confident of its potential to catalyze future research using parallel neural networks. \n\nIn terms of modeling the dorsal visual stream using deep neural networks, four prior studies stand out as most relevant to our study. Specifically, the studies by [1,2] focus on training a deep neural network for motion detection or action recognition, while the research by [3] emphasizes predicting self-motion in a 3D environment. The work by [4] demonstrates that employing predictive learning with two distinct streams can lead to representations reminiscent of the dorsal and ventral streams. Distinct from these and other prior studies, to the best of our knowledge, our study is unique for its focus on overt attention as a core functionality of the dorsal stream.\n\n***\n__References__\n\n[1] Reuben Rideaux and Andrew E Welchman. But still it moves: static image statistics underlie how we see motion. Journal of Neuroscience, 40(12):2538–2552, 2020.\n\n[2] Umut Güçlü and Marcel AJ van Gerven. Increasingly complex representations of natural movies across the dorsal stream are shared between subjects. NeuroImage, 145:329–336, 2017.\n\n[3] Patrick Mineault, Shahab Bakhtiari, Blake Richards, and Christopher Pack. Your head is there to move you around: Goal-driven models of the primate dorsal pathway. Advances in Neural Information Processing Systems, 34:28757–28771, 2021.\n\n[4] Shahab Bakhtiari, Patrick Mineault, Timothy Lillicrap, Christopher Pack, and Blake Richards. The functional specialization of visual cortex emerges from training parallel pathways with self-supervised predictive learning. Advances in Neural Information Processing Systems, 34:25164–25178, 2021.\n\n"}, {"review_id": "qsAEe2kTtS", "reviewer": "Reviewer_7ncd", "summary": "This paper focuses on investigating differences between the human ventral and dorsal visual streams. The authors develop a dual-stream model that has different retinal inputs to support “what” processing (object recognition) or “where” processing (guiding the fixations). The authors then test how well the model activations are able to predict human fMRI activity and see a distinction where the “What” model better predicts ventral visual areas and the “Where” model better predicts dorsal areas. ", "questions": "a)\tIn Figure 3b, it looks like there are many ventral voxels that are predicted better with the “Where” CNN. In contrast, there are few dorsal voxels that are predicted well by the “What” CNN. Is there an interpretation for this? And are voxels from early visual areas included in this plot (if so, what color are they?) \n\nb)\tThe colorbar on Figure 3a is misleading because there is little difference between the darkest blue and the darkest red. A diverging color map would better capture the data. \n \nc)\tThere are control experiments presented with alternative models, but from what I can tell these alternative models always have “dual streams”. Is it necessary to have the two streams to get good predictivity for these voxels or could it be captured with a single model performing two tasks (especially given that the inputs seem to matter less than the task)?\n\nd)\tA fourth model with different inputs but predicting “perception” may show a different trend, and should possibly be included for completeness. Similarly a model with only wide inputs to each of the architectures.  \n\ne)\tAdditional details of training, or a direct reference to the supplement where these details are provided, would be helpful. Specifically, on line 165 it is stated that the WhereCNN “learns spatial attention from humans”. Is this separate training than the SALICON dataset? And the training dataset, updates/labels etc are not given for Stage 3 of training.  \n\nf)\tQuantification of the similarities between human and model attention/saccadic eye movements would be nice. Also, are the images in Figure 2 from the train or test dataset? \n\ng)\tAre the human eye movements during the fMRI activity recorded and available? If so, how well do these match up with those predicted by the model? (Maybe beyond the scope but an interesting analysis). \n\nMinor:\n\ni) Section 4.1 seems less of a result and more like validation that the training worked, as these validations are what the networks were trained to do. \n\nii) Lines 99-100 are technically true (in that the models do not fully replicate the way human eyes sample visual inputs) however neither does this study (it is still a model and does not capture all aspects of the eye). It would be better to explicitly state the differences between the previous studies and the model used here.\n\niii)\tClarifying Figure 4 and its caption to make it clear that the right most model is the standard model used in the rest of the paper would be helpful. \n\niv)\tOn line 265 there is a comment about WhatCNN and WhereCNN performing comparably to deeper models, but I don’t understand why depth matters in this context? \n\n", "limitations": "Limitations are addressed in the final paragraph of the discussion. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* Although there is a large amount of work studying the similarity of image-computable models and the ventral visual stream, there is less work done studying the dorsal stream. This paper provides an interesting analysis of the functional separation of these two pathways. \n* The retinal framework and its relation to biology is original and significatn, as well as the model leading to different “fixations”.  \n* The dual-task model presented is original, and the experiments on different versions of the model provide interesting interpretation of the training components.       \n", "weaknesses": "* In the abstract and elsewhere (ie line 90 and line 248-250), the authors state that “the functional segregation of the brain’s dorsal and ventral visual pathways is driven primarily by their different learning objectives”. This is too strong of a claim and not supported by the results (a correlation $\\neq$ causation fallacy), as there is no evidence presented in the paper showing that the brain itself relies on these specific learning objectives.\n* The “WhereCNN” is initially trained on Image Recognition and then fine-tuned on saliency maps. This makes the differences between the two models difficult to interpret, as it is *also* optimized for object recognition. \n* The authors detail the interesting addition of foveated vision and retinal transformations, however from the results it appears that this is less necessary for performance than the two different tasks, but the relevance of this to our understanding of biology is not discussed. It is hard to say whether this lack of a result is due to the retinal sampling being incorrect, or if it simply doesn’t matter for this level of analysis. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are glad that you recognize some strengths, interests, and novelties of our work. We also respect your critical evaluation and feedback.\n\n***\n1. __Overreach claims.__ We respect your concern regarding the potential overreach of our initial claim. We have revisited our wording to more accurately reflect our findings. Our revised statement now reads, “By evaluating the linear alignment between dual-stream models and brains during dynamic natural vision, we infer that the distinct responses and representations of the ventral and dorsal streams are more influenced by their distinct goals in visual attention and object recognition than by their specific bias or selectivity in retinal inputs.” Indeed, we do not present direct evidence concerning the brain’s learning processes. Our approach centers on reverse engineering the brain using neural network models. We embed hypotheses in these models and then test, compare, or falsify the hypotheses by comparing these models to the brain. While this method is correlational in nature, it is a valid inference framework established in the computational neuroscience community. In light of your concern, we will carefully reword our claims and findings in the final version of this paper.\n\n\n2.  __Pre-training concerns.__ Indeed, both streams were initialized by pre-training with the same task of image classification due to practical considerations. Due to the small size of the human saliency dataset, training WhereCNN without pre-training would be challenging. Following pre-training, subsequent training phases train both streams on consistently different tasks (spatial attention for WhereCNN, object recognition for WhatCNN). Although both streams start with the same pre-training task, their resulting representations post-training diverge (Figure 3) suggesting that the two streams have learned to build distinct representations through their respective learning objectives.\n\n\n3. __Retinal inputs vs. functional goals.__ The parallel dorsal vs. ventral visual streams have two major distinctions. They receive separate retinal inputs and serve different functional goals, for which we focus on spatial attention vs. visual perception in this paper. By accounting for these distinctions in our dual-stream model, we show that the functional segregation of the dorsal and ventral streams is more attributable to their divergent goals than their varied inputs. This finding is valid at least for the level of our analysis with fMRI data during movie viewing. However, we do not imply that foveated vision or retinal transformation is not important for vision in general. We recognize that prior studies have shown the utility of modeling the retina for adversarial robustness, neural activity control, to name a few [Vuyyuru et al., 2020, Bashivan et al., 2019].\n\n***\na) Figure 3b. Voxels and regions in Figure 3b exclude early visual areas, which are involved in both dorsal and ventral pathways. That said, WhatCNN and WhereCNN can capture the differentiation between foveal or peripheral representations in early visual areas (Figure 3a). While a significant portion of the ventral and dorsal streams align well with the predictions of the WhatCNN and WhereCNN, we concur that there are (many fewer) instances of misalignment or overlap. Pinpointing a single reason for this observation is challenging. A plausible interpretation is that there is shared information in the dorsal and the ventral streams [Perry et al., 2014].\n\nb) Figure 3a. We appreciate your suggestion and have updated the figure with a new colormap.\n\nc) Single-stream with dual tasks. The reviewer raises an interesting question.The brain’s parallel visual streams are divergent anatomical pathways, signifying two separate branches of the visual system. A model that attempts to simulate the human visual system should possess two branches or models, each simulating one stream. Our study’s core objective is not only to explain both streams but also to shed light on the factors differentiating them. Even if a single model trained on two tasks might explain the overall functionality of both streams, it would not be able to delineate their anatomical or functional segregation, a primary focus of our research.\n\nd) Alternative dual-stream models. Thanks for this thoughtful suggestion. We have added an additional alternative model. This alternative model does not explain the ventral and dorsal streams as distinctly as our proposed model, reinforcing our initial conclusion. See Figure R2 attached to the global response.\n\ne) Details about training. Thanks for the suggestion. We refer the reviewer to the global response section, where we provide further details about model training.\n\nf) Saliency prediction. Figure 2 shows results with validation images held out from training. The quantitative measures of the test set of SALICON are AUC: 0.699, SAUC: 0.618. Further qualitative results are presented in Appendix Figure S3.\n\ng) Eye movement data. Unfortunately, the eye movement data is unavailable in the movie fMRI dataset. This prevents us from evaluating a simultaneous prediction of eye movement alongside brain responses. We concur that this would, otherwise, be an interesting analysis.\n\n***\ni) Indeed, section 4.1 is intended to confirm that training works as intended.\n\nii) In line 99-100, our intent is to highlight that our model allows “active exploration” via eye movement, since the model includes a branch to control where the look next, distinguishing it from prior works.\n\niii) We have revised Figure 4 and its caption as suggested.\n\niv) Our statement of concern is a minor point. A deeper model tends to better explain neural activity [Yamins et al., 2016]. Although our model has fewer parameters, it explains brain responses better than or comparable as deeper models. In case the reviewer has strong objections, we can remove this statement, which would not compromise the main conclusion.\n"}, {"review_id": "qM9PseBaDT", "reviewer": "Reviewer_E5bU", "summary": "The main contributions is a dual stream neural network to model human visual processing. The network is composed of two CNNs (where and what) to model the dorsal and ventral pathways. The authors demonstrate that they can predict human gaze behavior and neural responses using their model, and gain new insights in to the role of the dorsal stream and how it interacts with the ventral stream.", "questions": "As I mentioned in weakness, could the authors touch on how their dual stream model is relevant a wider computer vision audience? What are potential performance benefits of the dual stream approach and/or some the gaze mechanisms you incorporated (better accuracy? robustness? efficiency?). Numerical results would be appreciated, but general statements/discussions are also fine.\n\nWhat is the size of your dual stream model? How costly is it to train? Could you point me to performance scores? I would like to know more of these details about the model.\n\nOn prior work, could the authors touch on how their work relates to metrics like BrainScore (Schrimpf et al. 2018)? Also, how does your model compare to GistNet (Wu et al. 2018) which I believe takes a two pathway approach.", "limitations": "Yes, the authors adequately address limitations. The authors note that their model does not address temporal sampling and does not completely capture how the dorsal and ventral streams interact with one another. I might argue that relying on small CNNs for their model and relying on tasks like recognition and detection to learn visual representation could be limiting, but I discuss these concerns in weaknesses and questions and find the work good otherwise.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "A dual stream CNN to model the dorsal and ventral stream is novel as far as I know and quite interesting. The authors do a good job of the motivating their work and being clear about their contributions. As they note, many approaches to comparing human and deep neural networks stop at classification CNNs, so moving beyond simple object recognition and separating out different streams of processing is an interesting contribution.\n\nIn addition, the experiments appear to be well reasoned. The authors create a model motivated by properties of the human brain, and provide logical comparisons points --the authors provide both an eye-gaze and fMRI analysis to compare their model to humans and also provide a baseline single stream CNN to make it clear what benefits arise from the dual stream design.  Overall, the work is very clear.", "weaknesses": "The work does not have enough discussion of how their dual stream model could be useful to a more broad computer vision audience. Although the authors show their dual stream model does better than single stream at explaining brain response, the work would be strengthened by more clearly demonstrating the use for a dual stream system directly on a computer vision tasks (i.e. does object recognition or detection improve? or robustness?).\n\nThe authors only consider CNNs and supervised training in their modeling. While these models are often used when comparing humans and deep neural networks, they are not top performers in the context of computer vision. Even in the context of biological modeling, more specialized models (for example, adversarially trained models) are thought to share more similarities with human visual representations than  standard CNNs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are glad that the reviewer recognizes the novelty, clarity, and contribution of our paper and offers constructive comments. Our responses to the weaknesses and questions are as below.\n\n***\n__Applications in computer vision__\n\nThe main focus of this paper is on modeling the human visual system, including its ventral and dorsal streams. We agree with the reviewer that demonstrating the benefits of our dual-stream model for a range of computer vision applications could enhance our work’s appeal. Given the page limit, we opt to keep our primary focus on computational neuroscience and will perform a comprehensive extension of the work for computer vision applications in a different paper. That said, we will include more discussions about several potential advantages for computer vision and the remaining challenges.\n\n__1) Efficiency.__ Unlike existing models that process entire images, our dual-stream model allows serial processing. It concentrates processing power on key image regions through attention directed fixations. This serial processing may significantly lower memory and computational overhead. It is plausible that such efficiency underpins the brain’s adoption of dual stream processing due to biological constraints on energy use.\n\n__2) Adaptability.__ The dual streams of our model offer complementary lenses for visual exploration and perception in real-world environments. One stream provides a broad yet rough overview of the environment. The other gathers detailed observations with precision. Their synergistic interaction may facilitate adaptive behaviors for tasks like visual search, object detection in complex and cluttered scenes. Moreover, the distinct functions of each of the parallel streams present a combinatorial flexibility when leveraged together, potentially enhancing the model’s overall capability to adapt to diverse visual challenges, including potential applications in robotics.\n\nHowever, leveraging such potential benefits within the scope of current study face challenges. First, mainstream datasets like ImageNet and MS-COCO offer a narrow view and lack the high-resolution detail our model thrives on. Moreover, these datasets often focus on large, central objects, limiting our model's adaptability that benefits object recognition. A better benchmark to our model would be high-resolution panoramic images or synthetic virtual reality environments to accommodate unlimited fixation variances. In such settings, the efficiency and adaptability of our model should be more appealing.\n\n***\n__Concerns on CNNs and supervised learning__\n\nThank you for the insightful comments. Our use of CNN and supervised learning in this study is motivated by a comprehensive literature on the similarity of CNNs and brain mechanisms in vision. However, we agree with the reviewer that our central idea is not limited to using CNNs exclusively as the architecture. For example, Transformer based architectures are increasingly are potential alternatives to CNNs. However, the biological relevance of Transformer, although hinted at in recent studies, remains unclear to us. Given our primary interest in brain-inspired vision models, our current position is still to prefer CNNs for this paper. Nevertheless, we concur with the reviewer’s suggestion that exploring a diverse set of architectures in dual-stream visual processing merits further investigations.\n\nLikewise, supervised learning is not the only strategy for training dual-stream vision models. The recent advancement in unsupervised and self-supervised learning is encouraging. Such approaches align with our aspirations in brain-inspired learning for vision. As of now, we have not yet pinpointed an effective self-supervised learning strategy tailored for dual-stream models. Moreover, we are amenable to exploring other learning strategies, such as adversarial training (mentioned by the reviewer). We also value any further recommendations from the reviewer. In the final version of this paper, we will add discussions about alternative architectures and learning strategies while citing related literature.\n\n***\n__Size and performance__\n\nOur model (including both streams) comprises 4.73 million trainable parameters, and performs multi-label classification on the test images of MS-COCO by F1 score 61.0. We recognize the importance of demonstrating the advantages of our dual-stream model to the broader computer vision community. As elaborated in our response to another comment, the datasets (ImageNet or MS-COCO) used in this study are not ideal for demonstrating the strengths of our model for computer vision. We acknowledge this weakness. \n\n***\n__BrainScore__\n\nYour suggestion about BrainScore is valuable. We agree with the premise. Indeed, brain-like vision models should be evaluated not just based on their performance on computer vision tasks but also their capability to predict neural responses and behavioral measures. While we have yet to submit our model to BrainScore, we are keen on leveraging BrainScore in the future. However, we anticipate certain challenges. For example, the neural data used by BrainScore are with central fixations, while our model actively adjusts its fixations. We prefer dynamic and naturalistic stimuli, while BrainScore mainly uses static images for its evaluations.\n\n***\n__GistNet__\n\nThanks for referring us to GistNet. We will discuss this work in the section of prior work. Indeed, our model and GistNet both utilize a dual-stream architecture. The primary distinction is our model’s active exploration of images by varying fixations, while GistNet assumes fixations centered on objects. Relevant to this distinction is our intended use of WhereCNN to learn spatial attention and use attention to drive eye movements.\n"}, {"review_id": "8yMYadMbcc", "reviewer": "Reviewer_xqDS", "summary": "This paper analyzes which ANN structures are more suitable for conversion to SNN. Based on the analysis results, the RepConv model is chosen as the baseline ANN model. The ANN-to-SNN conversion is performed using Value-range (VR) encoding and Clamp and Quantization (CQ) training, resulting in a high-performance and low-latency SNN network.", "questions": "See the weaknesses", "limitations": "Some experimental details and explanations are needed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe neural network model structure and training approach are novel. Compared to previous dual-stream models, this work trains the two streams separately using different tasks and combines their training by incorporating changes in attention, enabling simultaneous training.\n2.\tThe network successfully models the functional specialization of the ventral and dorsal visual pathways and demonstrates that the two streams correspond to different visual pathways. Additionally, it exhibits comparable predictive performance to other single-pathway neural networks.\n3.\tThe ablation experiments are comprehensive and provide insights into the importance of learning objectives and retinal input in the functional specialization of the human visual pathways.\n4.\tThe content of the paper is substantial, the experimental results are solid, and the writing is good.\n", "weaknesses": "1.\tSome details need to be addressed. What datasets were used for training the two streams synchronously? Why is the learning rate lower when training the two streams together compared to training them separately?\n2.\tAlthough the network has a recurrent structure and multiple time steps, the training dataset consists of static data. However, fMRI visual input is dynamic. Could dynamic datasets be considered for training the network?\n3.\tSingle-pathway networks with a single training objective (object recognition) such as ResNet18 and ResNet34 also exhibit good predictions of fMRI data for both visual pathways. However, intuitively, such models should not be able to predict the dorsal visual pathway well. Does this result indicate that there is no significant difference in fMRI data features between the ventral and dorsal pathways?\n4.\tWhen using network features to predict fMRI data, is it possible to predict separately for each layer to examine whether each layer corresponds to different regions of the two pathways?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your positive evaluation, constructive feedback, and insightful comments.\n***\n1\\. Regarding the questions on why the learning rate for stage3 was set lower than that for stages 1 and 2, the answer is that our empirical findings led us to this decision, as a reduced learning rate during stage3 resulted in a more stable learning curve. This is also a common strategy used in multi-task / multi-stage / transfer learning schemes in deep learning.\n\nFor training stage 3, we used the SALICON dataset for training. Stage 3 utilizes both objective functions of saliency prediction and object recognition. Because the images in SALICON are collected from the MS-COCO, images in SALICON have labels for saliency prediction and object recognition. This is the major reason that we use SALICON as the training dataset in stage 3. We have also included more training details in the global response section. In light of the reviewer’s suggestion, we will include substantial training details in Appendix B for the final version of this paper.\n\n***\n\n2\\. We thank the reviewer for the suggestion. We agree that training our model on dynamic datasets instead of static datasets, particularly for tasks like action recognition, would be a feasible and compelling idea, and may result in a better alignment with the fMRI data collected with dynamic visual input. Yet, a significant challenge is the scarcity of dynamic datasets for saliency prediction. The prevalent use of static images in most saliency prediction datasets have unfortunately limited our choice to use static images for model training.\n\n***\n3\\. Concerning the comment on Figure 5, we would like to firstly acknowledge an error in the figure's label. The current Figure 5 in the main text presents the two panels for WhereCNN / dorsal ROIs (left panel) and WhatCNN / ventral ROIs (right panel). However, The left and the right labels were switched from each other, meaning that the left panel is about WhatCNN / ventral ROIs, and the right panel is about WhereCNN / dorsal ROIs. The revised figure has been appended to the PDF in the global response, and this change does not affect the other analysis or conclusions we made. We deeply regret this oversight.\n\nFrom a neuroscience perspective, it is well-established that the dorsal and ventral streams are selective to different features of visual stimuli, each having unique roles. For instance, while the dorsal stream deals with spatial layouts and it is sensitive to motion, the ventral stream extracts object features [1, 2]. Thus, we expect that the two streams show divergent representations in fMRI responses to movies.\n\nThe updated Figure 5 illustrates that baseline models such as BaseCNN, AlexNet, ResNet18, and ResNet34 better predict the ventral ROIs than the dorsal ones. Several dorsal ROIs, with r values nearing 0, suggesting a lack of predictability. In contrast, our WhereCNN and WhatCNN models provide superior predictions across most dorsal and ventral ROIs, outperforming both the baseline models of comparable size like BaseCNN and AlexNet and the larger ones such as ResNet18 and ResNet34.\n\n***\n4\\. We appreciate the reviewer suggestion on layer-wise encoding. In light of this suggestion, we predicted fMRI responses using features from each layer in the WhereCNN and WhatCNN. The results have been included in the global response as Figure R4. Briefly, Figure R4 associates each voxel to one (color-coded) layer most predictive of that voxel for either (a) WhatCNN or (b) WhereCNN. Figure R4 (a) shows that the lower layers of WhatCNN better predict earlier visual areas such as V1/V2, whereas the higher layers of WhatCNN better predict higher-order visual areas such as LO and PIT, consistent with prior studies [3, 4]. The results with the WhereCNN show different patterns, as shown in Figure R4 (b). Within early visual areas, the lower layers of WhereCNN better predict foveal representations, whereas the higher layers better predict peripheral representations. However, the layer-wise encoding of higher visual areas seems not readily explainable. We need more time for deeper analysis before making an affirmative conclusion. In our provisional interpretation, the lack of a hierarchy of layer-wise prediction in WhereCNN, compared to WhatCNN, might suggest that the representational hierarchy in the visual cortex is likely more attributable to visual perception/recognition rather than attention.\n\n***\n__References__\n\n[1] Haxby, J.V., Grady, C.L., Horwitz, B., Ungerleider, L.G., Mishkin, M., Carson, R.E., Herscovitch, P., Schapiro, M.B. and Rapoport, S.I., 1991. Dissociation of object and spatial visual processing pathways in human extrastriate cortex. Proceedings of the National Academy of Sciences, 88(5), pp.1621-1625.\n\n[2]  Mishkin, M., Ungerleider, L.G. and Macko, K.A., 1983. Object vision and spatial vision: two cortical pathways. Trends in neurosciences, 6, pp.414-417.\n\n[3] Wen, H., Shi, J., Zhang, Y., Lu, K.H., Cao, J. and Liu, Z., 2018. Neural encoding and decoding with deep learning for dynamic natural vision. Cerebral cortex, 28(12), pp.4136-4160.\n\n[4] Cadena, S.A., Denfield, G.H., Walker, E.Y., Gatys, L.A., Tolias, A.S., Bethge, M. and Ecker, A.S., 2019. Deep convolutional models improve predictions of macaque V1 responses to natural images. PLoS computational biology, 15(4), p.e1006897.\n"}]

Author rebuttal:
We appreciate the time, effort and feedback from all reviewers. We are glad that the review comments are overall positive and seem to merit acceptance. We have also carefully addressed all concerns from the reviewers, by providing new results (shown in the pdf file), discussions, and clarifications, and have performed new experiments according to the suggestions. While we intend to focus mainly on computational neuroscience with brain-inspired neural networks, our model described in this paper would potentially advance the capabilities of adaptive, efficient, and robust computer vision systems – a premise that merits future investigations. 

***
0. Correction on Figure 5. 

Firstly, we must acknowledge an error in the labeling of Figure 5. The left panel was originally labeled as "WhereCNN” and “Dorsal ROIs" and the right panel as "WhatCNN” and “Ventral ROIs". However, the labels should be reversed. The correct labeling should have the left panel presenting "WhatCNN” and “Ventral ROIs" and the right panel "WhereCNN” and “Dorsal ROIs". We have attached the corrected figure (Figure R3) in the global response section of the supplementary material. This error does not influence our interpretations or the conclusions drawn from the data. We deeply regret this oversight.

***
1. Novelty and Significance

We would like to emphasize the novelty and significance of our work. The goal of our study is to build a better computational model that explains and differentiates the ventral and dorsal streams in the human brain. Compared to numerous studies of the ventral stream, only a few prior works have studied the dorsal stream using neural networks. Further fewer works tried to model both streams altogether. Specifically, in this study we used a fully computable deep neural network to simultaneously model overt visual attention (eye movements) through the dorsal stream, as well as the object recognition by the ventral stream. The combination of the overt attention and the object recognition tasks is also useful for the image computable models in the field of computer vision. Such models can concentrate its resources to informative regions in images or videos, with potential advantages in efficiency and adaptability. Based on the idea, we built an image computable model by capturing key biologically-inspired characteristics of the human visual system (two-stream architecture, retinal transformation and eye movements), and trained each stream with the distinct inputs and learning objectives to mimic their functionalities and behaviors.

Our study reveals that the learned representations from the WhereCNN and WhatCNN correspond to the dorsal and the ventral streams of the human visual systems. We then specifically investigated the key factor responsible for representational differentiation in the two streams, which has been rarely studied with neural network models. Through a set of control experiments to disassociate different factors in the two streams, we report that the major distinction between WhereCNN and WhatCNN representations is driven by the roles that the two streams perform, rather than the distinct retinal inputs they receive. This finding is novel.  

In summary, the primary achievements of our research include:
- The construction of a model encapsulating both streams of the human visual system.
- Novelty in modeling the dorsal stream with overt visual attention.
- Novel insights into the distinction in visual stream representations, emphasizing the role of their distinct functionality over diverse inputs.

***
2. Training Details

We thank reviewers' comments regarding the specifics of our training stages. We will include more training details in Appendix Section B in the final version to ensure reproducibility. We have also fully addressed all questions from the reviewers on methodological details or ambiguities. We will the final paper and its appendix reflects full clarity. 

Regarding the questions on how stage 3 training was conducted, the brief answer is that we incorporated both object recognition and saliency prediction losses simultaneously, thus it requires a training dataset with labels for both tasks. Because SALICON has labels for both tasks, we used SALICON for stage3. We will ensure greater clarity on this in both the main text and Appendix Section B.

Lastly, we acknowledge the need to clarify the statement in line 165: "After this stage, WhereCNN learns spatial attention from humans.", because it may raise confusions to readers. Here, our intention was to convey that post stage1 training, the WhereCNN is primed to generate saliency maps. There isn't an additional training phase implied. This will be made more explicit in our revised text.




Meta-review: The paper introduces a dual-path architecture to simulate the function of the dorsal and ventral visual streams in the brain, with the WhereCNN guided by spatial attention matching brain activity in the dorsal stream and the WhatCNN guided by object recognition as learning objectives matches responses in the ventral stream better. The reviewers appreciate the overall question and approach, but also raise some issues related to novelty of the work and whether all claims are fully supported. After the response and discussions, some of the reviewers remain somewhat reserved, but all of them see the value of the paper suggest acceptance despite the concerns.
Decision: Accept (poster)