SlowGuess/NeurIPS_2023_OCR huggingface dataset:
By using linear encoding models, we use the WhereCNN and WhatCNN branches to predict fMRI responses during the processing of identical movie stimuli by both the model and the brain. Together, these two branches can predict responses across a wide range of cortical locations involved in visual processing. However, they exhibit distinct predictive power in relation to the dorsal and ventral streams. Generally, the WhereCNN branch exhibits superior predictive performance for the
WhereCNN learns human attention to mimic human gazes
Figure 2: Saliency prediction. Given an image (1st row), WhereCNN generates a saliency map (3rd row) similar to the map of human attention (2nd row). Sampling this saliency map generates a sequence of fixations (as red/orange/blue circles in the order of time in the 4th row) similar to human saccadic eye movements (not shown).
(a) Relative contributions of WhereCNN and WhatCNN to prediction of brain responses
(b) Brain response predictability by WhereCNN vs WhatCNN
Figure 3: Differential encoding of the dorsal and ventral streams. (a) Relative contributions of WhereCNN and WhatCNN to the prediction of the fMRI response observed at each cortical location. Color highlights the locations significantly predictable by the model (FDR<0.05, block permutation test). The color itself indicates the degree by which WhereCNN is more predictive than WhatCNN (warm-tone) or the opposite (cool-tone). Visual areas are delineated and labeled based on brain altas [73]. Panel (b) plots the predictive performance by WhereCNN (y-axis) against that by WhatCNN (x-axis) and shows a clear separation of voxels (left panel) or ROIs (right panel) along the dorsal stream (red) vs. ventral stream (blue) relative to the dashed line of equal predictability. See Appendix A for the full ROI labels.
Combination 1
Combination 2
Combination 3
Combination 4
(a) Alternative inputs and objectives for WhereCNN and WhatCNN
Combination 5
(Proposed Model)
(b) Brain response predictability by WhereCNN vs. WhatCNN
Figure 4: Contributing factors of the dorsal-ventral functional segmentation. (a) Alternative designs of the two-stream model for investigating the contributing factors of the functional segregation of two streams in the proposed model. In addition to WhereCNN and WhatCNN in the proposed model, we included ControlCNN-a (narrow input field of view for predicting location) and ControlCNN-b (wide input field of view for predicting perception) for an ablation study. (b) The predictive performances and functional segregations by the two streams are plotted for the dorsal (red squares) vs. ventral (blue triangles) ROIs for each of the alternative models in (a), correspondingly. The dashed diagonal lines represent equal predictive abilities of ventral and dorsal ROIs. (c) Quantitative evaluation of the functional segregation of the dorsal and ventral ROIs relative to the dashed line of equal predictability. Separability measures how far the predictions are away from the dashed line. Assuming the coordinates of a certain ROI is  $(x,y)$ , separability is calculated as the average of  $|x - y|$  for all ROIs.
(c) Effect Summary
dorsal stream, while the WhatCNN branch performs better in predicting responses within the ventral stream (Fig.3). For the early visual areas (V1, V2, V3), WhereCNN better predicts the peripheral representations, while WhatCNN better predicts the foveal representations.
4.3 Factors underlying the functional segregation of the dorsal and ventral streams
We further investigate the underlying factors contributing to the model's ability to explain the functional segregation of the brain's dorsal and ventral visual streams (as depicted in Fig.3). Specifically, we examine the input sampling pattern and output learning objective, both of which are distinct for the WhereCNN and WhatCNN branches in our dual-stream model.
To investigate the contributing factors of the functional segregation in predicting human ventral and dorsal streams, we introduce four variations of the proposed model, where the two branches either share their inputs or have the same learning objectives, and compare their abilities to account for the functional segregation of the dorsal and ventral visual streams (as shown in Fig. 4). When the two branches solely differ in their input sampling, they are unable to explain the dorsal-ventral segregation (combination 1 and 2). However, when the two branches exclusively differ in their learning objectives, the functional segregation is better explained (combination 3 and 4). Moreover, when the two branches differ in both input sampling and learning objectives (combination 5), as utilized in our proposed model, the functional segregation is even more pronounced. These ablation experiments suggest that the distinct learning objectives of the brain's dorsal and ventral streams is the primary factor underlying their functional segregation.
4.4 Dual-stream: a better brain model than single-stream
We also compare our dual-stream model with single-stream alternatives. One of these alternatives is a baseline CNN that shares the same backbone architecture as a single branch in our dual-stream model. However, this baseline CNN is trained with original (224x224) images to recognize objects in ImageNet [65] and MS-COCO [68]. Thus, it serves as a direct comparison with either the WhereCNN or WhatCNN branch in our model. In addition, we also include AlexNet [14], ResNet18,
Comparisons to feed-forward CNNs
Figure 5: WhatCNN (left) or WhereCNN (right) vs. alternative single-stream CNNs. The boxplot shows the encoding performance of different models for ventral or dorsal visual areas. Each dot within the box plot signifies the average prediction accuracy r within a respective ROI in the ventral or dorsal region. Asterisk (*) represents a significant difference by the Wilcoxon signed-rank test  $(\alpha = 0.05)$ .
Effect of Fixations (Learned vs Random)
Figure 6: Effects of attention-driven eye movements. The use of attention to determine fixations vs. the use of random fixations is evaluated in terms of the resulting difference in the encoding performance by WhereCNN (left) and WhatCNN (right), denoted and color-coded as  $\Delta r_{\text{where}}$  and  $\Delta r_{\text{what}}$ , respectively. Voxels displayed in warm colors indicate that the predictions are more accurate with learned fixations, whereas those in cool colors signify better predictions with random fixations.
Prediction By WhatCNN
and ResNet34 [15] as additional alternatives, which have been previously evaluated in relation to brain responses [30, 74]. We compare these single-stream alternatives with either branch in our model in terms of their ability to predict brain responses within dorsal or ventral visual areas (Fig.5). Despite their use of the same backbone architecture, the baseline under-performs WhatCNN in predicting responses in ventral visual areas, and under-performs WhereCNN in predicting responses in dorsal visual areas. This result suggests that the interactive and parallel nature of the dual-stream model renders each stream more akin to the functioning of the human brain, surpassing the performance of isolating a single stream. Moreover, WhatCNN or WhereCNN also performs better than AlexNet and comparably with ResNet18 and ResNet34, which are deeper than the architecture of our model.
4.5 Attention-driven eye movements improve encoding
Similar to human gaze behavior towards salient objects, our model learns spatial attention to guide fixations for parallel visual processing. In this study, we investigate whether and how the model's ability to predict brain responses depends on its utilization of attention-driven fixations. To examine this, we conduct experiments where the model is allowed to use either attention-driven fixations or random fixations to collect retinal samples, and we evaluate how this choice impacts the model's capability to predict brain responses by calculating  $\Delta r = r^{learned} - r^{random}$  for all voxels, where  $r^{learned}$  and  $r^{random}$  are encoding performances of each voxel using learned or random fixations respectively. As depicted in Fig.6, employing attention-driven fixations leads to higher encoding accuracy by both WhereCNN and WhatCNN compared to the use of random fixations for a majority of visual cortical locations within both the dorsal and ventral streams.
In summary, we introduce a new dual-stream neural network that incorporates the brain's mechanisms for parallel visual processing. The defining features of our model include 1) using retinal transformation to separate complementary inputs to each stream, 2) using different learning objectives to train each stream to learn either spatial attention or object recognition, and 3) controlling sequential fixations for active and interactive visual sensing and processing. We demonstrate that the combination of these features renders the model more akin to the human brain and better predictive of brain responses in humans freely engaged in naturalistic visual environments. Importantly, the two streams in our model differentially explain the two streams in the brain, contributing to the computational understanding as to how and why the brain exhibits and organizes distinct responses and processes along the structurally segregated dorsal and ventral visual pathways. Our findings suggest that the primary factor contributing to the dorsal-ventral functional segregation is the different goals of the dorsal and ventral pathways. That is, the dorsal pathway learns spatial attention to control eye movements [75, 7, 76, 77], while the ventral stream learns object recognition.
Although our model demonstrates initial steps to model parallel visual processing in the brain, it has limitations that remain to be addressed in future studies. For one limitation, the model uses different spatial sampling to generate the retinal inputs to the two streams but does not consider different temporal sampling that makes the dorsal stream more sensitive to motion than the ventral stream [38, 40, 39, 41]. For another limitation, the interaction between the two streams is limited to the common fixation that determines the complementary retinal input to each stream. Although attention-driven eye movement is an important aspect of human visual behavior shaping brain responses for both dorsal and ventral streams, the two streams also interact and exchange information at higher levels. The precise mechanisms for dorsal-ventral interactions remain unclear but may be important to understanding human vision or improving brain-inspired computer vision.
This research is supported by the Collaborative Research in Computational Neuroscience (CRCNS) program from National Science Foundation (Award#: IIS 2112773) and the University of Michigan.
[1] Mortimer Mishkin, Leslie G Ungerleider, and Kathleen A Macko. Object vision and spatial vision: two cortical pathways. Trends in neurosciences, 6:414-417, 1983.
[2] William H Merigan and John HR Maunsell. How parallel are the primate visual pathways? Annual review of neuroscience, 16(1):369-402, 1993.
[3] Jonathan J Nassi and Edward M Callaway. Parallel processing strategies of the primate visual system. Nature reviews neuroscience, 10(5):360-372, 2009.
[4] Margaret Livingstone and David Hubel. Segregation of form, color, movement, and depth: anatomy, physiology, and perception. Science, 240(4853):740-749, 1988.
[5] James V Haxby, Cheryl L Grady, Barry Horwitz, Leslie G Ungerleider, Mortimer Mishkin, Richard E Carson, Peter Herscovitch, Mark B Schapiro, and Stanley I Rapoport. Dissociation of object and spatial visual processing pathways in human extrastriate cortex. Proceedings of the National Academy of Sciences, 88(5):1621-1625, 1991.
[6] Giacomo Rizzolatti and Massimo Matelli. Two different streams form the dorsal visual system: anatomy and functions. Experimental brain research, 153:146-157, 2003.
[7] Maurizio Corbetta and Gordon L Shulman. Control of goal-directed and stimulus-driven attention in the brain. Nature reviews neuroscience, 3(3):201-215, 2002.
[8] Gustavo Deco and Edmund T Rolls. A neurodynamical cortical model of visual attention and invariant object recognition. Vision research, 44(6):621-642, 2004.
[9] Melvyn A Goodale and A David Milner. Separate visual pathways for perception and action. Trends in neurosciences, 15(1):20-25, 1992.
[64] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014.
[65] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211-252, 2015.
[66] Ming Jiang, Shengsheng Huang, Juanyong Duan, and Qi Zhao. Silicon: Saliency in context. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1072-1080, 2015.
[67] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[68] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dóllár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[69] Thomas Naselaris, Kendrick N Kay, Shinji Nishimoto, and Jack L Gallant. Encoding and decoding in fmri. Neuroimage, 56(2):400-410, 2011.
[70] James V Haxby, J Swaroop Guntupalli, Andrew C Connolly, Yaroslav O Halchenko, Bryan R Conroy, M Ida Gobbini, Michael Hanke, and Peter J Ramadge. A common, high-dimensional model of the representational space in human ventral temporal cortex. Neuron, 72(2):404-416, 2011.
[71] Matthew F Glasser, Stamatos N Sotiropoulos, J Anthony Wilson, Timothy S Coalson, Bruce Fischl, Jesper L Andersson, Junqian Xu, Saad Jbabdi, Matthew Webster, Jonathan R Polimeni, et al. The minimal preprocessing pipelines for the human connectome project. Neuroimage, 80:105-124, 2013.
[72] Daniela Adolf, Snezhana Weston, Sebastian Baecke, Michael Luchtmann, Johannes Bernarding, and Siegfried Kropf. Increasing the reliability of data analysis of functional magnetic resonance imaging by applying a new blockwise permutation method. Frontiers in neuroinformatics, 8:72, 2014.
[73] Matthew F Glasser, Timothy S Coalson, Emma C Robinson, Carl D Hacker, John Harwell, Essa Yacoub, Kamil Ugurbil, Jesper Andersson, Christian F Beckmann, Mark Jenkinson, et al. A multi-modal parcellation of human cerebral cortex. Nature, 536(7615):171-178, 2016.
[74] Haiguang Wen, Junxing Shi, Wei Chen, and Zhongming Liu. Deep residual network predicts cortical representation and organization of visual features for rapid categorization. Scientific reports, 8(1):3752, 2018.
[75] James W Bisley and Michael E Goldberg. Attention, intention, and priority in the parietal lobe. Annual review of neuroscience, 33:1-21, 2010.
[76] Steven Yantis and John T Serences. Cortical mechanisms of space-based and object-based attentional control. Current opinion in neurobiology, 13(2):187-193, 2003.
[77] John HR Maunsell and Stefan Treue. Feature-based attention in visual cortex. Trends in neurosciences, 29(6):317-322, 2006.
Appendix: A Dual-Stream Neural Network Explains the Functional Segregation of Dorsal and Ventral Visual Pathways in Human Brains
$^{1}$  Department of Electrical Engineering and Computer Science, University of Michigan, Ann Arbor, MI 48109  
 $^{2}$  Department of Biomedical Engineering, University of Michigan, Ann Arbor, MI 48109
A Regions of Interests
In our study, we delineated our regions of interest (ROIs) into two primary segments: 1) the ventral visual stream and object recognition-related regions and 2) the dorsal visual stream and overt attention-related regions. This approach followed the parcellations proposed by [1]. For the dorsal visual stream, the ROIs includes V3A, V3B, V6, V6A, and V7. Within the parietal cortex, visuo-spatial information and overt attention are processed by the intraparietal sulcus (IPS) and the superior parietal lobule (SPL) [2, 3, 4, 5, 6]. The IPS encompasses V7, IPS1, IPO, IP1, and IP2; whereas the SPL consists of lateral intraparietal cortex (LIPv, LIPd), ventral intraparietal complex (VIP), anterior intraparietal (AIP), medial intraparietal area (MIP), 7PC, 7AL, 7Am, 7PL, and 7Pm. We also included the frontal eye field (FEF), which is acknowledged for controlling eye movements [7, 8, 9, 10]. In contrast, the ROIs associated with object recognition and the ventral visual stream encompassed V8, the posterior inferotemporal (PIT) complex, the fusiform face complex (FFC), and ventromedial visual (VMV) areas 1, 2, 3, along with the lateral occipital area (LO). In addition, we included the superior temporal sulcus (STS), which is recognized for processing multimodal signals, including auditory and visual cues [11, 12, 13]. Fig. S1 displays the full set of region labels, corresponding to Fig.3(a) from the main text. Among the parcellations by [1], regions including significantly predicted voxels either by the WhereCNN or WhatCNN are presented in Fig. S1.
Figure S1: Region labels. Regions including significant voxels from Fig.3(a) in the main text are presented.
B Training Details
The backbone Convolutional Neural Networks (CNNs) of both the WhereCNN and WhatCNN share the same architecture, consisting of four blocks of convolutional operations. Situated atop the backbone CNN, the WhereCNN and WhatCNN possess additional layers tailored to their specific objectives: the WhereCNN features two convolutional layers that produce 2D saliency maps, whereas the WhatCNN includes a Gated Recurrent Unit (GRU) layer followed by a fully connected layer for object classification.
During the pre-training of the backbone CNN, a global average pooling and a fully connected layer are integrated atop the backbone CNN, serving as a classifier. Upon completion of the pre-training process, the classifier is detached, allowing the pre-trained backbone CNN to be incorporated as a component of the WhereCNN or WhatCNN.
As detailed in Section 3.1 of the main text, our model underwent a three-stage training process. In this section, we will elaborate on the specifics of the pre-training phase.
Stage 1 - WhereCNN The backbone architecture of the WhereCNN was pre-trained on ILSVRC2012 [14] for an image classification task over 120 epochs. A batch size of 1, 024 was employed, along with the Adam optimizer [15]  $(\mathrm{lr} = 0.001, \beta_{1} = 0.9, \beta_{2} = 0.99)$ . During pre-training, fixations for the retinal transformation were randomly generated across the image area. Once the backbone architecture had been pre-trained, we detached the classifier and initialized the WhereCNN using the model parameters obtained from the pre-training stage. We then performed SALICON training, as described in Section 3.1 of the main text.
Stage 2 - WhatCNN In a process mirroring Stage 1, the backbone of the WhatCNN was also pre-trained on ILSVRC2012 [14] for an image classification task over 120 epochs, utilizing random fixations and the Adam optimizer  $(\mathrm{lr} = 0.001, \beta_{1} = 0.9, \beta_{2} = 0.99)$ . After pre-training the backbone CNN, we initialized the WhatCNN using the weights of the pre-trained backbone CNN.
Subsequently, the WhatCNN, initialized with the pre-trained weights as a whole, was trained on ILSVRC2012 [14] for object recognition using four fixations. Four randomly generated fixations were employed for training the WhatCNN for 55 epochs, again utilizing the Adam optimizer  $(\mathrm{lr} = 0.001, \beta_{1} = 0.9, \beta_{2} = 0.99)$ . After this stage, we conducted a fine-tuning process using the learned fixations from the WhereCNN. In this stage, the WhereCNN, after the pre-training in Stage 1, was incorporated to guide the WhatCNN's fixations. However, only the WhatCNN was optimized, while the WhereCNN remained unchanged. This fine-tuning with learned fixations deployed four gazes, utilizing the Adam optimizer  $(\mathrm{lr} = 0.0001, \beta_{1} = 0.9, \beta_{2} = 0.99)$  over 25 epochs. Finally, the WhatCNN underwent further training on MSCOCO, as described in Section 3.1 of the main text.
Stage3 - WhereCNN & WhatCNN During this stage, both WhereCNN and WhatCNN, trained in the previous stages, were used to initialize model weights, followed by further end-to-end training, leveraging the stream-specific objectives (object recognition and saliency prediction, respectively). As the training requires labels for both tasks, the model was trained using images in the SALICON dataset, which contain labels for both saliency prediction and object recognition.
The model samples fixations from the predicted saliency maps from WhereCNN. As this sampling process is non-differentiable, the gradients from object recognition cannot optimize the weights of WhereCNN. To tackle this issue, we utilized REINFORCE [16] to approximate the gradient for WhereCNN. At the time  $t$ , a fixation  $l_{t}$  is generated by WhereCNN, based on which WhatCNN predicts a class prediction  $p_t$ . Then, in the context of REINFORCE, the reward  $r_t$  of choosing  $l_{t}$  as the fixation is calculated as the reduced classification loss relative to the previous time step  $r_t = CE(p_{t - 1},\mathrm{label}_c) - CE(p_t,\mathrm{label}_c)$ , where  $CE$  is the cross-entropy loss,  $\mathrm{label}_c$  is class labels. The goal of REINFORCE is to maximize the discounted sum of rewards,  $R = \sum_{t = 1}^{T}\gamma^{t - 1}r_{t}$ , where  $\gamma \in (0,1)$  is the discount factor and set as 0.8.
In this stage, we strived to minimize the object recognition and saliency prediction losses while maximizing the discounted sum of rewards. As indicated in Section 3.1 of the main text, we utilized the Adam optimizer  $(\mathrm{lr} = 0.0002, \beta_{1} = 0.9, \beta_{2} = 0.99)$  for 25 epochs for this training stage.
For All Stages All training stages were conducted using four NVIDIA A40 GPUs. All codes are written in Pytorch 1.9.1.
C Saliency Maps and Inhibition of Returns
Once the saliency maps were generated by WhereCNN, inhibition of return (IOR) was used to prohibit future fixations to re-visit image areas that had been already explored. This process is illustrated in Fig. S2
Figure S2: Process of determining the next fixation point given the current fixation. A saliency map, generated by WhereCNN, is multiplied element-wise (indicated by *) with the inhibition of return (IOR) to prevent future fixations from reverting to previous positions. In the IOR, white and black colors correspond to values of 1 and 0, respectively.
In the process of determining the next fixation, the WhereCNN generate a saliency map based on the current fixation. The location of this subsequent fixation is guided by the saliency map's probabilistic distribution. However, it's important to note that if the current fixation point possesses a high probability, subsequent fixations are likely to occur in proximity to the present fixation.
To ensure a more dynamic and comprehensive exploration of the visual field, we employed the principle of Inhibition of Return (IOR), detailed in Eq.2 of the main text, and presented again here in Eq.4.
$$
\mathbf {I O R} (t) = \operatorname {R e L U} \left(\mathbf {1} - \sum_ {\tau = 1} ^ {t} G \left(\boldsymbol {\mu} = \boldsymbol {l} _ {\tau}, \boldsymbol {\Sigma} = \sigma^ {2} \boldsymbol {I}\right)\right) \tag {4}
$$