SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Minkyu Choi $^{1}$ , Kuan Han $^{1}$ , Xiaokai Wang $^{2}$ , Yizhen Zhang $^{1,3}$ , and Zhongming Liu $^{1,2}$
$^{1}$  Department of Electrical Engineering and Computer Science, University of Michigan, Ann Arbor, MI 48109
$^{2}$  Department of Biomedical Engineering, University of Michigan, Ann Arbor, MI 48109
$^{3}$  Department of Neurological Surgery, University of California, San Francisco, San Francisco, CA 94143
{cminkyu, kuanhan, xiaokaiw, zhyz, zmliu}@umich.edu
The human visual system uses two parallel pathways for spatial processing and object recognition. In contrast, computer vision systems tend to use a single feedforward pathway, rendering them less robust, adaptive, or efficient than human vision. To bridge this gap, we developed a dual-stream vision model inspired by the human eyes and brain. At the input level, the model samples two complementary visual patterns to mimic how the human eyes use magnocellular and parvocellular retinal ganglion cells to separate retinal inputs to the brain. At the backend, the model processes the separate input patterns through two branches of convolutional neural networks (CNN) to mimic how the human brain uses the dorsal and ventral cortical pathways for parallel visual processing. The first branch (WhereCNN) samples a global view to learn spatial attention and control eye movements. The second branch (WhatCNN) samples a local view to represent the object around the fixation. Over time, the two branches interact recurrently to build a scene representation from moving fixations. We compared this model with the human brains processing the same movie and evaluated their functional alignment by linear transformation. The WhereCNN and WhatCNN branches were found to differentially match the dorsal and ventral pathways of the visual cortex, respectively, primarily due to their different learning objectives, rather than their distinctions in retinal sampling or sensitivity to attention-driven eye movements. These model-based results lead us to speculate that the distinct responses and representations of the ventral and dorsal streams are more influenced by their distinct goals in visual attention and object recognition than by their specific bias or selectivity in retinal inputs. This dual-stream model takes a further step in brain-inspired computer vision, enabling parallel neural networks to actively explore and understand the visual surroundings.
The human visual system comprises two parallel and segregated streams of neural networks: the "where" stream and the "what" stream [1]. The "where" stream originates from magnocellular retinal ganglion cells and extends along the dorsal visual cortex. The "what" stream originates from parvocellular retinal ganglion cells and extends along the ventral visual cortex [2]. The two streams exhibit selective responses to different aspects of visual stimuli [3]. The "where" stream is tuned to coarse but fast information from a wide view, while the "what" stream is selective to fine but slow
Figure 1: Brain-inspired dual-stream vision model. The top illustrates the subcortical (dashed arrows) and cortical (solid arrows) pathways for parallel visual processing in the brain. Given a scene (e.g., "two foxes on the lawn"), the retina samples incoming light relative to the fixation of the eyes (shown as the cross). Magnocellular (orange) and parvocellular (blue) retinal ganglion cells encode complementary visual information into two sets of retinal inputs relayed onto separate layers in the lateral geniculate nuclei (LGN) and further onto different neurons in the primary visual cortex (V1). Within V1, the relative ratio of magnocellular vs. parvocellular projections is higher for the periphery and lower for the fovea. Beyond V1, the magnocellular pathway continues along the dorsal visual cortex towards the intraparietal areas and further onto the frontal eye field (FEF) for oculomotor control, while the parvocellular pathway continues along the ventral visual cortex towards the inferior temporal cortex and further onto the superior temporal areas for semantic cognition. The bottom illustrates our model architecture including WhereCNN and WhatCNN. The model's frontend mimics the human retina and generates two separate input patterns relative to the fixation. One pattern is wider but coarser while the other is narrower but finer, providing the respective inputs to WhereCNN and WhatCNN. With the wide-view input, WhereCNN generates a probability map of saliency from which the next fixation is sampled. With a narrow-view input, WhatCNN generates an object representation per each fixation and constructs a scene representation recurrently from multiple fixations.
information from a narrow view [2, 4]. The two streams are thought to serve different purposes. The "where" stream zooms out for spatial analysis [5], visual attention [6, 7, 8], and guiding actions [9] such as eye movements [10], while the "what" stream zooms in to recognize the object around the fixation [11]. While being largely parallel, the two streams interact with each other [12]. In one way of their interaction, the "where" stream decides where to look next and guides the "what" stream to focus on a salient location for visual perception. As the eyes move around the visual environment, the interaction between the "where" and "what" streams builds a scene representation by accumulating object representations over time and space. This dual-stream architecture allows the brain to efficiently process visual information and support dynamic visual behaviors [13].
In contrast, computer vision systems tend to use a single stream of feedforward processing, acting as passive observers that sample visual information all at once with fixed and uniform patterns [14, 15, 16]. Compared to human vision, this processing is less robust, especially given adversarial attacks [17, 18]; it is less efficient since it samples visual information equally regardless of salience or nuisance [19]; it is less adaptive, lacking spatial attention for active sensing [20, 21]. These distinctions define a major gap between human and computer vision. Many visual tasks that are
straightforward for humans are still challenging for machines [22, 23]. Therefore, computer vision may benefit from taking further inspiration from the brain by using a dual-stream architecture to learn adaptive and robust visual behaviors.
To gain insights into the computational mechanisms of human vision, researchers have developed image-computable models by utilizing goal-driven deep neural networks that simulate human perceptual behavior. In particular, convolutional neural networks (CNNs) are leading models of visual perception, capturing the hierarchical processing by the brain's ventral visual stream [24, 25, 26, 27, 28]. Previous models of this nature commonly utilize CNNs trained through supervised learning [24, 27, 25, 29, 26, 30], adversarial training [31, 32], unsupervised learning [33, 34], or self-supervised learning [35, 36, 37]. However, models of the dorsal stream remain relatively under-explored, despite few studies [38, 39, 40, 41]. Existing testing of these models has primarily focused on static images presented briefly to the fovea, thus limiting their assessment to a narrow range of visual behaviors and processes [42]. A more comprehensive approach is needed to develop models that incorporate both dorsal and ventral stream processing and to assess those models against brain responses when humans engage both the dorsal and ventral streams to freely explore complex and dynamic visual environments, which may be simulated in experimental settings [43].
To meet this need, we have developed a dual-stream model to mimic the parallel ventral and dorsal streams in the human brain [1, 2, 3, 9]. The model includes two branches of convolutional neural networks: WhereCNN and WhatCNN, which share the same architecture but receive distinct visual inputs and generate different outputs. WhereCNN samples a wide view to learn spatial attention and where to direct the subsequent gaze, while WhatCNN samples a narrow view to learn object representations. By taking multiple gazes at a given scene, the model sequentially samples the salient locations and progressively constructs a scene representation over both space and time. To evaluate this dual-stream model as a model of the human visual system, we have tested its ability to reproduce human gaze behavior and predict functional brain scans from humans watching a movie with unconstrained eye movements. Our hypothesis is that the model's WhereCNN and WhatCNN branches can effectively predict the brain responses along the brain's dorsal and ventral visual pathways, respectively. In addition, we have also conducted experiments to evaluate the underlying factors contributing to the functional segregation of the brain's dorsal and ventral visual streams. Of particular interest were the relative contributions of retinal sampling, spatial attention, and attention-guided eye movement in shaping the function of the dorsal stream and its interplay with the ventral stream during dynamic natural vision.
2.1 Dorsal-stream vision
Image-computable models of the brain's dorsal stream have been relatively limited compared to models of the ventral stream. Previous work has attempted to model the dorsal stream by training deep neural networks to detect motion [38] or classify actions [40] using video inputs. However, these models do not fully capture the neuroscientific understanding that the dorsal stream is involved in locating objects and guiding actions, leading to its designation as the "where" or "how" visual pathway. More recent work by Mineault et al. focused on training a dorsal-stream model to emulate human head movements during visual exploration [39]. Additionally, Bakhtiari et al. utilized predictive learning to train parallel pathways and observed the ventral-like and dorsal-like representations as an emergent consequence of structural segregation [41]. However, no prior work has explored neural network models that emulate how the dorsal stream learns spatial attention and guides eye movements for visual navigation.
2.2 Spatial attention and eye movement
Prior research in the field of computer vision has attempted to train models to attend to and selectively focus on salient objects within a scene [21, 44, 45], rather than processing the entire scene as a whole. This approach aligns with the brain's mechanism of spatial attention, where the dorsal stream acts as a global navigator, and the ventral stream functions as a local perceiver. In line with this mechanism, previous studies have employed dual-stream neural networks that process global and local features in parallel, aiming to achieve enhanced computational efficiency as a unified system [46, 44, 47, 48, 49].
However, these models do not fully replicate the way human eyes sample visual inputs during active exploration of the scene and thus still fall short in biological relevance.
2.3 Foveated vision and retinal transformation
The human retina functions as a sophisticated camera that intelligently samples and transmits visual information. It exhibits the highest visual acuity in the central region of the visual field, a phenomenon referred to as foveated vision [50, 51, 52]. In contrast, peripheral vision uses lower spatial acuity but higher temporal sensitivity, making it better suited for detecting motion. These properties of retinal sampling are potentially useful for training neural networks to enhance performance [53, 54, 55, 56] or robustness [57, 58, 19], augment data or synthesize images [59]. The retina also transmits information to the brain using distinct types of cells. The magnocellular and parvocellular retinal ganglion cells have different distributions, selectivity, and relay information through largely separate pathways. Taken together, the retina transforms visual information into segregated inputs for parallel visual processing. This biological mechanism has not been systematically investigated.
Unlike the above prior works, our work combines multiple biologically inspired mechanisms into an integral learnable model. It uses a frontend inspired by the human retina and applies complementary retinal sampling and transformation. It uses two parallel pathways inspired by the human dorsal and ventral streams. It uses spatial attention and object recognition as the distinct learning objectives for training the two pathways. It further uses the attention to move fixations and thus allows the two pathways to interact for active sensing. Although these mechanisms have been explored separately in prior studies, their combination is novel and motivates our work to build and test such a model against the human brain and behavior in a naturalistic condition of freely watching a movie.
In our model, WhereCNN and WhatCNN serve distinct functions in processing objects within a scene. WhereCNN identifies the spatial location of an object, determining "where" it is situated, while WhatCNN focuses on recognizing the identity of the object, determining "what" it is. When multiple objects are present in a scene, WhereCNN learns spatial attention and "how" to sequentially locate and fixate on each object. This allows the model to selectively attend to different objects in a sequence, mirroring the dynamic nature of human eye movements during visual exploration. Fig.1 illustrates and describes the human visual system that inspires us to design our model.
3.1 Model design and training
Akin to the human eyes [60, 61], our model uses retinal transformation [62] to generate separate inputs to WhereCNN and WhatCNN. For both, the retinal input consists of  $64 \times 64$  samples non-uniformly distributed around the fixation. When describing a point in the retinal image and its corresponding point in the visual world in terms of the radial distance and the polar angle with respect to the fixation, their polar angles are the same while their radial distances are related by Eq. 1.
$$
r = g \left(r ^ {\prime}\right) = \frac {b}{\sqrt {\pi}} \frac {1 - \exp (\ln (a) r ^ {\prime} / 2)}{1 - \exp (\ln (a) / 2)} \tag {1}
$$
where  $r'$  and  $r$  are the radial distances in the retinal and original images, respectively,  $b$  is a constant that ensures  $r_{\mathrm{max}} / g(r_{\mathrm{max}}') = 1$ , and  $a$  controls the degree of center-concentration. Given a larger  $a$ , more retinal samples are closer to the fovea relatively to the periphery. We set  $a = 15$  for WhatCNN and  $a = 2.5$  for WhereCNN. In this setting, WhereCNN is more selective to global features, while WhatCNN is more selective to local features, mirroring the sampling bias of magnocellular and parvocellular retinal ganglion cells, as illustrated in Fig.1.
Both WhereCNN and WhatCNN use similar backbone architectures. The backbone consists of four blocks of convolutional layers. Each block includes two Conv2D layers (kernel size  $3 \times 3$ ) followed by ReLU and BatchNorm. Applying  $2 \times 2$  MaxPool between adjacent blocks progressively reduces the spatial dimension. The feature dimension following each block is 64, 128, 256, or 512. Atop this backbone CNN, both WhereCNN and WhatCNN use additional components to support different goals. For WhereCNN, the feature maps from the 3rd and 4th convolutional blocks are resized to
$16 \times 16$  and concatenated, providing the input to an additional convolutional block. Its output feature map is subject to SoftMax to generate a probability map of visual saliency. By random sampling by the probability of saliency, WhereCNN decides a single location for the next fixation. To avoid future fixations to revisit previously attended areas, inhibition of return (IOR) [63] is used. IOR keeps records of locations of prior visits as defined in Eq.2.
$$
\mathbf {I O R} (t) = \operatorname {R e L U} \left(\mathbf {1} - \sum_ {\tau = 1} ^ {t} G (\boldsymbol {\mu} = \boldsymbol {l} _ {\tau}, \boldsymbol {\Sigma} = \sigma^ {2} \boldsymbol {I})\right) \tag {2}
$$
where  $G(\pmb{\mu}, \pmb{\Sigma})$  is a 2D Gaussian function centered at  $l_{\tau}$  (prior fixations) with a standard deviation  $\sigma$  at the  $\tau$ -th step. Its values are normalized so that its maximum equals 1. By applying the IOR to the predicted saliency map using an element-wise multiplication, the future fixations would not revisit the areas already explored.
For WhatCNN, the output feature map from the 4th convolutional block, after global average pooling, is given as the input to an additional layer of Gated Recurrent Units (GRU) [64], which recurrently update the representation from a sequence of fixations to construct a cumulative representation following another fully-connected layer.
We first pre-train each stream separately and then fine-tune them together through three stages.
Stage 1 - WhereCNN. We first train WhereCNN for image recognition using ILSVRC2012 [65] for object recognition and then fine-tune it for generating the saliency map to match human attention using SALICON dataset [66]. In this stage, we use random fixations to generate the retinal inputs to WhereCNN. Adam optimizer [67] ( $\mathrm{lr} = 0.002$ ,  $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.99$ ) is used with 25 epochs for SALICON training. At this stage, WhereCNN learns spatial attention from humans.
Stage 2 - WhatCNN. We first train WhatCNN for single-object recognition using ILSVRC2012 [65] and then fine-tune it for multi-object recognition using MSCOCO [68]. In this stage, we use the pre-trained WhereCNN to generate a sequence of eight fixations and accordingly apply the retinal transformation to generate a sequence of retinal inputs to WhatCNN for recurrent object recognition. Note that the training in Stage 2 is confined to WhatCNN, while leaving WhereCNN as pre-trained in Stage 1. Adam optimizer  $(\mathrm{lr} = 0.002, \beta_{1} = 0.9, \beta_{2} = 0.99)$  is used with 40 epochs for MSCOCO training.
Stage 3 - WhereCNN & WhatCNN. Lastly, we equally combine the two learning objectives to train both WhereCNN and WhatCNN altogether and end-to-end using eight fixations. Adam optimizer  $(\mathrm{lr} = 0.0002, \beta_{1} = 0.9, \beta_{2} = 0.99)$  is used with 25 epochs for training. In this stage, SALICON, which contain labels for both saliency prediction and object recognition, is used for training. More details can be found at Appendix B<sup>1</sup>.
3.2 Model evaluation with human gaze behavior and fMRI responses
We use two criteria to evaluate how well a model matches the brain given naturalistic and dynamic visual stimuli. First, the model should generate similar human visual behaviors, such as visual perception and gaze behavior. Second, the model's internal responses to the stimuli should predict the brain's responses to the same stimuli through linear projection implemented as linear encoding models [69]. For our dual-stream model, we hypothesize that WhereCNN better predicts dorsal-stream voxels and WhatCNN better predicts ventral-stream voxels.
For this purpose, we use a publicly available fMRI dataset from a prior study [70], in which a total of 11 human subjects (4 females) were instructed to watch the movie Raiders of the Lost Ark (115 minutes) with unconstrained eye movements. The movie was displayed on an LCD projector with a visual angle of  $17^{\circ} \times 22.7^{\circ}$ . Whole-brain fMRI data was acquired in a 3-T MRI system with a gradient-recalled echo planar imaging sequence (TR/TE = 2.5s/35ms, flip angle =  $90^{\circ}$ , nominal resolution = 3mm × 3mm × 3mm). We preprocess the data by using the minimal preprocessing pipeline released by the Human Connectome Project (HCP) [71]
We test how well the model can predict the voxel-wise fMRI response to the movie stimuli through a learnable linear projection of artificial units in the model. To evaluate whether and how the two branches in the model differentially predict the two streams in the brain, we define two encoding
<sup>1</sup>The code is available at https://github.com/minkyu-choi04/DualStreamBrains/
models for each voxel: one based on WhereCNN and the other based on WhatCNN. We train and test the encoding models with data during different segments of the movie. To avoid overfitting, we apply dimension reduction to the internal responses in either WhereCNN or WhatCNN by applying principal component analysis (PCA) first to each layer and then to all layers while retaining  $99\%$  of the variance [30, 33]. We further convolve the resulting principal components with a canonical hemodynamic response function (HRF) that peaks at 5 seconds, down-sample them to match the sampling rate of fMRI, generating the linear regressors used in the encoding model. Using the training data  $(81\%$  of the total data), we estimate the encoding parameters using L2-regularized least squares estimation. Using the held-out testing data  $(19\%)$ , we test the encoding models for their ability predicting the fMRI responses observed at each voxel and measure the accuracy of prediction as the correlation between the predicted and measured fMRI responses, denoted as  $r_{where}$  and  $r_{what}$  for the encoding models based on WhereCNN and WhatCNN. We test the significance of the prediction using a block permutation test [72] with a block size of 20-seconds and 100,000 permutations and apply the false discovery rate (FDR)  $(p < 0.05)$ . We further differentiate the relative roles of the brain's WhereCNN vs. WhatCNN in predicting the brain's dorsal and ventral streams for single voxels as well as regions of interest. For this, we define a relative performance (Eq.3).
$$
p _ {w h e r e} = \frac {r _ {w h e r e} ^ {2}}{r _ {w h e r e} ^ {2} + r _ {w h a t} ^ {2}} \tag {3}
$$
In the range from 0 to 1,  $p_{\text{where}} > 0.5$  indicates better predictive performance by WhereCNN, while  $p_{\text{where}} < 0.5$  indicates better predictive performance by WhatCNN.
3.3 Alternative models and control experiments
By design, the WhereCNN and WhatCNN branches within our model exhibit two key distinctions. WhereCNN is specifically trained to learn spatial attention by utilizing wider views, while WhatCNN focuses on object recognition through the use of local views. To explore the impact of input views and learning objectives on the model's capacity to predict brain responses, we introduce two modified control streams: ControlCNN-a and ControlCNN-b, designed as hybrid variants encapsulating diverse input views and learning objectives. ControlCNN-a receives a narrower view as its input and is trained for saliency prediction. Conversely, ControlCNN-b is curated to accommodate a broader view and primarily focuses its training on object recognition. These adjustments yield a versatile examination of their respective influences and functionalities. By combining WhereCNN or WhatCNN with ControlCNN-a or ControlCNN-b, we create four alternative dual-stream models (illustrated in Fig.4) and examine their abilities to explain the functional segregation of the brain's dorsal and ventral streams.
4 Results
4.1 WhereCNN learns attention and WhatCNN learns perception
The WhereCNN and WhatCNN branches in our model are specifically designed to fulfill different objectives: predicting human visual saliency and recognizing visual objects, respectively. In Fig.2, we present examples comparing human attention with the model's attention based on the SALICON's validation set. WhereCNN can successfully identify salient locations where humans are more likely to direct gaze. Additionally, WhereCNN can mimic human saccadic eye movements by generating a sequence of fixations that navigate the model's attention to those salient locations. In contrast, WhatCNN can recognize either single or multiple objects (macro F1 score on MSCOCO's validation set: 61.0).
4.2 WhereCNN and WhatCNN matches dorsal and ventral visual streams