Your head is there to move you around: Goal-driven models of the primate dorsal pathway

1 Introduction . The mammalian visual cortex is organized into two processing streams [ 1 ] : the ventral stream , where neurons are selective for object class and identity ; and the dorsal stream , where neurons are selective for motion . Neurons in the ventral stream exhibit selectivity for increasingly complex stimulus features at successive stages , from oriented lines in V1 , to textures in V2 , curved lines in V4 , and culminating in representations of natural objects in the inferotemporal ( IT ) cortex [ 2 , 3 , 4 , 5 ] . The myriad response properties within and across stages have been difficult to understand computationally [ 6 ] . However , in recent years , a large body of work [ 7 , 8 , 9 , 10 , 11 , 12 , 13 ] has found that modern convolutional neural networks trained on image classification develop representations that match those found in the ventral stream . Early CNN layers match primary visual cortex ( V1 ) , while higherlevel layers better match higher-level ventral stream areas , both in terms of qualitative preferred 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . features [ 12 ] and quantitative predictions of responses to arbitrary stimuli [ 13 , 14 ] . Moreover , models which perform well on ImageNet image classification tend to explain a larger proportion of the variance in ventral stream area IT [ 8 ] . It has recently been found that high-performing networks can emerge through more biologically plausible self-supervised training [ 15 , 16 ] . These results make it possible to interpret the sometimes baffling data about neural responses in the ventral stream in terms of a biologically plausible distributed learning algorithm whose goal is to develop invariant representations that can support object recognition behavior [ 9 , 17 ] . Although this approach has been similarly fruitful in other domains ( e.g. , audition [ 18 , 19 ] ) , it has not yet been applied to the dorsal visual pathway . From physiological recordings in dorsal stream areas like MT and MST [ 20 , 21 , 22 ] , we know that neurons in this pathway are exquisitely selective for motion and increase in receptive field size and complexity along their hierarchy . These properties have inspired different conceptions of dorsal pathway function , including action recognition [ 23 , 24 ] , prediction of image sequences [ 25 ] , and tracking of object motion [ 26 ] , to name just a few . At present , there is no way to know which , if any , of these proposals is correct . We hypothesized that dorsal pathway representations emerge from a simple objective : the need for the organism to orient itself during self-motion . As animals move through the world , they must estimate the parameters of their own motion , in order to avoid collisions , to plan trajectories , and to stabilize their gaze on objects of interest ; the latter is critical for maintaining visual acuity . We suggest that this can be accomplished by learning , in a self-supervised way , the relationship between retinal images and self-motion parameters inferred from oculomotor and vestibular signals that exist in the brain [ 27 ] [ 28 ] . To test this hypothesis , we trained a 3D ResNet to predict the parameters of simulated self-motion - walking speed and head rotation – in short sequences of motion through simulated environments . We found that this network , dubbed DorsalNet , learned motion representations that were qualitatively similar to those found in the dorsal visual stream . Specifically , units were tuned for local motion direction in the earliest layers , object motion in intermediate layers , and complex optic flow in the highest layers [ 29 ] . To test our hypothesis quantitatively , we built a database of neural recordings from different regions of the dorsal visual pathway in non-human primates [ 14 ] . We then compared the ability of different networks to explain responses in areas V1 , MT , and MST . We found that DorsalNet consistently outperformed 3D ResNets trained on action recognition in a supervised manner . Both the self-motion estimation objective and the training stimulus seemed to be critical , since 3D ResNets trained with a predictive objective [ CPC ; 30 ] or supervised on action sequences showed weaker performance . Thus , we demonstrate that the diverse neural response properties in the dorsal stream can be captured by a network that has the goal of estimating self-motion parameters from natural image sequences , both elucidating the functional role of the dorsal stream and creating a best-in-class , in-silico model of the dorsal stream . 2 Background and related work . Dorsal stream processing The dorsal stream - also known as the where pathway - is a network of cortical areas that are selective for visual motion ( Figure 1A ) . It originates in primary visual cortex ( area V1 ) with a subpopulation of neurons that respond selectively to oriented edges moving in a particular direction . These cells project to areas MT/V5 [ 27 , 31 ] , where most neurons respond selectively to motion direction , even for relatively complex stimuli comprised of multiple edges or features [ 21 , 32 ] . These neurons in turn project to area MST , where many neurons are selective for the kinds of complex motion patterns that arise during locomotion [ 22 ] . MST is considered the terminal stage of the dorsal stream , with subsequent areas integrating information from other senses to support diverse roles in action recognition [ 23 ] , decision-making [ 33 ] , and spatial memory [ 34 ] . Models of the dorsal stream Previous models of the dorsal stream have emphasized different possible functions . Giese and Poggio [ 23 ] have argued that the progression of selectivity along the pathway is well-suited to the recognition of biological movements , and this is consistent with studies showing that the ability to identify shapes from motion patterns is disrupted by lesions to area MT [ 35 ] . Other models have posited a role for the dorsal pathway in segmenting moving objects [ 36 , 37 ] , predicting future image frames [ 25 ] , or supporting reaching movements [ 38 ] . Finally , a body of computational [ 39 , 40 ] and experimental [ 41 ] work has analyzed the potential role of dorsal stream neurons in the perception of heading or path [ 42 ] . None of these models has been quantitatively compared to the detailed properties of neural responses in the dorsal stream . Other models have made this kind of comparison , but they have been based on shallow architectures and fit directly to the data from dorsal stream areas , including V1 [ 43 ] , MT [ 44 , 45 , 46 ] and MST [ 47 ] . Although these models shed light on the mechanisms by which neurons attain their stimulus selectivity , they do not relate in any clear way to the functional hypotheses mentioned above . We have therefore attempted to link the properties of dorsal stream neurons , obtained from a database of recordings in non-human primate cortex , to specific functional objectives hypothesized in previous work . In this sense our work is in line with the goals of BrainScore [ 14 ] , which seeks to benchmark ANNs by their ability to explain ventral stream neurons , and to recent work examining self-supervised networks ’ fits to ventral areas [ 16 , 15 ] . 3 Methods . Training network for self-motion We generated a dataset consisting of short videos ( 10 frames ) of self-motion in AirSim , a package for drone and land vehicle simulations in Unreal Engine [ 48 ] . These videos simulated walking along linear trajectories with constant head rotations in two environments ( Figure 1B ) , starting at random positions , varying environmental conditions , hour of day , starting head pose speed and walking speed ( Table S1 in the Appendix ) . Sequences that led to collisions with the environment were removed . We trained a 6-layer 3D ResNet ( layer definitions in table S1 ) to predict 2 of the components of head rotation ( yaw and pitch rotation speed ; roll was not simulated ) and the 3 components of linear velocity ( parametrized as yaw and pitch heading and speed ) . We chose a 3D ResNet architecture over alternatives for its stable training , wide use in video tasks [ 49 , 50 ] , and the high performance of 2D resnets in modeling the ventral stream [ 14 ] . We discretized each component into 72 bins and trained the network with a cross-entropy objective for each of the 5 components . We used the Adam optimizer with a step size of 0.003 , batch norm , and trained for 100 epochs . Neural datasets Datasets are listed in Table 1 . All experiments were conducted in non-human primates ( macaca fascicularis and macaca mulatta ) and were approved by the governing IRB ; detailed experimental procedures are available in the corresponding publications . Data are used under the license terms listed on crcns.org or by permission from the authors [ 47 ] . Methods varied from dataset to dataset , but generally , non-human primates were instructed to fixate on a small target while a contiguous image sequence was presented for several minutes . In some cases , parts of the image sequence were repeated when fixation was lost . Image sequences consisted of color movies , black-and-white movies , static pictures with simulated motions , and random dot kinematograms . Data was collected using single electrodes or multi-electrode arrays . Where available , we used previously published sorted spikes ; when spikes were unsorted , we used multi-unit activity . We split each dataset into a train and test set [ 51 ] ; when only a subset of these stimuli were repeated several times , or a dataset had a designated test subset , we used this subset as the test set ; in other cases , we split the data into 6-second blocks , and concatenated every 10th block to form a test set . We kept the sampling rate of the image sequence at its natural rate and resampled neural activity at the same rate , indicated in Table 1 . We resampled all stimuli spatially to 112x112 . In control analyses , we resampled the input to 74x74 or 168x168 to measure the sensitivity of the results to scale . In the case of [ 47 , 44 ] , the seeds originally used to determine the exact location of dots in the random dot kinematograms were lost , hence we regenerated stimuli with dots in different locations ; it should be noted that this could limit the maximal performance of networks [ 52 ] . All data used in this paper has been previously published ; we release preprocessing scripts and PyTorch loaders to facilitate replication . Aligning ANNs and neural activity We computed latent representations at different layers of the target ANNs , listed in Table 2 , in windows of 10 image frames preceding neural activity . We cropped the first and last latent activity frame and downsampled the activity 2-fold temporally to obtain 4 frames of latent representations preceding the neural activity , and spatially averaged and downsampled each layer output to 8 by 8 . Following [ 14 ] , we kept the first 500 PCs of the intermediate representation and used ridge regression to find mappings from latent space to experimental neural activity . We selected the ridge parameter using 5-fold cross-validation within the train set . Where test sets with 5 or more disaggregated repeats were available , we report an R score normalized against the maximum attainable R score [ 53 ] ; otherwise , we report the raw R score . In control analyses , we replaced ridge regression with a sparse regression estimated through boosting . To fit all the intermediate representations in memory and fit a boosted regression model , we used two different strategies to reduce the memory footprint : downsampling layer outputs ( with spatial averaging as in the linear regression ; table S4 in the appendix ) or subsampling ( without spatial averaging ; Table S4 ) . We selected the number of boosting iterations using 5-fold cross-validation within the train set . We fit these models on a commodity GPUs including P5000 and 1080Ti locally , in Paperspace and in AWS for a total of ∼ 1000 single-GPU-hours . Model weights and code are available 1 under an MIT license . Contrastive Predictive Coding ( CPC ) CPC is a self-supervised learning algorithm that learns to predict the next latent state of a sequence ( e.g . a video sequence ) given its present and past states . The details of the CPC algorithm can be found in [ 30 ] and [ 54 ] , but we summarize it briefly here . A sequence of video frames ( xt ) are passed as input to a 3D CNN . The CNN output ( zt ) , which is a latent representation of the video sequence , is fed to a recurrent neural net ( RNN ) . The RNN aggregates past and present latent states ( i.e . CNN output ) and generates a context variable as its output ( ct ) . The context variable is then passed to a single layer MLP which predicts the future latent state of the video . The predicted latent state and the true latent state ( positive pairs ) , along with some incorrect examples of the next state ( negative pairs ) are given to a contrastive loss function . Minimizing the contrastive loss maximizes the similarity of the predicted and the true next states , and minimizes the similarity of the predicted and the false next states .

Summary: The authors propose to use estimation of self-motion from short video snippets as an objective function to train visual representations resembling those of the dorsal stream of the primate visual system. They show that the first layers of small ResNets trained with this objective develop representations similar to areas V1, MT and MST. According to the authors, alternative objectives such as action classification or contrastive predictive coding or classical baselines like a Gabor filter bank did achieve the same performance on the task of predicting neural responses in the dorsal stream areas – a claim I do not fully agree with; see below. The paper proposes an original idea. It is fairly well executed and well written. I do have some reservations and questions, but I'm overall somewhat supportive of the paper – in particular if the authors can address my concerns.   ### Update after discussion period The authors' response addressed some of my concerns. Unfortunately they promised to report on the control regarding scale but did not post the results, so it's hard to assess this point. I therefore keep my score as borderline accept and hope that the authors will report the results in the final version of the paper.