Your head is there to move you around: Goal-driven models of the primate dorsal pathway
Authors: Patrick J Mineault
Shahab Bakhtiari
Blake Aaron Richards
Christopher C Pack
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: sensory neuroscience
systems identification
goal-driven models
vision neuroscience
motion processing

Abstract: Neurons in the dorsal visual pathway of the mammalian brain are selective for motion stimuli, with the complexity of stimulus representations increasing along the hierarchy. This progression is similar to that of the ventral visual pathway, which is well characterized by artificial neural networks (ANNs) optimized for object recognition. In contrast, there are no image-computable models of the dorsal stream with comparable explanatory power. We hypothesized that the properties of dorsal stream neurons could be explained by a simple learning objective: the need for an organism to orient itself during self-motion. To test this hypothesis, we trained a 3D ResNet to predict an agent's self-motion parameters from visual stimuli in a simulated environment. We found that the responses in this network accounted well for the selectivity of neurons in a large database of single-neuron recordings from the dorsal visual stream of non-human primates. In contrast, ANNs trained on an action recognition dataset through supervised or self-supervised learning  could not explain responses in the dorsal stream, despite also being trained on naturalistic videos with moving objects. These results demonstrate that an ecologically relevant cost function can account for dorsal stream properties in the primate brain.

Your head is there to move you around: Goal-driven models of the primate dorsal pathway
Patrick J Mineault
Shahab Bakhtiari Mila, McGill University bakhtias@mile la. quebec
patrick mwneaul t@gmail....c..ooo com
Blake A Richards
Mila, Montreal Neurological Institute, McGill University bl Blake.1 ri richards@hiillaaaa quebec
Christopher C Pack
Montreal Neurological Institute, McGill University christopher. pack@mcgill.ca ctristopher.packkmnggl....
Abstract
Neurons in the dorsal visual pathway of the mammalian brain are selective for motion stimuli, with the complexity of stimulus representations increasing along the hierarchy. This progression is similar to that of the ventral visual pathway, which is well characterized by artificial neural networks (ANNs) optimized for object recognition. In contrast, there are no image-computable models of the dorsal stream with comparable explanatory power. We hypothesized that the properties of dorsal stream neurons could be explained by a simple learning objective: the need for an organism to orient itself during self-motion. To test this hypothesis, we trained a 3D ResNet to predict an agent's 's self-motion parameters from visual stimuli in a simulated environment. We found that the responses in this network accounted well for the selectivity of neurons in large database of single-neuron recordings from the dorsal visual stream of non-human primates. contrast, ANNs trained on action recognition dataset through supervised superviid learning could not explain responses in the dorsal stream, despite also being trained on naturalistic videos with moving objects. These results demonstrate that ecologically relevant cost function can account for dorsal stream properties in the primate brain.
Introduction
The mammalian visual cortex is organized into two processing streams 11] the ventral stream, where neurons are selective for object class and identity; and the dorsal stream, where neurons are selective for motion. Neurons in the ventral stream exhibit selectivity for increasingly complex stimulus features at successive stages, from oriented lines in VI, to textures in V2, curved lines in V4, and culminating in representations of natural objects in the inferotemporal (IT) cortex
2
4 5) The
myriad response properties within and across stages have been difficult to understand computationally 6
However, recent years, large body of work g convolutional neural networks trained image classification develop representations that match those found in the ventral stream. Early CNN layers match primary visual cortex (V1), while higher- level layers better match higher-level ventral stream areas, both in terms of qualitative preferred
1 has found that modern
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

features [12 and quantitative predictions of responses to arbitrary stimuli [4). Moreover, models which perform well on ImageNet image classification tend explain a larger proportion of the variance in ventral stream area IT 8 It has recently been found that high-performing networks can emerge through more biologically plausible self-supervised training [15] 16]. These results make it possible to interpret the sometimes baffling data about neural responses in the ventral stream in terms of a biologically plausible distributed learning algorithm whose goal is to develop invariant representations that can support object recognition behavior ( [7].
Although this approach has been similarly fruitful in other domains (e.g., audition [8]
19 has not yet been applied to the dorsal visual pathway. From physiological recordings in dorsal stream areas like MT and MST [20] 21 [22], we know that neurons in this pathway are exquisitely selective for motion and increase in receptive field size and complexity along their hierarchy. These properties have inspired different conceptions of dorsal pathway function, including action recognition 23 24 prediction of image sequences (25], and tracking of object motion [26], to name just few. At present, there is no way to know which, any, of these proposals is correct.
We hypothesized that dorsal pathway representations emerge from a simple objective: the need for the organism to orient itself during self-motion. As animals move through the world, they must estimate the parameters of their own motion, in order to avoid collisions, to plan trajectories, and to stabilize their gaze on objects of interest; the latter is critical for maintaining visual acuity. We suggest that this can be accomplished by learning, in a self-superviser way, the relationship between retinal images and self-motion parameters inferred from oculomotor and vestibular signals that exist in the brain 27 [28]. To test this hypothesis, we trained a 3D ResNet to predict the parameters of simulated self-motion walking speed and head rotation in short sequences of motion through simulated environments. We found that this network, dubbed DorsalNet, learned motion representations that were qualitatively similar to those found in the dorsal visual stream. Specifically, units were tuned for local motion direction in the earliest layers, object motion in intermediate layers, and complex optic flow in the highest layers (29).
To test our hypothesis quantitatively, we built a database of neural recordings from different regions of the dorsal visual pathway in non-human primates [14). We then compared the ability of different networks to explain responses in areas VI, MT, and MST. We found that DorsalNet consistently outperformed 3D ResNets trained on action recognition in supervised manner. Both the self self-motion estimation objective and the training stimulus seemed to be critical, since 3D) ResNets trained with predictive objective [CPC: 30 or supervised on action sequences showed weaker performance. Thus, we demonstrate that the diverse neural response properties in the dorsal stream can be captured by network that has the goal of estimating self-motion parameters from natural image sequences, both elucidating the functional role of the dorsal stream and creating best-in-class, in-silico model of the dorsal stream.
2 Background and related work
Dorsal stream processing The dorsal stream- also known as the where pathway- - is a network of cortical areas that are selective for visual motion (Figure [TA). It originates in primary visual cortex (area VI) with a subpopulation of neurons that respond selectively to oriented edges moving in particular direction. These cells project areas MT/V5 (27 [1], where most neurons respond selectively to motion direction, even for relatively complex stimuli comprised of multiple edges or features 21 32]. These neurons in turn project to area MST, where many neurons are selective for the kinds of complex motion patterns that arise during locomotion (22). MST is considered the terminal stage of the dorsal stream, with subsequent areas integrating information from other senses to support diverse roles in action recognition (23 decision-
33 and spatial memory (34].
Models of the dorsal stream Previous models of the dorsal stream have emphasized different possible functions. Giese and Poggio 23 have argued that the progression of selectivity along the pathway is well-suited to the recognition of biological movements, and this is consistent with studies showing that the ability to identify shapes from motion patterns is disrupted by lesions to area MT (5 Other models have posited a role for the dorsal pathway in segmenting moving objects [36. 37). predicting future image frames (25). or supporting reaching movements [38]. Finally, a body of computational [39 40 and experimental [1] work has analyzed the potential role of dorsal stream

---

neurons in the perception of heading or path 44]. None of these models has been quantitatively compared to the detailed properties of neural responses the dorsal stream.
Other models have made this kind of comparison, but they have been based on shallow architectures and fit directly to the data from dorsal stream areas, including V1
[ MT
44
45
46] and MST
[4]] Although these models shed light on the mechanisms by which neurons attain their stimulus selectivity, they do not relate in any clear way the functional hypotheses mentioned above.
We have therefore attempted to link the properties dorsal stream neurons, obtained from a database of recordings in non- non-human primate cortex, specific functional objectives hypothesized in previous work. In this sense our work is in line with the goals of BrainScore [14,, which seeks to benchmark ANNs by their ability to explain ventral stream neurons, and to recent work examining self self-suue networks' fits to ventral areas [6 15
3 Methods
Training network for self-motion We generated a dataset consisting of short videos (10 frames) of self-motion in AirSim, a package for drone and land vehicle simulations in Unreal Engine [48]. These videos simulated walking along linear trajectories with constant head rotations in two environments (Figure B3), starting at random positions, varying environmental conditions, hour of day, starting head pose speed and walking speed (Table ST in the Appendix). Sequences that led to collisions with the environment were removed.
We trained 6-layer 3D ResNet (layer definitions in table SI) to predict 2 of the components of head rotation (yaw and pitch rotation speed; roll was not simulated) and the components of linear velocity (parametrized as yaw and pitch heading and speed). We chose a 3D ResNet architecture over alternatives for its stable training, wide use in video tasks [49 50). and the high performance of 2D resnets in modeling the ventral stream [14]. We diseretized each component into 72 bins and trained the network with a cross cross-entropy objective for each of the components. We used the Adam optimizer with a step size of 0.003, batch norm, and trained for 100 epochs.
Neural datasets Datasets are listed in Table 1 All experiments were conducted in non-human primates (macaca fascicularis and macaca mulatta) and were approved by the governing IRB; detailed experimental procedures are available in the corresponding publications. Data are used under the license terms listed on crens.org or by permission from the authors [47] Methods varied from dataset to dataset, but generally, non-human primates were instructed fixate a small target while a contiguous image sequence was presented for several minutes. In some cases, parts of the image sequence were repeated when fixation was lost. Image sequences consisted of color movies, black-and-whit movies, static pictures with simulated motions, and random dot kinematograms. Data was collected using single electrodes or multi-electrode arrays. Where available, we used previously published sorted spikes; when spikes were unsorted, we used multi-unit activity.
We split each dataset into train and test set [51; when only subset of these stimuli were repeated several times, or a dataset had a designated test subset, we used this subset as the test set; in other cases, split the data into 6-second blocks, and concatenated every 10th block to form test set. We kept the sampling rate of the image sequence at its natural rate and resampled neural activity at the same rate, indicated in Table I We resampled all stimuli spatially to 1 112x112 12. In control analyses. we resampled the input to 74x74 or 168x168 to measure the sensitivity of the results to scale. In the case of 44 44l the seeds originally used to determine the exact location of dots in the random dot kinematograms were lost, hence we regenerated stimuli with dots in different locations; it should be noted that this could limit the maximal performance of networks [52]. All data used in this paper has been previously published; we release preprocessing scripts and Py Torch loaders to facilitate replication.
Aligning ANNS and neural activity We computed latent representations at different layers of the target ANNs, listed in T in windows of 10 image frames preceding neural activity. We cropped the first and last latent activity frame and downsampled the activity 2-fold temporally to obtain 4 frames of latent representations preceding the neural activity, and spatially averaged and downsampled each layer output to by 8. Following [4), we kept the first 500 PCs of the intermediate representation and used ridge regression to find mappings from latent space experimental neural
3

---

activity. We selected the ridge parameter using 5-fold cross-validation within the train set. Where test sets with 5 or more disaggregated repeats were available, we report an score normalized against the maximum attainable R score [53]; otherwise, we report the raw R score.
In control analyses, we replaced ridge regression with a sparse regression estimated through boosting. To fit all the intermediate representations in memory and fit a boosted regression model, we used two different strategies reduce the memory footprint: downsampling layer outputs (with spatial averaging as in the linear regression; table S4 in the appendix) or subsampling (without spatial averaging; Table S4). We selected the number of boosting iterations using 5-fold cross- coossvvaliattion within the train set. We fit these models on a commodity GPUs including P5000 and 1080Ti locally, in Paperspace and in AWS for a total of 1000 Linll-GPU--hours. Model weights and code are available under an MIT license.
Contrastive Predictive Coding (CPC) CPC is self- learning algorithm that learns to predict the next latent state of a sequence (e.g. a video sequence) given its present and past states. The details of the CPC algorithm can be found in [30] and [54), but we summarize it briefly here. A sequence of video frames (2t) are passed as input to a 3D CNN. The CNN output (2t). which is a latent representation of the video sequence, is fed to recurrent neural net (RNN). The RNN aggregates past and present latent states (i.e. CNN output) and generates a context variable as its output (c.). The context variable is then passed to single layer MLP which predicts the future latent state of the video. The predicted latent state and the true latent state (positive pairs), along with some incorrect examples of the next state (negative pairs) are given to contrastive loss function. Minimizing the contrastive loss maximizes the similarity of the predicted and the true next states, and minimizes the similarity of the predicted and the false next states.
4 Results
4.1
3D resnets trained for sell-motion learn dorsal-like representations
We hypothesized that learning to estimate self-motion from visual inputs would lead to dorsal stream- like representations. As in the ventral stream, these representations begin in VI with receptive fields that encode simple, local features of stimuli. Through subsequent recombinations at different layers, more complex and ecologically relevant encoding emerges. To test this hypothesis, we generated self-motion videos in simulation environment, and trained a 3D ResNet to predict its self-motion parameters, namely head rotation and linear locomotion (see Methods for details).
Qualitative matches to the dorsal stream The 6-layer 3D ResNet trained in this way learned representations similar to single units in the primate dorsal stream. We focus our attention here on layers 1, 2 and 3 of the network. Preferred features of layer contained many spatiotemporally slanted filters (Figure which are the building blocks of motion selectivity in primate V1 (55). We quantified this slant with the separability index of/ Σ; of from the singular values of the grayscale filters o; this matched values reported in the literature for V1 [20] [.22 +/-.16 for trained network, .71 +/- .15 in real VI neurons; figure iI the appendix].
To gain insight into the stimulus selectivity of these representations, we generated optimal stimuli for individual units in intermediate layers of the network by optimization [12]; we present static images of the intermediate preferred frame here, while animations can be visualized on the companion websita") Probed in this fashion, many intermediate features in layer 1 preferred what looked like drifting gratings (examples in Figure [D), consistent with the selectivity of VI cells [5 (6). Hence, to further probe the selectivity of these units, used full contrast, drifting gratings of different spatial and temporal frequencies, placed in the center of the visual field. Tuning curves in layer 1 (samples in [ tended to have bias towards direction selectivity, with mean circular variance at the preferred spatial and temporal frequency of 0.75 and a median direction selectivity index- defined as 1 T / STantipre on the centered tuning curves- of 0.98. This is somewhat higher than is typically found in V (57). likely due to lack of noise, but it is close to the selectivity of the V1 neurons that actually project to higher levels of the dorsal visual pathway (58).
-oworo---o--rouunddd around
app

---

A
B
C
fiters
semporal oriemtaton
JMII-BTT
Image sequeenca
D
layer
E
layer
layer
(edlaed kpold
Laki DURO
pedloce solkd
Flod saggaan
Speee wer
procect DAR
prodand spaiic
pediece
Figure I: A 3D ResNet trained for self-motion estimation learns dorsal-like representations. A. The goal of this study is to model the dorsal visual stream, including V1, MT and MST. B. The 3D Resnet model trained estimate self-motion parameters from image sequences. C. Weights of the first layer. First layer filters are selective in space-time time D. Sample tuming of layer features. Layer contains many direction-selectivv cells reminiscent of V1. Sample tuning of layer 2 features. Many layer 2 features exhibit tuning for rigid motion, similar to MT. F. Sample tuning curve of layer features. Many cells layer are tuned for complex optic flow, like MST
Layer 2 units tended to prefer more spatially broadband moving stimuli, not unlike the plaids conven- tionally used in probing MT cells [21] (Figure II. left column). Indeed, probing the representations with sums of gratings revealed similar selectivity to a single grating in subset of cells (Figure IE. middle column; pattern selectivity plots in Figure [ST in the appendix). These cells likely encode stimulus velocity in manner that is invariant of the composition of the stimulus (21). Like MT cells, subunits in this layer tended to be highly direction selective, with the average circular variance of the direction tuning curves being .41.
MT cells are also known to be selective for stimulus speed, which the ratio of temporal to spatial frequencies (59]. A similar kind of selectivity emerged in layer 2 of the model, where many units preferred higher temporal frequencies when the spatial frequencies were higher (example tuning curve in Figure [... To quantify this selectivity, probed the model units with range of spatiotemporal frequencies and fit the data with slanted Gaussian functions [60], which revealed a mean speed selectivity index of 14 in layer 1, compared to 0.58 in layer 2, the latter being similar the value of 0.52 reported in MT [60]. Probing layer 2 units with moving dots, we found a majority of neurons with simple receptive fields that prefer linear motion, with smaller number of complex receptive fields (Figure TE, bottom right).
Finally, we found many cells in layer 3 that combined the outputs of lower-level units to generate selectivity for more complex motion patterns (example cells in Figure Dot pattern probes revealed selectivity for rotations, spirals or single axis expansion. As in primate area MST, these units tended
5

---

R relative to maximum for area
1.0
0.8
0.6
mst
0.4
mt
v1
0.2
0.0
layer
Figure Layers 2, and of DorsalNet best match areas V1, MT and MST. Lines show correlation
(R) relative to maximum for area. Horizontal lines: 95% CI of layer with maximal alignment to area.
emphasize expansion motion rather than contraction, similar the bias experienced during forward navigation (Figure
61
Area
Dataset
Data
Sampling
Stimulus
V1
crens-pvel
23 multi-units
30Hz
Color movies
crens-pve4
66
25 single units
75Hz
B&W movies
MT
crens-mtl
44
67
88 single units
30Hz
optic flow kinematograms
crens-mt2
[46
68
44 single units
83Hz
B&W motion-enhanced movies
MST
packlab-mst
47
36 single units
30Hz
optic flow kinematograms
Table Datasets
Regression analysis of representations Given that the trained network recapitulated many qual- itative properties of the dorsal stream, we next investigated whether they quantitatively matched dorsal stream areas, for which ninglenneuron data was available. We used ridge regression to learn mapping from latent representations at each layer of the network single neural responses to complex stimuli, including black and white and color movies, along with random dot kinematograms (See Methods and T details). We learned separate mapping for each layer of the network. allowing us to match the depth of the network to each brain area. As seen in Figure this showed hierarchical progression, with ligeer-level cortex matching higher-level layers in the ResNet. The average best matching layer across cells with report correlation greater than ,01, illustrated by the horizontal lines, was 1.1 for VI cells [00.9 1.5) 95% CI, bootstrap across cells], 2.0 for MT cells [(1.8, 2.3)] and 2.9 for MST cells [(2.3, 3.4)].
We noticed that the mapping was less distinct for layer We investigated this further by measuring the response of the network to gauntlet of stimuli from the Airsim dataset. Centered kernel alignment (CKA) 669 revealed that while layers 1, and 2 had highly distinct representations, subsequent layers were less distinct S3, We also investigated the robustness of the mapping to a change of scale
of the input sequences 770 We saw some minor shifts: the median mean layer assignment for
was 1.0 at 0.66X scale, 1.1 at the standard 1X scale, and 1.3 at 1.5X scale (Figure (S2). Overall, change of scale. Thus, broadly speaking, layers 1, and of the network recapitulated V1, MT and MST, respectively.
however, mappings were robust
4.2 Networks with alternative objectives do not account for responses in the dorsal stream
Action recognition networks To examine the specificity of these results, we tested other networks trained with different objective functions. Action recognition is popular computer vision task.

---

Category
Name
Dataset
License
Notes
SlowFast
slowfast
Kinetics400
Apache
Fast branch only
i3d
R3D 50
r3d 18
Kinetics400
BSD
12plus__18
me3_18
CPC
epc_ucf
UCFI01 Airsim
own work
R3D with 10 res blocks R3D with 10 res blocks
epe_ani _e__irsim
Gabors 46
gabor
work
gabor nomotion
Opposite dirs averaged
MotionNet 26
motionnet
shifted images
CCBY4.0
DorsalNet
dorsalnet
Airsim
own work
R3D with 4 res blocks
Table 2: Models tested
MT mtl
MST mst
pvcl
pvc4
mt2
slowfast i3d
.471 (.034)
.361 (.042)
.211 (.018)
.281 (.015)
.189 (.044)
.457 (.036)
.389 (.046)
.213 (.018)
.284 (.015)
.219 (.044)
r3d r3d_18
.403 (.032)
.383 (.042)
.217 (.018)
.899 (.015)
.224 (.046)
r2plustd_ns
.428 (.035)
.382 (.042)
.215 215((018)
282 (.015)
.226 (.043)
mc3_18
.455 (.034)
.393 (.045)
.288 (.018)
.76 (.014)
.228 (.045)
epc_ucf
.271 (.044)
.394 (.046)
(.188
.411 (0016)
.190 (.045)
epe_airsim
.422 (.036)
.384 (.045)
.250 (.020)
360 (.017)
.992 (.045)
Labor_nomotion
.73 (.035)
.353 (.038)
.212 (.018)
.188 (.014)
.248 (.045)
gabor
.355 (.036)
.366 (.037)
.249 (.019)
300 (.015)
.394 (.054)
motionnet
276 (.042)
.364 (.039)
.238 (.018)
.333 (.016)
.441 (.053)
dorsalnet
.364 (.043) .370 (.039)
.251 (.019)
.381 (.017)
.454 (.054)
Table DorsalNet quantitatively performs best across the dorsal stream. Table shows normalized pearson correlation (R; see Methods for definition) of different models on different datasets. In parenthesis: standard error of the mean over cells.
and tested 3D ResNets trained Kinetics400 (71]. These networks performed admirably in explaining VI responses, reaching an average R on the pvel dataset. However, across our MT and MST datasets, performance
poor, failing to exceed that of null model [72] consisting of 3D Gabor pyramid (Table 3). We note that only a small fraction of V1 neurons project to the dorsal stream, with the majority projecting to ventral stream areas; interpret the relative performance in V1 vs. MT and MST as a sign that these networks learned representations more aligned with the ventral stream, supporting object recognition and by extension action recognition. Consistent with this interpretation, we found that the first layer of 3D ResNets trained for action recognition did not learn motion in the traditional sense (Figure (S)) Instead, their filters were mostly separable in space and time, meaning they were not selective for motion energy per se.
CPC Our results indicate that learning estimate self-motion in simulated environment creates representations similar to those in the primate dorsal stream. The neural network architecture (3D ResNets) was similar for the self-motion estimation objective and action recognition tasks. However, both the task prediction of self-motion parameters and the stimulus ensemble self- motion sequences in Airsim environment differed. To tease apart relative importance of these two factors, we tested the ability of contrastive predictive coding (CPC) networks (30] to account responses the dorsal stream when trained over different stimulus ensembles. CPC is self-supervised training method that finds predictive latent representations that can distinguish
7

---

between image sequences. Importantly, it is possible to apply the CPC objective to different stimulus ensembles, thereby differentiating between task and stimulus ensemble effects. We trained an 11 11-layer network with CPC objective on the UCF101 dataset and our Airsim dataset. The Airsim- trained network performed significantly better than the UCF101-trained network, approaching the performance of DorsalNet in MT but not in MST. Examining first layer filters revealed direction- selective receptive fields after training on the Airsim dataset but not with UCF101 (Figure (55). This is consistent with the training set being necessary, though not sufficient, to match primate dorsal stream neurons.
MotionNet We next tested much simpler 2-layer network from the neuroscience literature, which was trained to estimate the linear motion of black and white image patches [26). The original model was a fully connected architecture working on small image patches, and we made it convolutional by tiling. We used the checkpoints shared by the authors as the model weights. This model had not previously been directly benchmarked against neural data, and given the small size of its stimulus ensemble, we did not expect it to perform well. Surprisingly, it scored far better in predicting MT and MST responses than action recognition networks (Table 3). We found in a control analysis (Table S3 in the appendix) that the relative performance of MotionNet could be improved still by spatially scaling up the stimulus, matching the performance of Net on out of 3 MT and MST datasets. These results are consistent with solving 2D motion being an important sub-goal of the dorsal stream.
We next asked whether there existed a one-to-one or few-to-one relationship between model subunits and single neurons. Using sparse regression, we found that DorsalNet better matched individual neurons across all MT and MST datasets than MotionNet, regardless of scaling (Tables [S4 and SS in the appendix). Thus, DorsalNet subunits were more directly aligned to single neurons across the dorsal stream.
Self-motion estimation performance correlates with dorsal stream match
Across our baselines, there was a large range in the ability of different models to reproduce dorsal stream data. We asked whether this heterogeneity could be linked to performance on a self-motion estimation task. We froze the weights of our baseline networks and trained linear decoders to estimate self-motion parameters on the AirSim dataset from hidden layer representations. We excluded DorsalNet and trrsie-trained CPC from these comparisons. Across our baselines, there was a highly significant correlation between self-motion estimation performance and match to MT and MST neurons (Figure S44 Table (able in the appendix). Interestingly, when looking at individual self- motion parameters, head rotation estimation accuracy was most correlated with performance on MT and MST datasets. Thus, those networks which happen to be best at self-motion estimation, especially head rotation, can best explain responses in the dorsal stream, consistent with formative role of self-motion estimation in dorsal stream representations.
6 Limitations
Multiple interpretations We show that learning to estimate onc's self-motion from visual cues leads to representations which are similar to those of the dorsal stream. We benchmark against several other candidate models, including localized frequency detectors, which form a sparse basis for images [73), predictive coding models, models trained for action recognition, and models trained trained to estimate the motion of small image patches. While DorsalNet performed best overall across the dorsal stream, we found that MotionNet and CPC-based network trained on our AirSim dataset were close contenders. With the available data, we cannot conclusively rule out that these alternative objectives, with the right tweaks, could not account for the data. One interesting possibility is that, as MotionNet hints, solving rigid 2D) motion is a sub-goal of the dorsal stream; and, as DorsalNet shows, the supervisory signal needed to learn solve that sub-goal could come from head movements, especially head rotations, via efference copy. An open benchmark in the style of [14] could reveal other objectives compatible with the data and refine these results.
To the best of our knowledge, we used all of the relevant publicly available non-human primate data for this study. Most of this data was collected more than a decade ago in time-consuming single-clectrode experiments, with electrode drift, loss of fixation, short recording
Data limitations
8

---

times and small numbers of recordings per experiment being sigmificant limitations. The MST dataset in particular not very discriminative across models. Differences in stimuli and number of repetitions make absolute comparisons across areas difficult. Improvements in recording technology as well as better-designed hypothesis-driven studies will allow the collection of more discriminative data in the future. Our study paves the way for closed-loop experiments to verify that the estimated stimuli indeed maximally drive dorsal stream neurons [74 75
7 Discussion
Systems neuroscience aims to explain how the brain solves behavioral tasks at the algorithmic level [17]. While rich literature has linked the ventral visual stream to the task of object recognition, little work has focused on understanding how and why dorsal streams representations emerge. Noting the critical role of self- celf-motion estimation across the animal kingdom [76), hypothesized that training an artificial neural net on self-motion estimation from image sequences would lead to representations similar to the dorsal stream. We verified this qualitatively by probing networks with artificial stimuli and by finding maximizing stimuli. We confirmed these findings quantitatively by benchmarking existing computer vision networks gauntlet of neural data 14.
In the framework of [17], the objective, learning rule and architecture specify how a task is to be solved by an artificial or biological neural network. Implicit in the framework is a fourth critical ingredient: the dataset, or distribution of training examples. Our work focuses on how an objective, learning rule and dataset interact to form representations similar to the dorsal stream. In contradistinction with previous work, we focus on a single architecture of 3D ResNets, a coarse approximation to early and intermediate visual processing stages, highlighting the formative role of objective, learning rule and dataset in the creation of useful representations for action.
Maximizing stimuli reveal selectivity Systems identification has long been used in systems neu-
roscience to estimate preferred stimuli in different brain areas [55] 56 43 46
51
20
47 777 44).
More recently, systems identification has been used better understand mechanisms of selectivity in deep neural nets [12). Given the breadth of available systems identification results in brains, we suggest that systems identification is a particularly powerful tool to relate brains and artificial neural nets, especially when combined with benchmarking: it can offer clues as to why certain networks perform better than others. In this article, we identified direction selectivity in the first layer as a strong clue that networks develop good motion representations.
Action recognition is poorly aligned to the dorsal stream Our work shows that ANNs trained on the standard computer vision task of action recognition fail learn motion representations that correlate with single neurons in MT and MST. [78) reported that on Kinetics400 and UCF101, a single image is sufficient get within 6% of the action recognition accuracy of full image sequence, indicating that motion has a limited role in action recognition in these datasets. Motion selectivity can be reintroduced via parallel optic flow pathway [79] by enforcing that the network reproduce dense optic flow following early layers [80 81]. with modest improvements in classification accuracy. Our benchmarks strongly suggest that current action recognition datasets can be solved without motion and that good motion representations don't 't emerge from supervised learning on them alone.
Self-supervision through cross-modal prediction We train DorsalNet in a supervised way. From the agent's 's perspective, however, corollary discharges of the motor plan are available, well as vestibular inputs. Thus, the objective can be viewed as a self-supervised objective which aims to predict one modality channel of the input from the other, in line with other proxy tasks including Because multisensory integration and corollary self-supervision through mooss-modal prediction could be potentially widely used across species to learn useful representations.
colorization and audiovisual alignment [82 discharges ubiquitous across mobile animals 85
83
84
Evolution and learning in sensory systems Thompson 86 identifies four set of constraints
against which in silico models of sensory systems can be evaluated:
Whether it can perform relevant task
Whether it accounts for neural activity

---

Whether it is biologically plausible
Whether it could have evolved
We presented model of the dorsal stream that trained to estimate self-motion. It accounts for neural responses in 3 different areas, taken from 5 different datasets. The model weights can be learned by the agent through biologically plausible self-supervision, since the approximate parameters of self-motion are known to the agent. via corollary discharges and vestibular and proprioceptive inputs (28]. Self-motion estimation particularly important for gaze stabilization, which evolved in tandem with the earliest visual functions [87 88 and continues to be necessary for visual processing. including that performed in the ventral pathway [89]. Given this evolutionary pressure, some aspects of the dorsal pathway are likely hard-coded in the genome, while others are learned through development [90]; further work will focus on better understanding the relative role of evolution vs. learning in dorsal stream processing. This work and its follow-ups thus have the potential to elucidate long-standing questions about how sensory systems evolved.
Acknowledgments and Disclosure of Funding
This work was supported by a NSERC (Discovery Grant: RGPIN-2020-05100 Discovery Accelerator Supplement: 1010000000000,,,,,, Healthy Brains, Healthy Lives (New Investigator Award: 2b- NISU-8; Innovative Ideas Grant: Ic-II-15), and CIFAR (Canada Al Chair; Learning in Machine and Brains Fellowship). CCP was funded by a CIHR grant (NOP-115178) This work was also funded by the Canada First Research Excellence Fund (CEREF Competition 2015-2016) awarded the Healthy Brains, Healthy Lives initiative at McGill University, through the Helmholtz International BigBrain Analytics and Learning Laboratory (HIBALL).
References
Leslie G. Ungerleider and Mortimer Mishkin. Two cortical visual systems. Analysis of visual behavior, pages 549-586, 1982.
[2] A. Pasupathy and C. E Connor, Population coding of shape in area V4, Nature Neuroscience, 5(12):1332- 1338, 2002.
[3] Nicole C. Rust and James J. DiCarlo. Selectivity and Tolerance ("Invariance") Both Increase as Visual Information Propagates from Cortical Area V4 to IT. The Journal (ott:sciience,303)))11332333 -12995. 2010.
[4] S. L Brincat and C. E Connor. Underlying principles of visual shape selectivity in posterior inferotemporal cortex. Nature Neuroscience, 78))8880-886, 2004.
[5] Jeremy Freeman, Corey M. Ziemba, David J. Heeger, Ecro P. Simoncelli, and J. Anthony Movshon. A functional and perceptual signature of the second visual area in primates. Nature Neuroscience, 16(7):974- 981, July 2013.
[6] Andrew Parker. Intermediate level cortical areas and the multiple roles of area V4, Current Opinion in Physiology, 16:61-67, August 2020.
[7] Daniel Yamins, Ha Hong. Charles Cadieu, and James J. DiCarlo. Hierarchical modular optimization of convolutional networks achieves representations similar to macaque IT and human ventral stream. 2013. Publisher: Neural Information Pracessing Systems Foundation.
[8] Daniel LK Yamins, Ha Hong, Charles F. Cadicu, Ethan A. Solomon, Darren Seibert, and James J. DiCarlo. cerformance-optimize hierarchical models predict neural responses in higher visual cortex. Proceedings of the national academy of sciences, (1(3)):619986222 2014. Publisher: National Acad Sciences.
[9] Daniel LK Yamins and James J. DiCarlo. Using goal-driven deep learning models to understand sensory cortex. Nature neuroscience, 1933)3356-35,, 2016. Publisher: Nature Publishing Group.
[10] Seyed-Mahdi Khaligh-Razavi and Nikolaus Kriegeskorte. Deep supervised, but not unsupervised, models may explain IT cortical representation. PLoS computational biology, 10(1 10003915, 2014. Publisher: Public Library of Science San Francisco. USA.
[11] Grace W. Lindsay. Convolutional neural networks as a model of the visual system: past, present, and future. Journal of cognitive neuroscience, pages 1-15, 2020. Publisher: MIT Press.
10

---

[12] Nick Cammarata, Shan Carter, Gabriel Goh, Chris Olah, Michael Petrov, and Ludwig Schubert. Thread: Circuits. Distill, 5(3):224 March 2020.
[13] Umut Güşlü and Marcel AJ van Gerven. Deep neural networks reveal gradient in the complexity of neural representations across the ventral stream, Journal of Neuroscience, 35(27):10055-0001 2015. Publisher: Soe Neuroscience.
[14] Martin Schrimpf, Jonas Kubilius, Ha Hong, Najib J. Majaj, Rishi Rajalingham, Elias B. Issa, Kohitij Kar, Pouya Bashivan, Jonathan Prescott-Roy Roy, and Kailyn Schmidt. Brain-score: Which artificial neural network for object recognition is most brain-like? BioRxiv, page 407007, 2018. Publisher: Cold Spring Harbor Laboratory.
[15] Talia Konkle and George Alvarez. Deepnets do not need category supervision to predict visual system responses to objects. Journal of Vision, 20(11):498 498, October 2020. Publisher: The Association for Research in Vision and Ophthalmology.
[16] Chengxu Zhuang. Siming Yan, Aran Nayebi, Martin Schrimpf, Michael C. Frank, James J. DiCarlo, and Daniel L. K. Yamins. Unsupervised neural network models of the ventral visual stream. Proceedings of the National Academy of Sciences, 118(3), January 2021. Publisher: National Academy Sciences Section: Biological Seiences.
[17] Blake A. Richards, Timothy P. Lillicrap. Philippe Beaudoin, Yoshua Bengio, Rafal Bogacz, Amelia Christensen, Claudia Clopath, Rui Ponte Costa, Archy de Berker, and Surya Ganguli. A deep learning framework for neuroscience, Nature neuroscience, 22(11):171111770 2019. Publisher: Nature Publishing Group.
[18] Alexander J. Kell, Daniel L. Yamins, Erica Shook, Sam V. Norman-Haainerre, and Josh McDermott. A Task fask-Optimized Neural Network Replicates Human Auditory Behavior, Predicts Brain Responses, and Reveals a Cortical Processing Hierarchy. Neuron, 98(3)6630 644.c16, May 2018.
[19] Alexander JE Kell and Josh H. McDermott. Deep neural network models of sensory systems: windows onto the role of task constraints. Current opinion neurobiology, 55:121-132, 2019. Publisher: Elsevier.
[20] Christopher C. Pack, Bevil R. Conway, Richard T. Bom, and Margaret S. Livingstone. Spatiotemporal structure of nonlinear subunits in macaque visual cortex. Journal of Neuroscience, 260)8993-77 2006. Publisher: Soc Neuroscience.
[21] J. Movshon, E. H. Adelson, M. S. Gizzi, and William T. Newsome. The analysis of moving visual patterns. Puttern recognition mechanisms, pages 117-151, 1985. Publisher: Vatican Press.
[22] Charles J. Duffy and Robert H. Wurtz. Sensitivity of MST neurons to optic flow stimuli. I. A continuum of response selectivity to large-field stimuli. Journal of neurophysiology. 6596):1339-1335,, 1991. Publisher: American Physiological Society Bethesda, MD.
[23] Martin A. Giese and Tomaso Poggio. Neural mechanisms for the recognition of biological movements. Nature Reviews. Neuroscience, 4(3):179- 192, March 2003.
[24] Hucihan Jhuang. Dorsal stream : from algorithm to neuroscience. Thesis, Massachusetts Institute of Technology, 2011. Accepted: 201 0011-09-27718:3133922
[25] Janneke F. M. Jehee, Constantin Rothkopf, Jeffrey M. Beck, and Dana H. Ballard. Learning receptive fields using predictive feedback. Journal of Physiology-Pari, 100(1):125-132 July 2006.
[26] Reuben Rideaux and Andrew Welchman. But still moves: static image statisties underlie how we see motion. Journal of Neuroscience, 40(12):2538 2252,, 2020. Publisher: Soc Neuroscience.
[27] William T. Newsome, Robert H. Wurtz, and Hidehiko Komatsu. Relation of cortical areas MT and MST to pursuit eye movements. II Differentiation of retinal from extraretinal inputs. Journal of neurophysiology. 60(2):604-620, 1988. Publisher: American Physiological Society Bethesda, MD.
[28] Yong Gu, Paul V. Watkins, Dora E. Angelaki, and Gregory C. DeAngelis. Visual and nonvisual contributions to thrce-dimensional heading selectivity in the medial superior temporal area. Journal of Neuroscience, 26(1):73-88 85, 2006. Publisher: Soc Neuroscience.
[29] R. A. Andersen, L. H. Snyder, D. C. Bradley, and J. Xing. Multimodal representation of space in the posterior parietal cortex and its use in planning movements. Annual Review of Neuroscience, 20:303-333 1997.

---

[30] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation Learning with Contrastive Predictive Coding. 187.:7887.03344 [cs, stat], January 2019. arXiv: 1807.03748.
[31] Semir M. Zeki. Functional organization of a visual area in the posterior bank of the superior temporal sulcus of the thesus monkey. The Journal of physiology, 236(3):549-573, 1974. Publisher: Wiley Online Library.
[32] Farhan A. Khawaja, James MG Tsui, and Christopher C. Pack. Pattern motion selectivity of spiking outputs and local field potentials in macaque visual cortex. Journal of Neuroscience, 29(43):13702-133) 2009. Publisher: Soc Neuroscience.
[33] Alexander C. Huk and Michael N. Shadlen. Neural activity in macaque parictal cortex reflects tempo- ral integration of visual motion signals during pereeptual decision making. Journal of Neuroscience, 25445):00420-104336 2005. Publisher: Soc Neuroscience.
[34] Bijan Pesaran, John S. Pezaris, Maneesh Sahani, Partha P. Mitra, and Richard A. Andersen. Temporal structure in neuronal activity during working memory in macaque parictal cortex. Nature neuroscience, 5(8):805 811, 2002. Publisher: Nature Publishing Group.
[35] P. H. Schiller. The effects of V4 and middle temporal (MT) area lesions on visual performance in the rhesus monkey. Visual Newroscience, 10(4):717-746, August 1993.
[36] Steven J. Nowlan and Terrence J. Sejnowski. A selection model for motion processing in area MT of primates. Journal of Neuroscience, (ariice,,5(())))) 15(2):1195-1211 1995. Publisher: Soc Neuroscience.
[37] Richard Zemel and Terrence J. Sejnowski. A model for encoding multiple object motions and self-motion in area MST of primate visual cortex. Journal of Neuroscience, 18(1)5531-547 1998. Publisher: Soc Neuroscience.
[38] Claudio Galletti and Patrizia Fattori. Neuronal mechanisms for detection of motion in the field of view. Neuropsychologia, 41(13):1177-1277 January 2003.
[39] A. Perrone. Model for the computation self self-motion in biological systems. Journal of the Optical Society of America A, Optics and Image Science, 972):177-144, 1992.
[40] M. Lappe. Computational mechanisms for optic flow analysis in primate cortex. International Review of Neurobiology. pages 235-268. 2000.
[41] Yong Gu, Christopher R. Fetsch, Babatunde Adeyemo, Gregory C. DeAngelis, and Dora E. Angelaki. Decoding of MSTd population activity accounts for variations in the precision of heading perception. Neuron, 66(4):596-609, 2010.
[42] Oliver W. Layton and N. Andrew Browning. A unified model of heading and path perception in primate MSTd, PLoS Compur Biol, 10(2)e1003476, 2014. Publisher: Public Library of Science.
[43] Nicole C. Rust, Odelia Schwartz. J. Anthony Movshon, and Eero P. Simoncelli. Spatiotemporal elements of macaque v1 receptive fields. Neuron, 46(6))945-956, 2005.
[44] Yuwei Cui, Liu D. Liu, Farhan A. Khawaja, Christopher C. Pack, and Daniel A. Butts. Diverse suppressive influences in area MT and selectivity to complex motion features. Journal of Neuroscience, 33(42):16715- 16728, 2013. Publisher: Soc Neuroscience.
[45] Nicole C. Rust, Valerio Mante, Eero P. Simoncelli, and J. Anthony Movshon. How MT cells analyze the motion of visual patterns. Nature neuroscience, 1421-114,,, 2006. Publisher: Nature Publishing Group.
[46] Shinji Nishimoto and Jack L. Gallant, A three-dimensional spatiotemporal receptive field model explains responses of area MT neurons to naturalistic movies. Journal of Neuroscience, 31(41):44551-1456, 2011. Publisher: Soc Neuroscience.
[47] Patrick J. Mincault, Farhan A. Khawaja, Daniel A. Butts, and Christopher C. Pack. Hicrarchical processing of complex motion along the primate dorsal visual pathway. Proceedings of the National Academy of Sciences, 1092169117,2--980 2012. Publisher: National Acad Sciences.
[48] Shital Shah, Dehadcepta Dey, Chris Lovett, and Ashish Kapoor. AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles. In Field and Service Robotics, 2017. _eprint: 1rXiv:1705.0555 700.00506.
[49] Christoph Feichtenhofer. Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6202 6211, 2019.
12

---

[50] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at spatiotemporal convolutions for action recognition. In Proceedings of the IEEE conference Computer Vision and Pattern Recognition, pages 6450-6459, 2018.
[51] Michael C.-K. Wu, Stephen V. David, and Jack I.. Gallant. Complete functional characterization of sensory neurons by system identification. Annu. Rev. Neumsci., 29:477-505, 2006. Publisher: Annual Reviews.
[52] Wyeth Bair and Christof Koch. Temporal precision of spike trains in extrastriate cortex of the behaving macaque monkey. Neural computation, 8(6):1185 1202. 1996. Publisher: MIT Press.
[53] Maneesh Sahani and Jennifer F. Linden. How linear are auditory cortical responses? Advances in neural information processing systems, pages 125-132, 2003. Publisher: MIT; 1998.
[54] Tengda Han, Weidi Xie, and Andrew Zisserman. Video Representation Learning by Dense Predictive Coding. arXiv:909, 1909,04666 [cs], September 2019. arXiv: 1909.04656.
[55] S. Livingstone. Mechanisms direction selectivity in macaque V1. Neuron, 20(3):00--526, March 1998.
[56] Nicole C. Rust, Odelia Schwartz, J. Anthony Movshon, and Ecro Simoncelli. Spike-triggered characteri- zation of excitatory and suppressive stimulus dimensions in monkey V1. Neurocomputing, 58:793-799, -999, 2004. Publisher: Elsevier.
[57] Russell L. De Valois, E. William Yund, and Norva Hepler. The orientation and direction selectivity of cells in macaque visual cortex. Vision Research, (2(5):331554, January 1982.
[58] J. Anthony Movshon and William Newsome. Visual response properties of striate cortical neurons projecting to area MT macaque monkeys. Journal of Neuroscience, 16(23):773377441 1996. Puhlisher: Soe Neuroscience.
[59] A. Perrone and A. Thiele. Speed skills: measuring the visual speed analyzing properties of primate MT neurons. Nat Neurosci, -2,, May 2001.
[60] Mark M. Churchland, Nicholas J. Priebe, and Stephen G. Lisberger. Comparison of the Spatial Limits on Direction Selectivity in Visual Areas MT and V1. Journal of Neurophysiology, 93(3): 1235-122354 March 2005. Publisher: American Physiological Society.
[61] Michael SA Graziano and Charles G. Gross. The representation of extrapersonal space: A possible role for bimodal, visual-tactile neurons. The cognitive neurosciences, pages 1021-1034, 1995.
[62] lan Nauhaus and Dario L. Ringach. Precise alignment of micromachined electrode arrays with V1 functional maps. Journal of neurophysiology. 97(5):3781-3788 2007. Publisher: American Physiological Society.
[63] Dario Ringach and Ian Nauhaus. Single- and multi-unit recordings from monkey primary visual cortex.
[64] William E. Vinje and Jack L. Gallant. Sparse coding and decorrelation in primary visual cortex during nat- 2000. Publisher: American Association for the Advancement
ural vision. Science, 287(5456):1273 of Science.
[65] Stephen V. David, William E. Vinje. and Jack L. Gallant. Natural stimulus statistics alter the receptive field structure of vI neurons. Journal of Neuroscience, (49917669977 2004. Publisher: Soc Neuroscience.
[66] Stephen V David, William E Vinje, and Jack I. Gallant. Single electrode recordings from primary visual cortex.
[67] Y Cui, DL Liu, FA Khawaja, CC Pack, and DA Butts. Spiking activity in area MT of awake adult macaques in response to complex motion features.
[68] Nishimoto and JL Gallant. Extracellular recordings from area MT awake macaques in response to naturalistic movies.
[69] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited, International Conference on Machine Learning, pages 3519-3529. PMIR, 2019.
[70] Santiago A. Cadena, George H. Denfield, Edgar Walker, Leon A. Gatys, Andreas Tolias, Matthias Bethge, and Alexander S. Ecker. Deep convolutional models improve predictions of macaque V1 responses to natural images. PLoS computational biology. 15(4)e 2019. Publisher: Public Library of Science San Francisco, CA USA.
13

---

[71] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier. Sudheendra Vjjaannraaiimhan Fabio Viola, Tim Green, Trevor Back. Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. The Kinetics Human Action Video Dataset. arXiv:1705.06955 [cs], May 2017. arXiv: 769500650.
[72] Nicolas Pinto, David D. Cox, and James DiCarlo. Why is Real-World Visual Object Recognition Hard?
PLOS Computational Biology, 4(1)xe27, January 2008. Publisher: Public Library of Science.
[73] B. A. Olshausen and D. J. Field. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381:607-60000 1996.
[74] Carlos R. Ponce, Will Xiao, Peter F. Schade, Till S. Hartmann, Gabriel Kreiman, and Margaret S. Living- stone. Evolving Images for Visual Neurons Using Deep Generative Network Reveals Coding Principles and Neuronal Preferences. Cell, 177(4):99-1009.c10 May 2019.
[75] Pouya Bashivan, Kohitij Kar, and James DiCarlo. Neural population control via deep image synthesis. Science, 364(6439), May 2019, Publisher: American Association for the Advancement of Science Section: Research Article.
[76] Ben J. Hardcastle and Holger G. Krapp. Evolution of Biological Image Stahilization. Current biology: CB, 26(20):R 1010-R1021, October 2016.
[77] Dario L. Ringach, Guillermo Sapiro, and Robert Shapley, A subspace reversc-corrclation technique for the study of visual neurons. Vision research, 37(17):2455-2464 1997.
[78] De-An Huang, Vignesh Ramanathan, Dhruv Mahajan, Lorenzo Torresani, Manohar Paluri, Li Fei-Fei, and Juan Carlos Niebles. What Makes a Video a Video: Analyzing Temporal Information in Video Understanding Models and Datasets. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR). pages 7366-7377 June 2018.
[79] Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. In Proceedings of the Neural Information Processing Systems (NIPS), 2014.
[80] Nieves Crasto, Philippe Weinzaepfel, Karteek Alahari, and Cordelia Schmid. Mars: Motion-augmente rgb stream for action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7882-7891, 2019.
[81] Jonathan Stroud, David Ross, Chen Sun, Jia Deng, and Rahul Sukthankar, D3d: Distilled 3d networks for video action recognition. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 625 634, 2020.
[82] Richard Zhang, Phillip Isola, and Alexei A. Efros. Colorful Image Colorization. In arXiv: arXiv:1603.00 1603.0851 [cs), Octoher 2016. arXiv: 1603.08511.
[83] Andrew Owens and Alexii A. Efros. Audio-Visual Visual Scene Analysis with Self-Supervised Multisensory Features. 3.XXvv/804.0364 [cs, eess], October 2018, arXiv: 1804.03641
[84] Humam Alwassel, Dhruv Mahajan, Bruno Korbar, Lorenzo Torresani, Bernard Ghanem, and Du Tran. Self-Supervised IaaenddLLamnn Learning by Cross-Modal Audio- Video Clustering. arXiv:1911.1260 [cs], October 2020.
arXiv: 1911 1.26677.
[85] Trinity B. Crapse and Marc A. Sommer. Corollary discharge across the animal kingdom. Nature Reviews Neuroscience, 98):587-600, August 2008. Number: 8 Publisher: Nature Publishing Group
[86] Jessica AF Thompson. Characterizing and comparing acoustic representations in convolutional neural networks and the human auditory system. PhD thesis, Universite de Montreal, 2020.
[87] Georg Striedter and R. Glenn Northeutt, Brains Through Time: Natural History of Vertebrates. Oxford University Press, 2019.
[88] Frederick Albert Miles and Joshua Wallman. Visual Motion and its Role in the Stabilization of Gaze, volume 5. Elsevier Science Limited, 1993.
[89] Richard Le Grand, Catherine J. Mondloch, Daphne Maurer, and Henry P. Brent. Early visual experience and face processing. Nature, 410(6831):890-888 2001. Publisher: Nature Publishing Group
[90] Uri Hasson, Samuel A. Nastase, and Ariel Goldstein. Direct Fit to Nature: An Evolutionary Perspective on Biological and Artificial Neural Networks. Neuron, 105(3):416 434, February 2020.
14

---

Checklist
The checklist follows the references. Please read the ciccklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], [N/A] You are strongly encouraged to include justification your answer, either by referencing the appropriate section of your paper or providing brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section I
Did you include the license to the code and datasets? [No] The code and the data are proprietary.
. Did you include the license the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contribu- tions and scope? [Yes] We have been careful not to overclaim (for example, not claiming that this is the only possible explanation of dorsal stream processing), and we describe limitations of our work in the Limitations section.
(b) Did you describe the limitations of your work? [Yes] Yes, we descrihe limitations of our work in the Limitations section.
(c) Did you discuss any potential negative societal impacts of your work? [N/A] We don't anticipate negative social impacts from this work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] Yes, we have read the guidelines and ensured our paper conforms to them.
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A] We do not have theoretical results.
(b) Did you include complete proofs of all theoretical results? [N/A] We do not have theoretical results.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material as a URL)? [Yes] Yes, data, code and instructions are online
(h) Did you specify all the training details (e.g., data splits, hyperparamcters, how they were chosen)?
[Yss] Yes, discussed in the Methods section
(c) Did you report error hars (e.g., with respect the random seed after running experiments multiple timesy? [Yes] We reported standard error of the mean calculated over cells in Table 3 (d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] We reported our compute section 3
4. If you are using existing assets (c.g., code, data, models) or curating/relcasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] Yes, cited in section 3 and 2
(b) Did you mention the license of the assets? [Yes] Yes, listed in section 3 and table 2
(c) Did you include any new assets either in the supplemental material or as a URL? [Yss] Yes, the data generated in airsim is available online
(d) Did you discuss whether and how consent was obtained from people whose data you're us- ing/curating? [Yes] We mention that we received permission from the authors where applicable in scction 3 In other cases, reuse was allowed by the license under which data released.
(c) Did you discuss whether the data you are using/curating contains personally identifiable informa- tion or offensive content? [N/A] We did not use human data.
If you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A] We did not collect human data.
(b) Did you descrihe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] We did not collect human data.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A] We did not collect human data.
15


Reviews:
[{"review_id": "bFAUROE0Gfq", "reviewer": "Reviewer_APpw", "summary": "The authors propose to use estimation of self-motion from short video snippets as an objective function to train visual representations resembling those of the dorsal stream of the primate visual system. They show that the first layers of small ResNets trained with this objective develop representations similar to areas V1, MT and MST. According to the authors, alternative objectives such as action classification or contrastive predictive coding or classical baselines like a Gabor filter bank did achieve the same performance on the task of predicting neural responses in the dorsal stream areas – a claim I do not fully agree with; see below. The paper proposes an original idea. It is fairly well executed and well written. I do have some reservations and questions, but I'm overall somewhat supportive of the paper – in particular if the authors can address my concerns.\n\n\n### Update after discussion period\nThe authors' response addressed some of my concerns. Unfortunately they promised to report on the control regarding scale but did not post the results, so it's hard to assess this point. I therefore keep my score as borderline accept and hope that the authors will report the results in the final version of the paper.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n+ Original idea\n+ Proposes an image-computable model of a non-trivial task associated with the dorsal stream\n+ Model can be used to create in silico predictions that can be tested experimentally\n+ Well written and easy to read\n\n## Weaknesses\n- Performance difference to the MotionNet baseline is weak at best\n- Task seems too simplistic for a model of the dorsal stream\n- Relative scale of receptive fields and input stimuli not considered\n\n\n## Detailed comments on weaknesses\n\n### Performance difference to MotionNet\n\nThe claim that inference of self-motion is a crucial task for modeling the dorsal stream doesn't seem to be fully supported by the data. Neither in MST (the most high-level area considered) nor in one of the two MT datasets does the authors' DorsalNet outperform the much simpler MotionNet that is trained on inferring object motion. Don't these results argue against self-motion being important and rather suggest that the dorsal stream tries to resolve ambiguous local motion by inferring which regions belong to the same objects? I'm not super familiar with MotionNet, so please correct me if I'm interpreting those results wrong.\n\n\n### Task too simplistic?\n\nWhile estimating an agent's self-motion is certainly one important objective (and likely one of the dorsal stream), calling DorsalNet a model that explains \"the properties of dorsal stream neurons\" seems like a bit of an overstatement given that two of the three MT/MST datasets were with random dot kinematograms, where motion is the only cue. In the authors' defense, the MT dataset with natural movies is the only one where their model actually shows an advantage, which could be taken to suggest that they have a point. However, on this dataset CPC on the same stimulus performs almost equally well, suggesting that video sequences that feature optic flow related to self-motion might be just as important as the training objective. \n\nSo while I think the authors are on the right track with their approach, the publicly available data does not seem to allow strong conclusions at this point. I would suggest toning down the claims and providing a more nuanced discussion of the possible interpretations rather than claiming strong support of their hypothesis, which I don't see.\n\n\n### Relative scale of receptive fields and input stimuli\n\nSince the models are image-computable and different datasets from different brain areas and labs are combined, I was a bit surprised to not see any discussion of scale. How did the authors decide how large one pixel in the models should be in terms of degrees of visual angle? Neurons have different receptive field sizes depending on eccentricity and brain area. Depending on which input resolution one picks, the models probably perform better or worse, and the conclusion about which layer (or model) predicts neural activity best can change (see, e.g., the work by Cadena et al., PLoS Comp Bio 2019). Unless the authors considered this issue and I missed it, I think it is important to clearly describe the reasoning here and potentially perform some controls to ensure that the inputs were provided at the optimal resolution or the conclusions don't change as a function of input resolution.\n\n\n\n## Minor comments\n\n- I wouldn't call the training paradigm \"self-supervised\". It's using ground truth information from the rendering engine as labels. Self-supervised techniques usually derive their labels from the images themselves by augmentation, cropping other pretext tasks.\n\n- The discussion talks about the \"curriculum,\" but means the dataset. Curriculum is usually rather used to refer to a sequence of training examples that vary over time w.r.t some feature (e.g. complexity), so I was a bit surprised at first until I realized how it's meant.\n\n- It's not clear what the bold numbers in Table 3 mean. Given the SEM values in parentheses it doesn't seem to signify statistical significance. It would be great to clarify.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "F3Nm9NR8JY-", "reviewer": "Reviewer_VDAo", "summary": "This paper evaluated the ability of artificial neural networks with different learning objectives to explain single-neuron responses in the primate dorsal visual pathway. A 3D ResNet trained to predict action motion parameters in simulation videos matched dorsal stream responses better than networks trained for action recognition, suggesting that estimating self-motion may be a main role of neurons in the dorsal stream.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors evaluated a range of neural networks, including one trained to estimate self-motion parameters (DorsalNet), in terms of their match to primate neural data. DorsalNet matched dorsal stream data better than networks trained for action recognition or latent state prediction. The approach of comparing networks with different learning objectives is powerful, and different training sets are also represented across the neural networks tested.\n\nThe convergence of results across experiments and stimuli is another strength of the paper. However, the fact that recordings from different brain areas were collected in different datasets is also a challenge when directly comparing results across areas, which could be mentioned more clearly in the paper. This seems particularly important given the small difference between the DorsalNet and MotionNet performance in MST (a single dataset available).\n\nThe intermediate representations in layers 1-3 of DorsalNet are interesting in terms of their match to the hypothesized features that drive dorsal stream responses. Figure 2 suggests that the DorsalNet match to dorsal neural data peaks at ~layer 3, and the paper focuses on layers 1-3. I wonder how representations in later layers compare to the intermediate ones, since the decrease in performance in later layers is so small, especially in MST.\n\nDiscussion: Although the results do suggest a separation between action recognition and motion estimation, the paragraph in lines 295-305 seems to discuss conclusions that don’t necessarily follow from the data at hand. Since the paper did not look at action recognition with/without motion in the ventral stream, the conclusions here could be rephrased to be more focused.\n\nFinally, it is somewhat unclear in what sense dorsalnet is self-supervised. Although the discussion explains that the self-supervision refers to the goal from the agent’s perspective, the abstract describes the network as having been “trained in a self-supervised manner” – perhaps this could be rephrased to be more specific/accurate. \n\nHowever, the paper addresses the question of goals in the dorsal stream in an interesting way and paves the way for more direct confirmatory experiments using DNN-generated stimuli/simulation videos. \n\n------------------------------------------------\nAfter author response:\n\nI thank the authors for their thorough response. Interesting to see the results using sparse regression, as well as the layer representation results. I look forward to reading the updated paper!\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have discussed the main limitations of the work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "CmSP5NPskZ", "reviewer": "Reviewer_mYCG", "summary": "This paper positions itself in an important gap in the computational neuroscience literature which uses artificial neural networks as models of biological vision. The paper uses first-person perspective video stimuli to train a 3d ResNet model to predict self-motion parameters (namely head rotation and velocity) with contrastive predictive coding. The paper examines layer-by-layer the properties of the learned representations and compares them qualitatively and quantitatively to a database of neural recordings from the dorsal stream of non-human primates, recorded while those primates were viewing videos, or classical psychophysical visual stimuli. The authors find that their model learns representations that are more similar to the primates than baseline models or the same model trained on a different task. Finally the authors find that the networks which are functionally achieve  the best self-motion estimation performance are also those that match biological dorsal stream neural data the best. ", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is an original and complete contribution to the field which helps to address an important gap in models of biological perception. The approach to train on simulated egocentric video footage and estimate motion is novel for computational neuroscience work, and the layer-by-layer connections between the resulting representations in the ANN and primate NNs were well explored.\nFinally the discovery that the ANN models that perform best at predicting self-motion also form the representations most closely aligned with those in the primate dorsal stream is an exciting addition to the literature. \n\nThe work was also clearly presented, although as a very minor point more detailed figure legends would be helpful to communicate the main point of each figure more readily to the reader. Note also as an aside a missing 'TODO' link on line 203 which can be easily fixed.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "2QVKWG8BYr", "reviewer": "Reviewer_X8u7", "summary": "In this paper, the authors develop a new goal-driven model of the primate dorsal pathway based on a ResNet architecture trained to infer self motion parameters. They show that its units show properties reminiscent of properties of neurons found along the dorsal hierarchy. They benchmark this model extensively against other models including object recognition trained ones.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In general, this is a well-written paper, with a clear structure and well-done experiments. A few details could be improved (see below), but the main point of criticism one could voice against this paper is that of novelty and advance over previous work. Given the hype about the ventral stream goal driven models, the present paper seems like a relatively minor extensions. However, I think it is still worthwhile doing, and doing it well (as in the present work). The authors argue for the importance of self-motion cues in the dataset and the task being self motion inference, yet the quantitative advantage over MotionNet (their ref 26) seems small (Table 3, last two lines). Maybe the authors could elaborate on why they still think their task and network are superior in modeling dorsal stream neurons than MotionNets (which seems to use simple displaced images as stimuli and seems to be a bit shallower). \n\nAdditional comments:\n- Why was a ResNet used as a base model?\n- For the alignment between networks and neurons, why was L2 penalized regression used and not L1 regularized? \n- How many video segments were generated?\n- Line 96: \"the\" missing in front of Unreal\n- The authors call their approach self-supervised in the abstract. Why? To me this seems just a DNN with known ground truth. \n- The tuning index in line 170 is likely higher than usual due to the lack of noise\n- Why do the correlation values in Fig. 2 go beyond 1? Rather use non-parametric CIs. \n- The results in Line 203ff are not obvious from Fig. 2, maybe mark the maximum\n- For Table 3, what is the number of experiments and what is the SE over?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper received 4 accepts (including a marginal accept). The paper makes an original and complete contribution to the field and addresses an important gap in biological vision. I would add that unlike much of the current work in computational models of vision where the goal appears to simply quantitatively fit model responses to neural data, this paper performs extensive in-sillico electrophysiology on the model responses to highlight qualitative properties using classic stimuli that have been used to characterize the dorsal stream. The AC recommends accepting as a spotlight. 
Decision: Accept (Spotlight)