The functional specialization of visual cortex emerges from training parallel pathways with self-supervised predictive learning

1 Introduction . In the mammalian visual cortex information is processed in a hierarchical manner using two specialized pathways [ 15 , 52 ] : the ventral , or “ where ” pathway , and the dorsal , or “ what ” pathway . These two pathways are specialized for visual recognition and movement , respectively [ 42 , 24 , 19 , 58 , 59 , 17 ] . For example , damage to the ventral pathway may impair object recognition , whereas damage to the dorsal pathway may impair motion perception [ 64 ] . Deep artificial neural networks ( ANNs ) trained in a supervised manner on object categorization have been successful at matching the representations of the ventral visual stream [ 62 , 57 , 32 ] . They have 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . been shown to develop representations that map onto the ventral hierarchy , and which can be used to predict [ 62 ] and control [ 1 , 49 ] neural activity in the ventral pathway . However , when we look at the other principal visual pathway in the mammalian brain , i.e . the dorsal pathway , the situation is different . Very few studies have examined the ability of deep ANNs to develop representations that match the dorsal hierarchy ( though see the following fMRI study : [ 21 ] ) . Moreover , to the best of our knowledge , no studies have demonstrated both ventral-like and dorsal-like representations in a single network . This lack of deep ANN models that capture both ventral and dorsal pathways leads naturally to an important question : under what training conditions would specialized ventral-like and dorsal-like pathways emerge in a deep ANN ? Would a second loss function be required to obtain matches to dorsal pathways , or is there a single loss function that could induce both types of representations ? One promising set of candidates are predictive self-supervised loss functions [ 45 , 22 , 38 ] . Recent work has shown that self-supervised learning can produce similar results to supervised learning for the ventral pathway [ 63 , 34 , 26 ] . Moreover , there is a large body of work showing that mammals possess predictive processing mechanisms in their cortex [ 31 , 8 , 30 , 18 , 16 ] , including in the dorsal pathway [ 2 , 35 ] , which suggests that a predictive form of self-supervised learning could potentially lead to the emergence of both ventral and dorsal-like representations . Addressing this question requires recordings from different ventral and dorsal visual areas in the brain . Here , we explore these issues using publicly available data from the Allen Brain Observatory [ 9 ] , which provides recordings from a large number of areas in mouse visual cortex . We examine the ability of a self-supervised predictive loss function ( contrastive predictive coding [ 45 , 22 ] ) to induce representations that match mouse visual cortex . When we train a network with a single pathway , we find that it possesses more ventral-like representations . However , when we train a network with two parallel pathways we find that the predictive loss function induces distinct representations that map onto the ventral/dorsal division . This allows the network to better support both object categorization and motion recognition downstream tasks via the respective specialized pathways . In contrast , supervised training with an action categorization task only leads to matches to the ventral pathway , and not the dorsal pathway . Altogether , this work demonstrates that the two specialized pathways of visual cortex can be modelled with the same ANN if a self-supervised predictive loss function is applied to an architecture with parallel pathways . This suggests that self-supervised predictive loss functions may hold great promise for explaining the functional properties of mammalian visual cortex . 2 Background and related work . Self-supervised ANN models of the ventral visual stream Recently , [ 63 ] and [ 34 ] showed that the representations learned by self-supervised models trained on static images produce good matches to the ventral pathway . Our work builds on this by exploring the potential for self-supervised learning to also explain the dorsal pathway . ANN models of the mouse visual system Mice have become a common animal model in visual neuroscience due to the sophisticated array of experimental tools [ 44 , 27 ] . As such , [ 53 ] compared the responses of different areas in mouse visual cortex with the representations of a VGG16 trained on ImageNet . In this paper , we show that self-supervised learning can produce better fits to both ventral and dorsal areas than supervised learning . ANN models of the dorsal pathway The MotionNet ANN model [ 51 ] , is a feedforward ANN trained to predict the motion direction of segments of natural images , which was inspired by the role of the dorsal pathway in motion perception . In [ 6 ] , they show that learning both ventral and dorsal-like representations in a single ANN with two pathways is possible if one forces the two pathways to process the phase and amplitude of a complex decomposition of the stimuli separately . In [ 21 ] , they show that training a deep ANN on supervised action recognition can induce some match to dorsal pathway fMRI recordings . In this study , we show that with prediction as the learning objective and an architecture that has two parallel pathways , both ventral-like and dorsal-like representations can be learned . 3 Methods . 3.1 Datasets . We use the Allen Brain Observatory open 2-photon calcium imaging dataset for the experiments in this study . We select subsets of the dataset based on brain area , recording depth , and visual stimuli used . Recordings from five areas of mouse visual cortex are used ( VISp , VISlm , VISal , VISpm , VISam ; Figure 1a ) . We only exclude one area from our analyses ( VISrl ) because it is a multi-sensory area , and visual stimuli alone do not drive it well [ 9 ] . We use recordings from cortical depths of 175-250µm ( which corresponds to cortical layers 2-3 ) as these are the recordings with the largest number of neurons . When selecting visual stimuli , we use recordings elicited by the presentation of natural movies , because unlike static images , movies can elicit clear responses in both ventral and dorsal areas [ 9 ] . Thus , we only use those parts of the dataset in which natural movies ( 30 seconds long ) were presented as visual stimuli . More details about the dataset can be found in [ 9 ] . For training the deep ANNs , we use the UCF101 dataset [ 56 ] ( see section A.4 of supplementary materials ) . 3.2 Analysis techniques . Representational Similarity Analysis ( RSA ) To measure the representational similarities between real brains and ANNs we use the RSA method . RSA has been commonly used in both the neuroscience [ 11 , 33 ] and deep learning literature [ 41 , 43 ] . The details of RSA can be found in those citations , but we will summarize it briefly here . First , we create response matrices R ∈ IRN×M for every brain area and every layer of our ANNs , where N is the number of neurons and M is the number of video blocks , with each block comprising 15 frames of the video . Element ij of the response matrix represents the response of the ith neuron to the jth video sequence . We then use Pearson correlation to calculate the similarity of every pair of columns ( e.g . kth and lth columns ) in matrix R , and form the M by M Representation Similarity Matrix ( RSM ∈ IRM×M ) , in which every element ( kl ) quantifies the similarity of the responses to the kth and lth videos blocks . Thus , the RSMs describe the representation space in a network , be it a brain area or an ANN layer . Given two RSMs , we then use Kendall ’ s τ between the vectorized RSMs to quantify the similarity between the two representations . It is important to note that measurement noise can potentially induce bias in the RSM estimations , but since the variance of measurement noise is not expected to be different across different video blocks , using Kendall ’ s τ rank correlation should cancel the bias in our RSM estimations ( see [ 12 ] for more details ) . As an additional sanity check , we use RSA to compare the representations between mice . If RSA is identifying salient representational geometries , then the RSMs between different areas should be lower than the RSMs for the same areas . Indeed , as shown in Figure 1b , the representational similarity is highest for the same regions across animals ( i.e . the diagonal values are larger ) . These diagonal values also represent the noise ceiling for these areas ( see section A.1 ) . Throughout the paper ( with the exception of Figure 1b ) , RSM similarities are reported as the percentage of the noise ceiling . Identification of brain regions in the hierarchy The hierarchical organization of mouse visual areas can be inferred based on the anatomical and functional properties of each brain region . We adopted an approximation of hierarchical indices as reported in [ 23 ] . The relative hierarchical placing of the visual areas included in this study are shown in Figure 1c . Identification of brain regions into ventral and dorsal streams We group the brain regions into two sets , i.e . ventral and dorsal areas . Although the ventral/dorsal specialization of visual pathways in mice is not as clear and well understood as it is in primates , many anatomical and physiological studies do suggest that mice also possess such specialized pathways [ 40 , 58 , 39 , 37 ] . According to previous anatomical and physiological studies [ 17 , 58 ] , VISlm and VISam can be considered as the most ventral and dorsal areas of mouse visual cortex , respectively . We then use the similarity of representations between other areas and VISlm and VISam to estimate a ventral score ( V-score = rAlm/ ( r A am + r A lm ) ) and a dorsal score ( D-score = r A am/ ( r A am + r A lm ) ) for each area . rAam is the representational similarity between area A and VISam , and r A lm is the representational similarity between area A and VISlm . The D-score and V-score values for the five visual areas are plotted in Figure 1d . Based on the D-score and V-score values , we grouped VISlm , VISp , and VISpm as more ventral and VISal , and VISam as more dorsal in our analysis . This grouping is in keeping with a recent study that showed VISam and VISal are the first and second most responsive areas to motion stimuli : an important characteristic of dorsal areas [ 54 ] . 3.3 Model . Self-supervised predictive learning We used Contrastive Predictive Coding ( CPC ) for selfsupervised learning , which was developed for modeling sequential data , including video datasets [ 45 ] . The loss function relies on predicting the future latent representations of a video sequence , given its present and past representations . See Figure S1 for a schematic of the model , and A.2 for more details regarding the CPC loss function . ANN Architecture All the ANN backbones used in our study are variants of the ResNet architecture , similar to the ones used in [ 13 ] . Our ResNet architectures have either one or two pathways . The one-pathway ResNet ( ResNet-1p ) is a regular 3D ResNet ( Figure 2a ) . The two-pathway ResNet ( ResNet-2p ) is composed of two parallel ResNet branches , which split after a single convolutional layer , and merge after their final layers through concatenating their outputs along the channel dimension ( Figure 3a ) . Both pathways of ResNet-2p receive a copy of the first convolutional layer output , and each has 10 Res-blocks . To keep the total number of output channels the same in both architectures , each pathway in ResNet-2p has half the number of channels of the single pathway in ResNet-1p . For the ANNs trained on object categorization , we use ResNet-18 pretrained on the ImageNet database [ 25 ] . Our 3D ResNet architectures are summarized in Table S1 . Baseline models In terms of alignment with brain representations of video sequences , we compare the CPC trained ANNs with four other models : ( 1 ) a simple model based on Gabor filters , ( 2 ) a randomized deep ANN , ( 3 ) a ResNet-18 trained on ImageNet , and ( 4 ) 3D ResNets trained on action recognition in a supervised manner . See section A.3 of supplementary materials for more details . Training We use the backpropagation algorithm and Adam optimizer . CPC is trained with a batch size of 40 samples , a learning rate of 10−3 , and 100 epochs . Supervised action recognition is trained with a batch size of 256 samples , a learning rate of 5× 10−4 , and 300 epochs . All the models are implemented with PyTorch 1.7 and trained on RTX8000 NVIDIA GPUs . Downstream tasks In addition to comparisons with dorsal and ventral representations in mouse brain , we examine the two pathways of our trained ResNet-2p on two downstream tasks : object categorization and motion discrimination , which are supported in the real brain by the ventral and dorsal pathways , respectively . See section A.5 of supplementary materials for more details .

Summary: The authors show that a two-stream 3d ResNet architecture trained using a contrastive predictive coding (CPC) objective develops a representation that resembles the ventral/dorsal stream split in visual cortex. Although I do not buy into all claims of the paper, I am generally very supportive of it. The authors make their main point and the paper is well written. With some adjustments of the claims it will be a great paper.   ### Update after discussion period  Given the results from the control experiments reported by the authors, I support publication of the paper and increased my score to 7. I do ask the authors to include those results in the final version of the paper.