The functional specialization of visual cortex emerges from training parallel pathways with self-supervised predictive learning

1 Introduction . In the mammalian visual cortex information is processed in a hierarchical manner using two specialized pathways [ 15 , 52 ] : the ventral , or “ where ” pathway , and the dorsal , or “ what ” pathway . These two pathways are specialized for visual recognition and movement , respectively [ 42 , 24 , 19 , 58 , 59 , 17 ] . For example , damage to the ventral pathway may impair object recognition , whereas damage to the dorsal pathway may impair motion perception [ 64 ] . Deep artificial neural networks ( ANNs ) trained in a supervised manner on object categorization have been successful at matching the representations of the ventral visual stream [ 62 , 57 , 32 ] . They have 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . been shown to develop representations that map onto the ventral hierarchy , and which can be used to predict [ 62 ] and control [ 1 , 49 ] neural activity in the ventral pathway . However , when we look at the other principal visual pathway in the mammalian brain , i.e . the dorsal pathway , the situation is different . Very few studies have examined the ability of deep ANNs to develop representations that match the dorsal hierarchy ( though see the following fMRI study : [ 21 ] ) . Moreover , to the best of our knowledge , no studies have demonstrated both ventral-like and dorsal-like representations in a single network . This lack of deep ANN models that capture both ventral and dorsal pathways leads naturally to an important question : under what training conditions would specialized ventral-like and dorsal-like pathways emerge in a deep ANN ? Would a second loss function be required to obtain matches to dorsal pathways , or is there a single loss function that could induce both types of representations ? One promising set of candidates are predictive self-supervised loss functions [ 45 , 22 , 38 ] . Recent work has shown that self-supervised learning can produce similar results to supervised learning for the ventral pathway [ 63 , 34 , 26 ] . Moreover , there is a large body of work showing that mammals possess predictive processing mechanisms in their cortex [ 31 , 8 , 30 , 18 , 16 ] , including in the dorsal pathway [ 2 , 35 ] , which suggests that a predictive form of self-supervised learning could potentially lead to the emergence of both ventral and dorsal-like representations . Addressing this question requires recordings from different ventral and dorsal visual areas in the brain . Here , we explore these issues using publicly available data from the Allen Brain Observatory [ 9 ] , which provides recordings from a large number of areas in mouse visual cortex . We examine the ability of a self-supervised predictive loss function ( contrastive predictive coding [ 45 , 22 ] ) to induce representations that match mouse visual cortex . When we train a network with a single pathway , we find that it possesses more ventral-like representations . However , when we train a network with two parallel pathways we find that the predictive loss function induces distinct representations that map onto the ventral/dorsal division . This allows the network to better support both object categorization and motion recognition downstream tasks via the respective specialized pathways . In contrast , supervised training with an action categorization task only leads to matches to the ventral pathway , and not the dorsal pathway . Altogether , this work demonstrates that the two specialized pathways of visual cortex can be modelled with the same ANN if a self-supervised predictive loss function is applied to an architecture with parallel pathways . This suggests that self-supervised predictive loss functions may hold great promise for explaining the functional properties of mammalian visual cortex . 2 Background and related work . Self-supervised ANN models of the ventral visual stream Recently , [ 63 ] and [ 34 ] showed that the representations learned by self-supervised models trained on static images produce good matches to the ventral pathway . Our work builds on this by exploring the potential for self-supervised learning to also explain the dorsal pathway . ANN models of the mouse visual system Mice have become a common animal model in visual neuroscience due to the sophisticated array of experimental tools [ 44 , 27 ] . As such , [ 53 ] compared the responses of different areas in mouse visual cortex with the representations of a VGG16 trained on ImageNet . In this paper , we show that self-supervised learning can produce better fits to both ventral and dorsal areas than supervised learning . ANN models of the dorsal pathway The MotionNet ANN model [ 51 ] , is a feedforward ANN trained to predict the motion direction of segments of natural images , which was inspired by the role of the dorsal pathway in motion perception . In [ 6 ] , they show that learning both ventral and dorsal-like representations in a single ANN with two pathways is possible if one forces the two pathways to process the phase and amplitude of a complex decomposition of the stimuli separately . In [ 21 ] , they show that training a deep ANN on supervised action recognition can induce some match to dorsal pathway fMRI recordings . In this study , we show that with prediction as the learning objective and an architecture that has two parallel pathways , both ventral-like and dorsal-like representations can be learned . 3 Methods . 3.1 Datasets . We use the Allen Brain Observatory open 2-photon calcium imaging dataset for the experiments in this study . We select subsets of the dataset based on brain area , recording depth , and visual stimuli used . Recordings from five areas of mouse visual cortex are used ( VISp , VISlm , VISal , VISpm , VISam ; Figure 1a ) . We only exclude one area from our analyses ( VISrl ) because it is a multi-sensory area , and visual stimuli alone do not drive it well [ 9 ] . We use recordings from cortical depths of 175-250µm ( which corresponds to cortical layers 2-3 ) as these are the recordings with the largest number of neurons . When selecting visual stimuli , we use recordings elicited by the presentation of natural movies , because unlike static images , movies can elicit clear responses in both ventral and dorsal areas [ 9 ] . Thus , we only use those parts of the dataset in which natural movies ( 30 seconds long ) were presented as visual stimuli . More details about the dataset can be found in [ 9 ] . For training the deep ANNs , we use the UCF101 dataset [ 56 ] ( see section A.4 of supplementary materials ) . 3.2 Analysis techniques . Representational Similarity Analysis ( RSA ) To measure the representational similarities between real brains and ANNs we use the RSA method . RSA has been commonly used in both the neuroscience [ 11 , 33 ] and deep learning literature [ 41 , 43 ] . The details of RSA can be found in those citations , but we will summarize it briefly here . First , we create response matrices R ∈ IRN×M for every brain area and every layer of our ANNs , where N is the number of neurons and M is the number of video blocks , with each block comprising 15 frames of the video . Element ij of the response matrix represents the response of the ith neuron to the jth video sequence . We then use Pearson correlation to calculate the similarity of every pair of columns ( e.g . kth and lth columns ) in matrix R , and form the M by M Representation Similarity Matrix ( RSM ∈ IRM×M ) , in which every element ( kl ) quantifies the similarity of the responses to the kth and lth videos blocks . Thus , the RSMs describe the representation space in a network , be it a brain area or an ANN layer . Given two RSMs , we then use Kendall ’ s τ between the vectorized RSMs to quantify the similarity between the two representations . It is important to note that measurement noise can potentially induce bias in the RSM estimations , but since the variance of measurement noise is not expected to be different across different video blocks , using Kendall ’ s τ rank correlation should cancel the bias in our RSM estimations ( see [ 12 ] for more details ) . As an additional sanity check , we use RSA to compare the representations between mice . If RSA is identifying salient representational geometries , then the RSMs between different areas should be lower than the RSMs for the same areas . Indeed , as shown in Figure 1b , the representational similarity is highest for the same regions across animals ( i.e . the diagonal values are larger ) . These diagonal values also represent the noise ceiling for these areas ( see section A.1 ) . Throughout the paper ( with the exception of Figure 1b ) , RSM similarities are reported as the percentage of the noise ceiling . Identification of brain regions in the hierarchy The hierarchical organization of mouse visual areas can be inferred based on the anatomical and functional properties of each brain region . We adopted an approximation of hierarchical indices as reported in [ 23 ] . The relative hierarchical placing of the visual areas included in this study are shown in Figure 1c . Identification of brain regions into ventral and dorsal streams We group the brain regions into two sets , i.e . ventral and dorsal areas . Although the ventral/dorsal specialization of visual pathways in mice is not as clear and well understood as it is in primates , many anatomical and physiological studies do suggest that mice also possess such specialized pathways [ 40 , 58 , 39 , 37 ] . According to previous anatomical and physiological studies [ 17 , 58 ] , VISlm and VISam can be considered as the most ventral and dorsal areas of mouse visual cortex , respectively . We then use the similarity of representations between other areas and VISlm and VISam to estimate a ventral score ( V-score = rAlm/ ( r A am + r A lm ) ) and a dorsal score ( D-score = r A am/ ( r A am + r A lm ) ) for each area . rAam is the representational similarity between area A and VISam , and r A lm is the representational similarity between area A and VISlm . The D-score and V-score values for the five visual areas are plotted in Figure 1d . Based on the D-score and V-score values , we grouped VISlm , VISp , and VISpm as more ventral and VISal , and VISam as more dorsal in our analysis . This grouping is in keeping with a recent study that showed VISam and VISal are the first and second most responsive areas to motion stimuli : an important characteristic of dorsal areas [ 54 ] . 3.3 Model . Self-supervised predictive learning We used Contrastive Predictive Coding ( CPC ) for selfsupervised learning , which was developed for modeling sequential data , including video datasets [ 45 ] . The loss function relies on predicting the future latent representations of a video sequence , given its present and past representations . See Figure S1 for a schematic of the model , and A.2 for more details regarding the CPC loss function . ANN Architecture All the ANN backbones used in our study are variants of the ResNet architecture , similar to the ones used in [ 13 ] . Our ResNet architectures have either one or two pathways . The one-pathway ResNet ( ResNet-1p ) is a regular 3D ResNet ( Figure 2a ) . The two-pathway ResNet ( ResNet-2p ) is composed of two parallel ResNet branches , which split after a single convolutional layer , and merge after their final layers through concatenating their outputs along the channel dimension ( Figure 3a ) . Both pathways of ResNet-2p receive a copy of the first convolutional layer output , and each has 10 Res-blocks . To keep the total number of output channels the same in both architectures , each pathway in ResNet-2p has half the number of channels of the single pathway in ResNet-1p . For the ANNs trained on object categorization , we use ResNet-18 pretrained on the ImageNet database [ 25 ] . Our 3D ResNet architectures are summarized in Table S1 . Baseline models In terms of alignment with brain representations of video sequences , we compare the CPC trained ANNs with four other models : ( 1 ) a simple model based on Gabor filters , ( 2 ) a randomized deep ANN , ( 3 ) a ResNet-18 trained on ImageNet , and ( 4 ) 3D ResNets trained on action recognition in a supervised manner . See section A.3 of supplementary materials for more details . Training We use the backpropagation algorithm and Adam optimizer . CPC is trained with a batch size of 40 samples , a learning rate of 10−3 , and 100 epochs . Supervised action recognition is trained with a batch size of 256 samples , a learning rate of 5× 10−4 , and 300 epochs . All the models are implemented with PyTorch 1.7 and trained on RTX8000 NVIDIA GPUs . Downstream tasks In addition to comparisons with dorsal and ventral representations in mouse brain , we examine the two pathways of our trained ResNet-2p on two downstream tasks : object categorization and motion discrimination , which are supported in the real brain by the ventral and dorsal pathways , respectively . See section A.5 of supplementary materials for more details .

Summary: The paper uses contrastive predictive coding to train deep networks on UCF101 videos. It then uses representational similarity to compare their representations to those of mouse calcium-imaging data from the Allen Brain Observatory. The key result is that a two-stream network is a good match to mouse visual cortex data. One of the (structurally identical) streams develops a representation that is a good match to dorsal mouse areas, and the other matches ventral mouse areas. Deeper areas correspond better to areas higher in the mouse visual hierarchy. Furthermore, the ventral stream of the model better supports transfer to CIFAR-10, while the dorsal stream better supports discrimination of random-dot motion direction.