Generalization in data-driven models of primary visual cortex

1 INTRODUCTION . A long lasting challenge in sensory neuroscience is to understand the computations of neurons in the visual system stimulated by natural images ( Carandini et al. , 2005 ) . Important milestones towards this goal are general system identification models that can predict the response of large populations of neurons to arbitrary visual inputs . In recent years , deep neural networks have set new standards in predicting responses in the visual system ( Yamins et al. , 2014 ; Vintch et al. , 2015 ; Antolík et al. , 2016 ; Cadena et al. , 2019a ; Batty et al. , 2016 ; Kindel et al. , 2017 ; Klindt et al. , 2017 ; Zhang et al. , 2018 ; Ecker et al. , 2018 ; Sinz et al. , 2018 ) and the ability to yield novel response characterizations ( Walker et al. , 2019 ; Bashivan et al. , 2019 ; Ponce et al. , 2019 ; Kindel et al. , 2019 ; Ukita et al. , 2019 ) . Such a general system identification model is one way for neuroscientists to investigate the computations of the respective brain areas in silico . Such in silico experiments exhibit the possibility to study the system at a scale and level of detail that is impossible in real experiments which have to cope with limited experimental time and adaptation effects in neurons . Moreover , all parameters , connections and weights in an in silico model can be accessed directly , opening up the opportunity to manipulate the model or determine its detailed tuning properties using numerical optimization methods . In order for the results of such analyses performed on an in silico model to be reliable , however , one needs to make sure that the model does indeed replicate the responses of its biological counterpart faithfully . This work provides an important step towards obtaining such a generalizing model of mouse V1 . High performing predictive models need to account for the increasingly nonlinear response properties of neurons along the visual hierarchy . As many of the nonlinearities are currently unknown , one of the key challenges in neural system identification is to find a good set of characteristic nonlinear basis functions—so called representations . However , learning these complex nonlinearities from single neuron responses is difficult given limited experimental data . Two approaches have proven to be promising in the past : Task-driven system identification networks rely on transfer learning and use nonlinear representations pre-trained on large datasets for standard vision tasks , such as object recognition ( Yamins & DiCarlo , 2016 ) . Single neuron responses are predicted from a particular layer of a pre-trained network using a simple readout mechanism , usually an affine function followed by a static nonlinearity . Data-driven models share a common nonlinear representation among hundreds or thousands of neurons , and train the entire network end-to-end on stimulus response pairs from the experiment . Because the nonlinear representation is shared , it is trained via massive multi-task learning ( one neuron–one task ) and can be learned even from limited experimental data . Task-driven networks are appealing because they only need to fit the readout mechanisms on top of a given representation and thus are data-efficient in terms of the number of stimulus-response pairs needed to achieve good predictive performance ( Cadena et al. , 2019a ) . Moreover , as their representations are obtained independently of the neural data , a good predictive performance suggests that the nonlinear features are characteristic for a particular brain area . This additionally offers the interesting normative perspective that the functional representations in deep networks and biological vision could be aligned by common computational goals ( Yamins & DiCarlo , 2016 ; Kell et al. , 2018 ; Kubilius et al. , 2018 ; Nayebi et al. , 2018 ; Sinz et al. , 2019 ; Güçlü & van Gerven , 2014 ; Kriegeskorte , 2015 ; Khaligh-Razavi & Kriegeskorte , 2014 ; Kietzmann et al. , 2019 ) . In order to quantify the fit of the normative hypothesis , it is important to compare a given representation to other alternatives ( Schrimpf et al. , 2018 ; Cadena et al. , 2019a ) . However , while representations pre-trained on ImageNet are the state-of-the-art for predicting visual cortex in primates ( Cadena et al. , 2019a ; Yamins & DiCarlo , 2016 ) , recent work has demonstrated that pre-training on object categorization ( VGG16 ) yields no benefits over random initialization for mouse visual cortex ( Cadena et al. , 2019b ) . Since random representation should not be characteristic for a particular brain area and other tasks that might yield more meaningful representations have not been found yet , this raises the questions whether there are better ways to obtain a generalizing nonlinear representation for mouse visual cortex . Here , we investigate whether such a generalizing representation can instead be obtained from datadriven networks . For this purpose , we develop a new data efficient readout which is designed to push non-linear computations into the core and test whether this core has learned general characteristic features of mouse visual cortex by applying the same criteria as for the task-driven approach : The ability to predict a population of unseen neurons in a new animal ( transfer learning ) . Specifically , we make the following contributions : 1 We introduce a novel readout mechanism that keeps the number of per-neuron parameters at a minimum and learns a bivariate Gaussian distribution for the readout position from anatomical data using retinotopy . With this readout alone , we surpass the previous state-of-the-art performance in direct training by 7 % . 2 We demonstrate that a representation pre-trained on thousands of neurons from various animals generalizes to neurons from an unseen animal ( transfer learning ) . It exceeds the direct training condition by another 11 % , setting the new state-of-the-art and outperforms a task-driven representation—trained on object recognition—by about 33 % . 3 We then show that this generalization can be attributed to the representation and not the readout mechanism , indicating that the data-driven core indeed captures generalizing features of cortex : A representation trained on a single experiment ( 4.5k examples ) in combination with a readout trained on anatomically matched neurons from four experiments ( 17.5k examples ) did not achieve this performance . 4 Lastly , we find that transfer learning with our data-driven core is more data-efficient than direct training , achieving the same performance with only 40 % of the data . Ne ur on se ts Image sets Unique anim als Evaluation 11-S 4-S : matched 4-S : diff animals 1-S Figure 1 : Scans and training sets . Overview of the datasets and how they are combined into different training sets . Each scan was performed on a specific set of neurons ( rows ) using a specific set of unique images ( columns ) . Repeatedly presented test images were the same for all scans . Some scans were performed on the same neuron but with different image sets ( first row ) . Colors indicate grouping of scans into training sets and match line colors in Fig . 5 to indicate which dataset a representation/core ( not the readout ) was trained on . 2 METHODS . 2.1 DATA . Functional data The data used in our experiments consists of pairs of neural population responses and grayscale visual stimuli sampled and cropped from ImageNet , isotropically downsampled to 64× 36 px , with a resolution of 0.53 ppd ( pixels per degree of visual angle ) . The neural responses were recorded from layer L2/3 of the primary visual cortex ( area V1 ) of the mouse , using a wide field two photon microscope ( Sofroniew et al. , 2016 ) . Activity was measured using the genetically encoded calcium indicator GCaMP6s . V1 was targeted based on anatomical location as verified by numerous previous experiments performing retinotopic mapping using intrinsic imaging . We selected cells based on a classifier for somata on the segmented cell masks and deconvolved their fluorescence traces ( Pnevmatikakis et al. , 2016 ) . We did not filter cells according to visual responsiveness . The stimulation paradigm and data pre-processing followed the procedures described by Walker et al . ( 2019 ) . A single scan contained the responses of approximately 5000–9000 neurons to up to 6000 images , of which 1000 images consist of 100 unique images which were presented 10 times each to allow for an estimate of the reliability of the neuron ( see Appendix for a detailed description of the datasets ) . We used the repeated images for testing , and split the rest into 4500 training and 500 validation images . The neural data was preprocessed by normalizing the responses of the neurons by their standard deviation on the training set . To put the number of recorded neurons per scan into perspective , assuming that V1 has an area of about 4mm2 , that L2/3 is about 150-250µm thick and has a cell density of 80k excitatory cells per mm3 , entire V1 L2/3 should contain about 48k - 80k neurons ( Garrett et al. , 2014 ; Jurjut et al. , 2017 ; Schüz & Palm , 1989 ) , similar to the maximum number of neurons that we train a model on ( 72k neuron , 11-S , Fig . 1 , orange ) . Note , however , that this does not mean that these 72k neurons sample V1 or the visual field of a mouse evenly because of possible experimental biases in the choice of the recording location . All together , we used 13 scans from a total of 7 animals ( Fig . 1 ) . Each scan is defined by the set of neurons it was performed on ( rows/neuron sets in Fig . 1 ) and the set of images that were shown ( columns/image sets in Fig . 1 ) . Different image sets had non-overlapping training/validation images , but the same test images . Some of the scans were performed on the same neurons , but with different sets of natural images ( first row in Fig . 1 ) . These neurons were matched across scans by cross-correlating the structural scan planes against functionally recorded stacks ( Walker et al. , 2019 ) . Stitching data from several scans in this way allowed us to increase the number of image presentations per neuron beyond what would be possible in a single scan . We combined these scans into different training sets ( one color–one training set in Fig . 1 ) and named each one of them—e.g . 11-S for a set with 11 Scans . The different sets are further explained in the respective experiments they are used in . All data from the seven mice used in this work has been recorded by trained personnel under a strict protocol according to the regulations of the local authorities at Balor College of Medicine . 2.2 NETWORKS AND TRAINING . The networks are split conceptually into two parts : a core and a readout . The core captures the nonlinear image representation and is shared among all neurons . The readout maps the features of the core into neural responses and contains all neuron specific parameters . Readout position network Scan field Feature vector at sampled position x y y' x' x y Topmost block of core Figure 2 : Using retinotopy to learn the readout position from anatomical data . The Gaussian readout for each neuron uses features from a single location on the final tensor of the core CNN ( bottom ) . The position is drawn from a 2D Gaussian for every image during training . The parameters of the Gaussian for each neuron are learned during training . The means of the Gaussians are predicted from each neuron ’ s coordinates on cortex by a Readout Position Network whose weights are shared across neurons and learned during training ( top ) . During testing , the mean of the Gaussian is used as the neuron ’ s position . Representation/Core We model the core with a four-layer convolutional neural network ( CNN ) , with 64 feature channels per layer . In each layer , the 2d-convolutional layer is followed by a batch normalization layer and an ELU nonlinearity ( Ioffe & Szegedy , 2015 ; Clevert et al. , 2015 ) . All convolutional layers after the first one are depth-separable convolutions ( Chollet , 2017 ) which we found to yield better results than standard convolutional layers in a search among different architecture choices . Readouts We compared two different types of readouts to map the nonlinear features of the core to the response of each neuron . For each neuron , a tensor of x ∈ Rw×h×c ( width , height , channels ) needs to be mapped to a single scalar , corresponding to the target neuron ’ s response . All of our readouts assume that this function is affine with a linear weight tensor w ∈ Rw×h×c , followed by an ELU offset by one ( ELU+1 ) , to keep the response positive . Furthermore , both readouts assume that in feature space the receptive field of each neuron does not change its position across features , but they differ in how this receptive field location is constrained and learned . The factorized readout ( Klindt et al. , 2017 ) factorizes the 3d readout tensor into a lower-dimensional representation by using a spatial mask matrix uij and a vector of feature weights vk , i.e . wijk = uijvk . The spatial mask uij is restricted to be positive and encouraged to be sparse through an L1 regularizer . Our novel Gaussian readout reduces the number of per-neuron parameters . It computes a linear combination of the feature activations at a single spatial position— parametrized as ( x , y ) coordinates —via bilinear interpolation ( Sinz et al. , 2018 ) . To facilitate gradient flow during training , we replace the spatial downsampling used in ( Sinz et al. , 2018 ) by a sampling step , which during training draws the readout position of each nth neuron from a bivariate Gaussian distribution N ( µn , Σn ) for each image in a batch separately . This is the sampling version of ( St-Yves & Naselaris , 2017 ) where the readout location is weighted spatially with a Gaussian profile . In our case , µn and Σn are learned via the reparametrization trick ( Kingma & Welling , 2014 ) . Initializing Σn large enough ensures that there is gradient information available to learn µn reliably . During training , Σn shrinks as the estimate of the neuron position improves . During evaluation we always use the position defined by µn , making the readout deterministic . This version of the Gaussian readout has c+ 7 parameters per neuron ( 2 for µ , 4 for Σ because the linear mapping in the reparametrization trick is 2× 2 , and 1 for the scalar bias ) . The second innovation of our Gaussian readout is to couple the location estimation of single neurons by exploiting the retinotopic organization of primary visual cortex ( V1 ) and other areas . Since V1 preserves the topology of visual space , we estimate a neuron ’ s receptive field location from its position pn ∈ R2 along the cortical surface available from the experiments . To that end , we learn a common function µn = f ( pn ) represented by a neural network that is shared across all neurons ( Fig . 2 ) . Since we work with neurons from local patches of V1 , we model f as a linear fully connected network . This approach turns the problem of estimating each neuron ’ s receptive field location from limited data into estimating a single linear transformation shared by all neurons , and reduces the number of per-neuron parameters to c+ 5 . We initialized the Readout Position Network to a random orthonormal 2-2 matrix scaled by a factor which was optimized in hyper-parameter selection . Finally , when training on several scans of anatomically matched neurons from the same mouse ( see Data ) , we share the feature weights vk across scans . To account for differences in spike inference between scans , we introduced a scan-specific scale and bias for each neuron after the linear readout . We mention in the respective sections whether features are shared or not . The bias of each readout is initialized with the average response on the training set . The effects of both feature sharing and learning from cortical anatomy on the performance of the readout are shown in the Appendix . Training The networks were trained to minimize Poisson loss 1m ∑m i=1 ( r̂ ( i ) − r ( i ) log r̂ ( i ) ) where m denotes the number of neurons , r̂ the predicted neuronal response and r the observed response . We used early stopping on the correlation between predicted and measured neuronal responses on the validation set ( Prechelt , 1998 ) : if the correlation failed to increase during any 5 consecutive passes through the entire training set ( epochs ) , we stopped the training and restored the model to the best performing model over the course of training . We found that this combination of Poisson objective and early stopping on correlation yielded the best results . After the first stop , we decreased the learning rate from 5× 10−3 twice by a decay factor of 0.3 , and resumed training until it was stopped again . Network parameters were iteratively optimized via stochastic gradient descent using the Adam optimizer ( Kingma & Ba , 2015 ) with a batch size of 64 . Once training completed , the trained network was evaluated on the validation set to yield the score used for hyper-parameter selection . The hyper-parameters were then selected with a Bayesian search ( Snoek et al. , 2012 ) of 100 trials and subsequently kept fixed throughout all experiments . Only the scale of the readout regularization was fine-tuned with additional Bayesian searches for the cases of different amounts of data independently . In transfer experiments , we froze all parameters of the core and trained a new readout only . Evaluation We report performance as fraction oracle ( see Walker et al. , 2019 ) , which is defined as the correlation of the predicted response and the observed single-trial test responses relative to the maximally achievable correlation measured from repeated presentations . We estimated the oracle correlation using a jackknife estimator ( correlation of leave-one-out mean against single trial ) . Per data point , we trained 25 networks for all combinations of five different model initializations and five random partitions of the neurons into core and transfer sets . The image subsets were drawn randomly once and kept fixed across all experiments except in Fig . 5 where the full neuron set was used and 5 random partitions of image subsets were drawn instead . We selected the best performing models across initializations and calculated 95 % confidence intervals over neuron- or image seeds .

Summary: The authors adopt a data-driven approach to neural system identification. They train a neural network consisting of a "core" and a "readout" in an end-to-end fashion to learn stimulus (visual inputs) -- response (single neuron activity) pairs. Since the core is shared across neurons, these stimulus-response pairs can be learnt in a massively parallel manner. In particular, they propose a novel readout mechanism that is parameter efficient and drives the core to learn better and generalizable features of the visual inputs. They find that their representations are more suited to predict neural responses in the mouse visual cortex when compared to representations derived from task-driven learning, especially in the context of transfer to previously unseen animals. Lastly, they also observe that the combination of their core+readout is more sample efficient than other naive alternatives.