Generalization in data-driven models of primary visual cortex

1 INTRODUCTION . A long lasting challenge in sensory neuroscience is to understand the computations of neurons in the visual system stimulated by natural images ( Carandini et al. , 2005 ) . Important milestones towards this goal are general system identification models that can predict the response of large populations of neurons to arbitrary visual inputs . In recent years , deep neural networks have set new standards in predicting responses in the visual system ( Yamins et al. , 2014 ; Vintch et al. , 2015 ; Antolík et al. , 2016 ; Cadena et al. , 2019a ; Batty et al. , 2016 ; Kindel et al. , 2017 ; Klindt et al. , 2017 ; Zhang et al. , 2018 ; Ecker et al. , 2018 ; Sinz et al. , 2018 ) and the ability to yield novel response characterizations ( Walker et al. , 2019 ; Bashivan et al. , 2019 ; Ponce et al. , 2019 ; Kindel et al. , 2019 ; Ukita et al. , 2019 ) . Such a general system identification model is one way for neuroscientists to investigate the computations of the respective brain areas in silico . Such in silico experiments exhibit the possibility to study the system at a scale and level of detail that is impossible in real experiments which have to cope with limited experimental time and adaptation effects in neurons . Moreover , all parameters , connections and weights in an in silico model can be accessed directly , opening up the opportunity to manipulate the model or determine its detailed tuning properties using numerical optimization methods . In order for the results of such analyses performed on an in silico model to be reliable , however , one needs to make sure that the model does indeed replicate the responses of its biological counterpart faithfully . This work provides an important step towards obtaining such a generalizing model of mouse V1 . High performing predictive models need to account for the increasingly nonlinear response properties of neurons along the visual hierarchy . As many of the nonlinearities are currently unknown , one of the key challenges in neural system identification is to find a good set of characteristic nonlinear basis functions—so called representations . However , learning these complex nonlinearities from single neuron responses is difficult given limited experimental data . Two approaches have proven to be promising in the past : Task-driven system identification networks rely on transfer learning and use nonlinear representations pre-trained on large datasets for standard vision tasks , such as object recognition ( Yamins & DiCarlo , 2016 ) . Single neuron responses are predicted from a particular layer of a pre-trained network using a simple readout mechanism , usually an affine function followed by a static nonlinearity . Data-driven models share a common nonlinear representation among hundreds or thousands of neurons , and train the entire network end-to-end on stimulus response pairs from the experiment . Because the nonlinear representation is shared , it is trained via massive multi-task learning ( one neuron–one task ) and can be learned even from limited experimental data . Task-driven networks are appealing because they only need to fit the readout mechanisms on top of a given representation and thus are data-efficient in terms of the number of stimulus-response pairs needed to achieve good predictive performance ( Cadena et al. , 2019a ) . Moreover , as their representations are obtained independently of the neural data , a good predictive performance suggests that the nonlinear features are characteristic for a particular brain area . This additionally offers the interesting normative perspective that the functional representations in deep networks and biological vision could be aligned by common computational goals ( Yamins & DiCarlo , 2016 ; Kell et al. , 2018 ; Kubilius et al. , 2018 ; Nayebi et al. , 2018 ; Sinz et al. , 2019 ; Güçlü & van Gerven , 2014 ; Kriegeskorte , 2015 ; Khaligh-Razavi & Kriegeskorte , 2014 ; Kietzmann et al. , 2019 ) . In order to quantify the fit of the normative hypothesis , it is important to compare a given representation to other alternatives ( Schrimpf et al. , 2018 ; Cadena et al. , 2019a ) . However , while representations pre-trained on ImageNet are the state-of-the-art for predicting visual cortex in primates ( Cadena et al. , 2019a ; Yamins & DiCarlo , 2016 ) , recent work has demonstrated that pre-training on object categorization ( VGG16 ) yields no benefits over random initialization for mouse visual cortex ( Cadena et al. , 2019b ) . Since random representation should not be characteristic for a particular brain area and other tasks that might yield more meaningful representations have not been found yet , this raises the questions whether there are better ways to obtain a generalizing nonlinear representation for mouse visual cortex . Here , we investigate whether such a generalizing representation can instead be obtained from datadriven networks . For this purpose , we develop a new data efficient readout which is designed to push non-linear computations into the core and test whether this core has learned general characteristic features of mouse visual cortex by applying the same criteria as for the task-driven approach : The ability to predict a population of unseen neurons in a new animal ( transfer learning ) . Specifically , we make the following contributions : 1 We introduce a novel readout mechanism that keeps the number of per-neuron parameters at a minimum and learns a bivariate Gaussian distribution for the readout position from anatomical data using retinotopy . With this readout alone , we surpass the previous state-of-the-art performance in direct training by 7 % . 2 We demonstrate that a representation pre-trained on thousands of neurons from various animals generalizes to neurons from an unseen animal ( transfer learning ) . It exceeds the direct training condition by another 11 % , setting the new state-of-the-art and outperforms a task-driven representation—trained on object recognition—by about 33 % . 3 We then show that this generalization can be attributed to the representation and not the readout mechanism , indicating that the data-driven core indeed captures generalizing features of cortex : A representation trained on a single experiment ( 4.5k examples ) in combination with a readout trained on anatomically matched neurons from four experiments ( 17.5k examples ) did not achieve this performance . 4 Lastly , we find that transfer learning with our data-driven core is more data-efficient than direct training , achieving the same performance with only 40 % of the data . Ne ur on se ts Image sets Unique anim als Evaluation 11-S 4-S : matched 4-S : diff animals 1-S Figure 1 : Scans and training sets . Overview of the datasets and how they are combined into different training sets . Each scan was performed on a specific set of neurons ( rows ) using a specific set of unique images ( columns ) . Repeatedly presented test images were the same for all scans . Some scans were performed on the same neuron but with different image sets ( first row ) . Colors indicate grouping of scans into training sets and match line colors in Fig . 5 to indicate which dataset a representation/core ( not the readout ) was trained on . 2 METHODS . 2.1 DATA . Functional data The data used in our experiments consists of pairs of neural population responses and grayscale visual stimuli sampled and cropped from ImageNet , isotropically downsampled to 64× 36 px , with a resolution of 0.53 ppd ( pixels per degree of visual angle ) . The neural responses were recorded from layer L2/3 of the primary visual cortex ( area V1 ) of the mouse , using a wide field two photon microscope ( Sofroniew et al. , 2016 ) . Activity was measured using the genetically encoded calcium indicator GCaMP6s . V1 was targeted based on anatomical location as verified by numerous previous experiments performing retinotopic mapping using intrinsic imaging . We selected cells based on a classifier for somata on the segmented cell masks and deconvolved their fluorescence traces ( Pnevmatikakis et al. , 2016 ) . We did not filter cells according to visual responsiveness . The stimulation paradigm and data pre-processing followed the procedures described by Walker et al . ( 2019 ) . A single scan contained the responses of approximately 5000–9000 neurons to up to 6000 images , of which 1000 images consist of 100 unique images which were presented 10 times each to allow for an estimate of the reliability of the neuron ( see Appendix for a detailed description of the datasets ) . We used the repeated images for testing , and split the rest into 4500 training and 500 validation images . The neural data was preprocessed by normalizing the responses of the neurons by their standard deviation on the training set . To put the number of recorded neurons per scan into perspective , assuming that V1 has an area of about 4mm2 , that L2/3 is about 150-250µm thick and has a cell density of 80k excitatory cells per mm3 , entire V1 L2/3 should contain about 48k - 80k neurons ( Garrett et al. , 2014 ; Jurjut et al. , 2017 ; Schüz & Palm , 1989 ) , similar to the maximum number of neurons that we train a model on ( 72k neuron , 11-S , Fig . 1 , orange ) . Note , however , that this does not mean that these 72k neurons sample V1 or the visual field of a mouse evenly because of possible experimental biases in the choice of the recording location . All together , we used 13 scans from a total of 7 animals ( Fig . 1 ) . Each scan is defined by the set of neurons it was performed on ( rows/neuron sets in Fig . 1 ) and the set of images that were shown ( columns/image sets in Fig . 1 ) . Different image sets had non-overlapping training/validation images , but the same test images . Some of the scans were performed on the same neurons , but with different sets of natural images ( first row in Fig . 1 ) . These neurons were matched across scans by cross-correlating the structural scan planes against functionally recorded stacks ( Walker et al. , 2019 ) . Stitching data from several scans in this way allowed us to increase the number of image presentations per neuron beyond what would be possible in a single scan . We combined these scans into different training sets ( one color–one training set in Fig . 1 ) and named each one of them—e.g . 11-S for a set with 11 Scans . The different sets are further explained in the respective experiments they are used in . All data from the seven mice used in this work has been recorded by trained personnel under a strict protocol according to the regulations of the local authorities at Balor College of Medicine . 2.2 NETWORKS AND TRAINING . The networks are split conceptually into two parts : a core and a readout . The core captures the nonlinear image representation and is shared among all neurons . The readout maps the features of the core into neural responses and contains all neuron specific parameters . Readout position network Scan field Feature vector at sampled position x y y' x' x y Topmost block of core Figure 2 : Using retinotopy to learn the readout position from anatomical data . The Gaussian readout for each neuron uses features from a single location on the final tensor of the core CNN ( bottom ) . The position is drawn from a 2D Gaussian for every image during training . The parameters of the Gaussian for each neuron are learned during training . The means of the Gaussians are predicted from each neuron ’ s coordinates on cortex by a Readout Position Network whose weights are shared across neurons and learned during training ( top ) . During testing , the mean of the Gaussian is used as the neuron ’ s position . Representation/Core We model the core with a four-layer convolutional neural network ( CNN ) , with 64 feature channels per layer . In each layer , the 2d-convolutional layer is followed by a batch normalization layer and an ELU nonlinearity ( Ioffe & Szegedy , 2015 ; Clevert et al. , 2015 ) . All convolutional layers after the first one are depth-separable convolutions ( Chollet , 2017 ) which we found to yield better results than standard convolutional layers in a search among different architecture choices . Readouts We compared two different types of readouts to map the nonlinear features of the core to the response of each neuron . For each neuron , a tensor of x ∈ Rw×h×c ( width , height , channels ) needs to be mapped to a single scalar , corresponding to the target neuron ’ s response . All of our readouts assume that this function is affine with a linear weight tensor w ∈ Rw×h×c , followed by an ELU offset by one ( ELU+1 ) , to keep the response positive . Furthermore , both readouts assume that in feature space the receptive field of each neuron does not change its position across features , but they differ in how this receptive field location is constrained and learned . The factorized readout ( Klindt et al. , 2017 ) factorizes the 3d readout tensor into a lower-dimensional representation by using a spatial mask matrix uij and a vector of feature weights vk , i.e . wijk = uijvk . The spatial mask uij is restricted to be positive and encouraged to be sparse through an L1 regularizer . Our novel Gaussian readout reduces the number of per-neuron parameters . It computes a linear combination of the feature activations at a single spatial position— parametrized as ( x , y ) coordinates —via bilinear interpolation ( Sinz et al. , 2018 ) . To facilitate gradient flow during training , we replace the spatial downsampling used in ( Sinz et al. , 2018 ) by a sampling step , which during training draws the readout position of each nth neuron from a bivariate Gaussian distribution N ( µn , Σn ) for each image in a batch separately . This is the sampling version of ( St-Yves & Naselaris , 2017 ) where the readout location is weighted spatially with a Gaussian profile . In our case , µn and Σn are learned via the reparametrization trick ( Kingma & Welling , 2014 ) . Initializing Σn large enough ensures that there is gradient information available to learn µn reliably . During training , Σn shrinks as the estimate of the neuron position improves . During evaluation we always use the position defined by µn , making the readout deterministic . This version of the Gaussian readout has c+ 7 parameters per neuron ( 2 for µ , 4 for Σ because the linear mapping in the reparametrization trick is 2× 2 , and 1 for the scalar bias ) . The second innovation of our Gaussian readout is to couple the location estimation of single neurons by exploiting the retinotopic organization of primary visual cortex ( V1 ) and other areas . Since V1 preserves the topology of visual space , we estimate a neuron ’ s receptive field location from its position pn ∈ R2 along the cortical surface available from the experiments . To that end , we learn a common function µn = f ( pn ) represented by a neural network that is shared across all neurons ( Fig . 2 ) . Since we work with neurons from local patches of V1 , we model f as a linear fully connected network . This approach turns the problem of estimating each neuron ’ s receptive field location from limited data into estimating a single linear transformation shared by all neurons , and reduces the number of per-neuron parameters to c+ 5 . We initialized the Readout Position Network to a random orthonormal 2-2 matrix scaled by a factor which was optimized in hyper-parameter selection . Finally , when training on several scans of anatomically matched neurons from the same mouse ( see Data ) , we share the feature weights vk across scans . To account for differences in spike inference between scans , we introduced a scan-specific scale and bias for each neuron after the linear readout . We mention in the respective sections whether features are shared or not . The bias of each readout is initialized with the average response on the training set . The effects of both feature sharing and learning from cortical anatomy on the performance of the readout are shown in the Appendix . Training The networks were trained to minimize Poisson loss 1m ∑m i=1 ( r̂ ( i ) − r ( i ) log r̂ ( i ) ) where m denotes the number of neurons , r̂ the predicted neuronal response and r the observed response . We used early stopping on the correlation between predicted and measured neuronal responses on the validation set ( Prechelt , 1998 ) : if the correlation failed to increase during any 5 consecutive passes through the entire training set ( epochs ) , we stopped the training and restored the model to the best performing model over the course of training . We found that this combination of Poisson objective and early stopping on correlation yielded the best results . After the first stop , we decreased the learning rate from 5× 10−3 twice by a decay factor of 0.3 , and resumed training until it was stopped again . Network parameters were iteratively optimized via stochastic gradient descent using the Adam optimizer ( Kingma & Ba , 2015 ) with a batch size of 64 . Once training completed , the trained network was evaluated on the validation set to yield the score used for hyper-parameter selection . The hyper-parameters were then selected with a Bayesian search ( Snoek et al. , 2012 ) of 100 trials and subsequently kept fixed throughout all experiments . Only the scale of the readout regularization was fine-tuned with additional Bayesian searches for the cases of different amounts of data independently . In transfer experiments , we froze all parameters of the core and trained a new readout only . Evaluation We report performance as fraction oracle ( see Walker et al. , 2019 ) , which is defined as the correlation of the predicted response and the observed single-trial test responses relative to the maximally achievable correlation measured from repeated presentations . We estimated the oracle correlation using a jackknife estimator ( correlation of leave-one-out mean against single trial ) . Per data point , we trained 25 networks for all combinations of five different model initializations and five random partitions of the neurons into core and transfer sets . The image subsets were drawn randomly once and kept fixed across all experiments except in Fig . 5 where the full neuron set was used and 5 random partitions of image subsets were drawn instead . We selected the best performing models across initializations and calculated 95 % confidence intervals over neuron- or image seeds .

Summary: The authors train a neural net to predict responses of mouse V1 L2/3 neurons to visual stimulation. The NN has a "core" that is shared between all neurons, and a neuron-specific readout. They train the core on multiple animals and find that it can generalize well: it can be used in a new animal and (with sufficient training of the readouts) achieve high performance. They also use a neat approach of constraining the readout weights (receptive field location) using the known retinotopy of V1. Finally, they show that their network outperforms task-trained ones at predicting V1 responses.