Generalization in data-driven models of primary visual cortex
Authors: Konstantin-Klemens Lurz
Mohammad Bashiri
Konstantin Willeke
Akshay Jagadish
Eric Wang
Edgar Y. Walker
Santiago A Cadena
Taliah Muhammad
Erick Cobos
Andreas S. Tolias
Alexander S Ecker
Fabian H. Sinz
Venue: ICLR 2021 Spotlight (2021)
Keywords: neuroscience
cognitive science
multitask learning
transfer learning
representation learning
network architecture
computational biology
visual perception

Abstract: Deep neural networks (DNN) have set new standards at predicting responses of neural populations to visual input.  Most such DNNs consist of a convolutional network (core) shared across all neurons which learns a representation of neural computation in visual cortex and a neuron-specific readout that linearly combines the relevant features in this representation. The goal of this paper is to test whether such a representation is indeed generally characteristic for visual cortex, i.e. generalizes between animals of a species, and what factors contribute to obtaining such a generalizing core. To push all non-linear computations into the core where the generalizing cortical features should be learned, we devise a novel readout that reduces the number of parameters per neuron in the readout by up to two orders of magnitude compared to the previous state-of-the-art. It does so by taking advantage of retinotopy and learns a Gaussian distribution over the neuron’s receptive field position.  With this new readout we train our network on neural responses from mouse primary visual cortex (V1) and obtain a gain in performance of 7% compared to the previous state-of-the-art network.  We then investigate whether the convolutional core indeed captures general cortical features by using the core in transfer learning to a different animal.  When transferring a core trained on thousands of neurons from various animals and scans we exceed the performance of training directly on that animal by 12%, and outperform a commonly used VGG16 core pre-trained on imagenet by 33%. In addition, transfer learning with our data-driven core is more data-efficient than direct training, achieving the same performance with only 40% of the data. Our model with its novel readout thus sets a new state-of-the-art for neural response prediction in mouse visual cortex from natural images, generalizes between animals, and captures better characteristic cortical features than current task-driven pre-training approaches such as VGG16.

Published as a conference paper at ICLR 2021
GENERALIZATION IN DATA-DRIVEN MODELS OF PRIMARY VISUAL CORTEX
Konstantin-Klemen Lurz, Mohammad Bashiri," Konstantin Willeke," 1-2 Akshay K. Jagadish, Eric Wang, 4.5 Edgar Y. Walker, 1,2,4-5
Santiago A. Cadena, 2,3 Taliah Muhammad, 4-5 Erick Cobos, 4-5
Andreas S. Tolias, Alexander S. Ecker," Fabian H. Sinz!
Institute for Bioinformatics and Medical Informatics, University of Tübingen, Germany
2 International Max Planck Research School for Intelligent Systems, Tübingen, Germany
3 Bernstein Center for Computational Neuroscience, University of Tübingen, Germany
4 Department for Neuroscience, Baylor College of Medicine, Houston, TX, USA
5 Center for Neuroscience and Artificial Intelligence, Baylor College of Medicine, Houston, TX, USA
6 Department of Computer Science / Campus Institute Data Science, University of Götringen, Germany
"konstant in. Lurz -urreun--tuebbngenn...... de, "fabiann saabiannsinzzuuiittuueiiinn
ABSTRACT
Deep neural networks (DNN) have set new standards at predicting responses of neural populations to visual input. Most such DNNs consist of convolutional network (core) shared across all neurons which learns representation of neural computation in visual cortex and neuron-specific readout that linearly combines the relevant features in this representation. The goal of this paper is to test whether such representation is indeed generally characteristic for visual cortex, i.e. gener- alizes between animals of species, and what factors contribute to obtaining such a generalizing core. To push all non-lineat computations into the core where the generalizing cortical features should be learned, devise novel readout that reduces the number of parameters per neuron in the readout by up to two orders of magnitude compared the previous state-of-the-an It does by taking advantage of retinotopy and learns Gaussian distribution over the neuron's 's receptive field po- sition. With this new readout we train our network on neural responses from mouse primary visual cortex (V1) and obtain a gain in performance of 7% compared to the previous state-of-the-arr network. We then investigate whether the convolutional core indeed captures general cortical features by using the core in transfer learning to a different animal. When transferring core trained on thousands of neurons from various animals and scans we exceed the performance of training directly on that animal by 12%, and outperform a commonly used VGG16 core pre-trained on imagenet by 33%. In addition, transfer learning with data-driven core is more data-efficient than direct training, achieving the same performance with only 40% of the data. Our model with its novel readout thus sets neural response prediction in mouse visual cortex from natural images, generalizes between animals, and captures better characteristic cortical features than current task-driven pre-training approaches such as VGG16.
state-of-the-art for
INTRODUCTION
long lasting challenge in sensory neuroscience understand the computations of neurons in the visual system stimulated by natural images (Carandini et al., 2005). Important milestones towards this goal are general system identification models that can predict the response of large populations of neurons to arbitrary visual inputs. In recent years, deep neural networks have set new standards in predicting responses in the visual system (Yamins et al., 2014; Vintch et al., 2015; Antolík et al., 2016; Cadena et al., 2019a; Batty et al., 2016; Kindel et al., 2017; Klindt et al., 2017: Zhang al., 2018; Ecker et al., 2018; Sinz et al., 2018) and the ability to yield novel response characterizations (Walker al., 2019; Bashivan et al., 2019; Ponce et al., 2019; Kindel al., 2019; Ukita al., 2019).

---

Published as a conference paper at ICLR 2021
Such general system identification model is one way for neuroscientists to investigate the computa- tions of the respective brain areas in silico. Such in silico experiments exhibit the possibility to study the system at scale and level of detail that is impossible in real experiments which have to cope with limited experimental time and adaptation effects in neurons. Moreover, all parameters, connections and weights in in silico model can be accessed directly, opening up the opportunity to manipulate the model or determine its detailed tuning properties using numerical optimization methods. In order for the results of such analyses performed on an in silico model to be reliable, however, one needs to make sure that the model does indeed replicate the responses of its biological counterpart faithfully. This work provides an important step towards obtaining such generalizing model of mouse V1.
High performing predictive models need to account for the increasingly nonlinear response properties of neurons along the visual hierarchy. As many of the nonlinearities are currently unknown, one of the key challenges in neural system identification is to find a good set of characteristic nonlinear basis functions so called representations. However, learning these complex nonlinearities from single neuron responses is difficult given limited experimental data. Two approaches have proven to be promising in the past: Task-driven system identification networks rely on transfer learning and use nonlinear representations pre- traitreined on large datasets for standard vision tasks, such object recognition (Yamins& & DiCarlo, 2016). Single neuron responses are predicted from a particular layer of pre-trained network using simple readout mechanism, usually an affine function followed by static nonlinearity. Data-driven models share a common nonlinear representation among hundreds or thousands of neurons, and train the entire network end-to-end on stimulus response pairs from the experiment. Because the nonlinear representation is shared, it is trained via massive multi-task learning (one neuron-one task) and can be learned even from limited experimental data.
Task-driven networks are appealing because they only need to fit the readout mechanisms on top of a given representation and thus are cata-efficient in terms of the number of seimulus-response pairs needed to achieve good predictive performance (Cadena et al., 2019a). Moreover, as their representations are obtained independently of the neural data, a good predictive performance suggests that the nonlinear features are characteristic for particular brain area. This additionally offers the interesting normative perspective that the functional representations in deep networks and biological vision could be aligned by common computational goals (Yamins & DiCarlo, 2016; Kell et al., 2018; Kubilius et al., 2018; Nayebi et al., 2018; Sinz et al., 2019; 9; Güçlü & van Gerven, 2014 2014; Kriegeskorte, 2015; Khaligh- Razavi & Kriegeskorte, 2014; Kietzmann et al., 2019). In order to quantify the fit of the normative hypothesis, it is important to compare given representation to other alternatives (Schrimpf al., 2018; Cadena et al., 2019a). However, while representations pre-trained on ImageNet are the state-of-the-art for predicting visual cortex in primates (Cadena et al., 20192; Yamins & DiCarlo, 2016), recent work has demonstrated that pre-training on object categorization (VGG16) yields no benefits over random initialization for mouse visual cortex (Cadena et al.. 2019b). Since random representation should not be characteristic for a particular brain area and other tasks that might yield more meaningful representations have not been found yet, this raises the questions whether there are better ways to obtain a generalizing nonlinear representation for mouse visual cortex.
Here, we investigate whether such generalizing representation can instead be obtained from data- driven networks. For this purpose, we develop a new data efficient readout which is designed to push non-linear computations into the core and test whether this core has learned general characteristic features of mouse visual cortex by applying the same criteria as for the task-driven approach: The ability to predict a population of unseen neurons in new animal (transfer learning). Specifically, we make the following contributions: of per-neuron parameters at a minimum and learns bivariate Gaussian distribution for the readout position from anatomical data using retinotopy. With this readout alone, we surpass the previous state-of-the-art performance in direct training by 7%. We demonstrate that a representation pre-trained on thousands of neurons from various animals generalizes to neurons from an unseen animal (transfer learning). It exceeds the direct training condition by another 1 11%, setting the new siite-of-the-aae and outperforms a task-driven representation -ariiaiiionntnnned on object recognition-a about 33%. We then show that this generalization can be attributed to the representation and not the readout mechanism, indicating that the data-driven core indeed captures generalizing features of cortex: A representation trained on a single experiment (4.5k examples) in combination with a readout trained on anatomically matched neurons from four experiments (17.5k examples) did not achieve this performance. Lastly, we find that transfer learning with our data-driven core is more data-efficient than direct training, achieving the same performance with only 40% of the data.
We introduce a novel readout mechanism that keeps the number
2

---

Published as a conference paper at ICLR 2021
Image sets
Figure Scans and training sets. Overview of the datasets and how they are combined into different training sets. Each sean was performed on a specific set of neurons (rows) using specific set of unique images (columns). Re- peatedly presented test images were the same for all scans. Some scans were performed on the same neuron but with different image sets (first row). Colors indicate grouping of scans into training sets and match line colors in Fig. to in- dicate which dataset a representation/core (not the readout) trained
Evaluation
11-5
4-S: matched
Neuron sets
4-5: diff animals
1-5
mummis anbun
METHODS
2.1 DATA
Functional data The data used in our experiments consists pairs of neural population responses and grayscale visual stimuli sampled and cropped from ImageNet, isotropically downsampled to 64 x 36 px, with a resolution 0.53 ppd (pixels per degree of visual angle). The neural responses were recorded from layer L2/3 of the primary visual cortex (area V1) of the mouse, using wide field two photon microscope (Sofroniew et al., 2016). Activity was measured using the genetically encoded calcium indicator GCaMP6s. VI was targeted based anatomical location verified by numerous previous experiments performing retinotopic mapping using intrinsic imaging. We selected cells based classifier for somata on the segmented cell masks and deconvolved their fluorescence traces (Pnevmatikakis et al., 2). We did not filter cells according to visual responsiveness. The stimulation paradigm and data pre-processing followed the procedures described by Walker al. (2019). A single scan contained the responses of approximately 5000-9000 neurons to up to 6000 images, of which 1000 images consist of 100 unique images which were presented 10 times each to allow for an estimate of the reliability of the neuron (see Appendix for detailed description of the datasets). We used the repeated images for testing, and split the rest into 4500 training and 500 validation images. The neural data was preprocessed by normalizing the responses of the neurons by their standard deviation on the training set. To put the number of recorded neurons per sean into perspective, assuming that VI has an area of about 4mm?, that L2/3 is about 150-250pm thick and has a cell density of 80k excitatory cells per mm3 entire V1 L2/3 should contain about 48k - 80k neurons (Garrett al., 2014; Jurjut et al., 2017; Schüz & Palm, 1989), similar to the maximum number of neurons that we train a model on (72k neuron, 11-S- Fig. 1, orange). Note, however, that this does not mean that these 72k neurons sample VI the visual field of mouse evenly because of possible experimental biases in the choice of the recording location.
All together, we used 13 scans from total of 7 animals (Fig. 1). Each scan is defined by the set of neurons it was performed on (rows/neuron sets in Fig. 1) and the set of images that were shown (columns/image sets in Fig. 1). Different image sets had non-overlapping training/validation images, but the same test images. Some of the scans were performed on the same neurons, but with different sets of natural images (first row in Fig. 1). These neurons were matched across scans by cors-correl the structural scan planes against functionally recorded stacks (Walker et al., 2019). Stitching data from several scans this way allowed us to increase the number of image presentations neuron beyond what would be possible in single scan. We combined these scans into different training sets (one color-on one training set in Fig. 1) and named each one them-e,g, e.g. 11-S for a set with 11 Scans. The different sets are further explained in the respective experiments they are used in. All data from the seven mice used in this work has been recorded by trained personnel under a strict protocol according to the regulations of the local authorities Balor College of Medicine.
2.2 NETWORKS AND TRAINING
The networks split conceptually into two parts: core and readout. The captures the nonlinear image representation and is shared among all neurons. The readout maps the features of the core into neural responses and contains all neuron specific parameters.
3

---

Published as a conference paper at ICLR 2021
Using retinotopy to learn the readout position from anatomical data. The Gaussian readout for each neuron uses features from a single location on the final tensor of the core CNN (bottom). The posi- tion is drawn from a 2D Gaussian for every image during training. The parameters of the Gaussian for each neuron are learned during training. The means of the Gaus- sians are predicted from each neuron's 's co- ordinates on cortex by a Readout Position Network whose weights neurons and learned during training (top). During testing, the mean of the Gaussian is used as the neuron's 's position.
Figure 2:
Scan fiold
Roadout position network
Faature vector at sampled posttion
Massale
Madel
shared across
Topmost block of core
Representation/Core We model the core with four-layer convolutional neural network (CNN), with 64 feature channels per layer. each layer, the 2d-convolutional layer is followed by a batch normalization layer and an ELU nonlinearity (loffe & Szegedy, 2015; Clevert et al., 2015). All convolutional layers after the first one are depth-separable convolutions (Chollet, 2017) which we found to yield better results than standard convolutional layers in search among différent architecture choices.
Readouts We compared two different types of readouts to map the nonlinear features of the core to the response of each neuron. For each neuron, a tensor of x E xxex (width, height, channels) needs to be mapped to single scalar, corresponding to the target neuron's response. All of readouts assume that this function is affine with a linear weight tensor w E Rwxxxxe followed by an ELU offset by one (ELU+1), to keep the response positive. Furthermore, both readouts assume that in feature space the receptive field of each neuron does not change its position across features, but they differ in how this receptive field location is constrained and learned.
The factorized readout (Klindt et al., 2017) factorizes the 3d readout tensor into a lower-dimensions representation by using a spatial mask matrix Bay and a vector of feature weights 2k2 i.e. Wiyk = u The spatial mask Waj is restricted to be positive and encouraged to be sparse through an I.1 regularizer.
Our novel Gaussian readout reduces the number of per-neuron parameters. It computes a linear combination of the feature activations at a single spatial position- parametrized as (x., y) coordi- nates -via bilinear interpolation (Sinz et al., 2018). To facilitate gradient flow during training, we replace the spatial downsampling used in (Sinz et al., 2018) by sampling step, which during training draws the readout position of each th neuron from bivariate Gaussian distribution [(pp: E.,) for each image in batch separately. This is the sampling version of (St- Yves & Naselaris, 2017) where the readout location is weighted spatially with a Gaussian profile. In our case, Ma, and E,, are learned via the reparametrization trick (Kingma & Welling, 2014). Initializing Σ,, large enough ensures that there is gradient information available to learn Ph reliably. During training, Σ,, shrinks as the estimate of the neuron position improves. During evaluation we always use the position defined by Mn, making the readout deterministic. This version of the Gaussian readout has c + 7 parameters per neuron (2 for p. 4 for Σ because the linear mapping the reparametrization trick is 2 2, and I for the scalar bias).
The second innovation of our Gaussian readout is couple the location estimation of single neurons by exploiting the retinotopic organization of primary visual cortex (VI) and other areas. Since V1 preserves the topology of visual space, we estimate a neuron's 's receptive field location from its position P.. E R along the cortical surface available from the experiments. To that end, we learn a common function Hn, = f (px)) represented by a neural network that is shared across all neurons (Fig. 2). Since we work with neurons from local patches of V1, we model f as linear fully connected network. This approach turns the problem of estimating each neuron's 's receptive field location from limited data into estimating single linear transformation shared by all neurons, and reduces the number of per-neuron parameters to c + 5. We initialized the Readout Position Network to a random orthonormal 2-2 matrix scaled by a factor which was optimized in hyper-parameter selection.
4

---

Published as a conference paper at ICLR 2021
Finally, when training on several scans of anatomically matched neurons from the same mouse (see Data), we share the feature weights U/ across scans. To account for differences in spike inference between scans, introduced scan-specific scale and bias for each neuron after the linear readout. We mention in the respective sections whether features are shared or not. The bias of each readout is initialized with the average response on the training set. The effects of both feature sharing and learning from cortical anatomy on the performance of the readout are shown in the Appendix.
Training The networks were trained to minimize Poisson loss 1 Σ (f(+) - r(() logr()) (()) where m denotes the number of neurons, f the predicted neuronal response and r the observed response. We used early stopping on the correlation between predicted and measured neuronal responses on the validation set (Prechelt, 1998): if the correlation failed to increase during any consecutive passes through the entire training set (epochs), we stopped the training and restored the model to the best performing model over the course of training. We found that this combination of Poisson objective and early stopping on correlation yielded the best results. After the first stop, decreased the learning rate from 5 x 10-3 twice by decay factor of 0.3, and resumed training until it was stopped again. Network parameters were iteratively optimized via stochastic gradient descent using the Adam optimizer (Kingma & Ba, 2015) with a batch size of 64. Once training completed, the trained network was evaluated on the validation set to yield the score used for ppper-parameter selection. The
were then selected with a Bayesian search (Snoek et al., 2012) of 100 trials and subsequently kept fixed throughout all experiments. Only the scale of the readout regularization was fine-tuned with additional Bayesian searches for the cases of different amounts of data independently. In transfer experiments, we froze all parameters of the core and trained a new readout only.
Evaluation We report performance as fraction oracle (see Walker et al., 2019), which is defined as the correlation of the predicted response and the observed single-trial test responses relative to the maximally achievable correlation measured from repeated presentations. We estimated the oracle correlation using a jackknife estimator (correlation of leave-one-out mean against single trial). Per data point, we trained 25 networks for all combinations of five different model initializations and five random partitions of the neurons into core and transfer sets. The image subsets were drawn randomly once and kept fixed across all experiments except in Fig. 5 where the full neuron set was used and 5 random partitions of image subsets were drawn instead. We selected the best performing models across initializations and calculated 95% confidence intervals over neuron- or image seeds.
3 RESULTS
We investigated the conditions under which data-driven core generalizes to new neurons in the same or different animals. We did this by pre-training a core on differently composed datasets (core sets, see 3.1) and testing that core in transfer learning new set of neurons (transfer set, see 3.1). Our main finding is that transferring a core trained on multiple scans (up to 35k unique images and 70k unique neurons) yields an improvement in performance of about 12% compared to a network directly trained on the single scan. This result was independent of whether the transferred core was trained on the same animal or not. By carefully choosing the Transfer learning conditions, we can attribute this boost in performance to the generalization of the core.
set of 5335 unique neurons from single scan (4.5k unique images)
3.1 TRANSFER LEARNING CONDITIONS
To test the generalization performance of the core, we used separate set of 1000 neurons which we call transfer set. These neurons were not used to train the transferred core but only to fine-tune new readout (note that Fig. 4 and 5 use different transfer sets, for intra- and inter- animal transfer respectively). A transfer set is subset of neurons, not images. Thus, each transfer set also had train, validation, and test split of its images. We compared the performance in transfer learning to that of network directly trained end-to-end on the transfer Because the core of the direct condition was not transferred, it could adapt to the neurons at hand giving it fair chance outperform the transferred cores.
subsets thereof (direct condition).
In the transfer learning conditions (all except direct), the core was always trained on a separate set of neurons (core set) and subsequently frozen, while the readout was always trained on top of
5

---

Published as a conference paper at ICLR 2021
the frozen core using the transfer set. Again, core set is subset of the neurons. not images. In order to quantify the generalization of the core, we needed to decouple the amount of data it takes to train the core from the amount of data it takes to train the readout. We thus considered two transfer conditions: seff-corre/bsttreaddeeee
and best-core/diff-realluu In the best-core/diff-raaiiee condition, the core was trained on the core set using all images, while the readout was trained on the transfer set using different numbers of images. This condition tests the data efficiency of the readout. We expect that better cores lead higher data efficiency in the readout, i.e. require less data to achieve good performance. In the cofff-orr/beet--ee reaaeout condition, we trained the core on the core set using different numbers of images while the readout was trained on the transfer set using all images. Thereby we tested how the generalization of the core is affected by the amount of data used to train it.
3.2 DIRECT TRAINING AND WITHIN ANIMAL/ACROSS NEURON GENERALIZATION
The following transfer experiments (Fig. 3 and 4) aim at investigating how the number of images and neurons provided to the core and the readout affects generalization to new neurons. To this end, used dataset that includes many images as possible while still providing reasonably large number of neurons. The 4-S mat ched dataset (first row in Fig. 1, green) provides 17596 images and 4597 neurons anatomically matched across scans. Each scan was performed on the same neurons, but showing different sets of images (test set was identical). The dataset was concatenated along the image dimension and split into core and transfer sets of 3597 and 1000 neurons.
Direct training We used the S-S chec core set to compare the performance of the Gaussian and factorized readout in the direct condition on the original core set before transfer. We tested both readouts with and without feature sharing (see Networks and training). The performance of the networks increased with the number of images (Fig. 3). also increased with the number of neurons, but the number of images had a far stronger effect. While the performance saturated quickly with the number of neurons, saturation w.r.t. the number of images did not seem to be reached, even when using all 17.5k images. The Gaussian readout outperformed the factorized readout in predictive performance by 7% fraction oracle for the full 4-S : mat ched dataset, reaching 0.886 + 0.005 and 0.826 + 0.005 fraction oracle respectively (tea While the Gaussian readout profits from feature sharing, the factorized readout is hurt by it (Fig. 3, light vs. dark colors). This might be because the spatial masks in the factorized readout are less constrained in contrast the Gaussian readout where the position network and the usage of only a single readout point exerts a stronger inductive bias. In all future experiments, we thus use feature sharing only for the Gaussian readout.
Within animal/across neuron generalization For both readouts, the generalization performance of the learned core, tested in the porrebbsstreadoo condition, increased with the number of images used to train the core (Fig. 4, pink). The cores and readouts were trained on the core and transfer set of the 4-S ma::cehdd dataset. As before, the Gaussian readout outperformed the factorized readout, exhibiting stronger increase in performance with the number of images and a better final performance when the entire dataset was used to train the core. Even for a core trained on few data, readout can yield good performance if it has access to enough images (pink line). Importantly,
# images per scan
Figure 3: Performance of end-to-end trained networks. Performance for different subsets of neu- rons (linestyle) and number of training examples (x-axis). The same core architecture was trained for two different readouts with and without fea- ture sharing (color) on the matched neurons of the 4-S : mat ched core set (Fig. green). Both net- works show increasing performance with number of images. However, the network with the Gaus- sian readout achieves higher final performance (light blue vs. orange). While the Gaussian readout profits from feature sharing (light vs. dark blue), the factorized readout is hurt by (yellow vs. or- ange). Shaded areas depict 95% confidence inter- vals across random picks of the neuron subsets.
50
1000
2500
4399
0.9
Fraction oracle
0.7
0.5
Readout
# neurons
Factorized w/ fis
50
0.3
Factorized w/o F-s
500
Gaussian w/ f-s
3597
Gaussian w/o f-s
0.1
200
4000
10000 # images
17596

---

Published as a conference paper at ICLR 2021
A
Factorized readout
B
Gaussian readout
# images per readout
1000
2500
1000
2500
4399
0.9
Fraction oracle
0.7
direct clff-core/best-readuut
0.5
0.3
best-coreldiff-readouu
0.1
200
4000
10000 # images
17596
200
4000
10000
17596
Figure 4: Generalization to other neurons in the same animal. A core trained on 3597 neurons and up to 17.5k images generalizes to new neurons (pink and yellow line). core trained on the full data yields very good predictive performance even when the readout is trained on far less data (yellow). If the readout trained with all data, even a core trained on few data can yield a good performance (pink). Both transfer conditions outperform network directly trained end-to-end on the transfer dataset (brown). For the full dataset, all training conditions converge to the same performance. Except in the best-core/diff-ree ceaadud condition for very few training data, the Gaussian readout (B) outperforms the factorized readout (A). The data for both the training and transfer comes from the mat ched dataset (Fig green). Not that the different number images be from the core or transfer set, depending on the transfer condition.
0.8
Figure Generalization across ani- mals. Prediction performance in frac- tion oracle correlation as function of training examples in the transfer set for Gaussian readout (x-axis) and differ- ent ways obtain the core (colors). The transfer training was performed on the evaluation dataset (blue, Fig 1). Cores trained on several scans used in transfer learning outperform direct training on the transfer dataset (blue line; direct condition).
Fraction oracle
0.6
0.4
direct
1-S
11:5: cortex
random VGG16
4 S: matched
random care 128
0.0
4-5: diff animals random care 64
50 500 1000
2500 # images
4472
except for very low numbers of images, the fastest increase in performance occurred in the best- core/diff-readlon condition, where a core trained on all images (but different neurons) was frozen and a new readout was trained on the transfer set for varying numbers of images (yellow lines). This result shows that a data-driven core provides general characteristic features for mouse V1, and these features generalize to new neurons. Importantly, for 4k images (about the size of typical rrrf-oor/diif-reudoout experiment), the performance of the best-core/diff- readout was approximately 7% better than the
performance of the tiffffoor/beet-edddooo condition (0.834 + 0.003 and 0.780 + 0.007 fraction oracle respectively). This observation that readout on all 17.5k images on a core from 4k images could not reach the performance of readout on 4k images on performance due to the generalizing core and not the readout.
from 17.5k images suggests that the better
3.3 GENERALIZATION ACROSS ANIMALS
So far, we tested generalization performance of data-driven cores to different neurons the same animal using the 4 :ma ched dataset. A stronger test for generalization is transfer learning across animals, where we may have to deal with inter-subject variability. To this end, we compared several cores derived from different core sets, random initialization, or ImageNet pre-trale pre-training in terms of their generalization to neurons from a mouse which was not used to train the core. The transfer set consisted of 5335 neurons from different mouse presented with images that were also in the
7

---

Published as a conference paper at ICLR 2021
core set (Fig. 1, blue). Note that performance was still evaluated a set of test images that were neither used to train core nor readout. Apart from the previously used core set with anatomically matched neurons (4 (4-S= :mt maeched, Fig. 1, green), we also trained a another animal (1-S. Fig. 1, purple), and on a set of four scans from different animals with different images (4-S: diff an ma 1s Fig. 1, red). Finally we also trained core on all datasets together, without any information on neuron matching or image sets (11-S. Fig. 1, orange). For completeness, we also compared to a task-driven VGG core (Simonyan & Zisserman, 2015), randomly initialized VGG core (reading out from conv2-2 for both) (Cadena et al., 2019b), our core (64 features) with randomly initialized weights, and a scaled up version of our core with the same number of features as the VGG (128). As before, we compared the generalization performance to model trained under the direct condition trained on the transfer set. Due to its better performance, we used the Gaussian readout for these experiments but version with the factorized readout can be found in the Appendix.
single scan from
We tested the generalization in the catoor/eddfffrraddore condition on a single transfer set (Fig. 5), i.e. all performances are reported on the same transfer set (5335 neurons) (blue, Fig. I and 5), but differ in where the core was trained (colors, Fig. 5). The most striking finding is that representations trained several scans not only reached better performances for fewer training images, but they also reached a better overall performance for the available data from a single scan compared to direcr training (Fig. 5 orange, green, and red vs. blue). Interestingly, this improvement did not require matched neurons in the core set (Fig. 5) and any potential negative effects of inter-subject variability were outweighed by the benefits of using multiple scans to train the core. The two cores trained on a single scan reached about the same final performance (Fig. 5, blue vs. purple), with a slightly better performance for the directly trained model, as expected. However, the final performance of both single scan models was about 10% smaller than the transfer performance from the models pre-trained on four and eleven scans, respectively (Fig. 5, blue and purple vs. red, green, and orange). We provide visualizations of the receptive fields produced via response maximization (Walker et al., 2019; Bashivan et al., 2019)- - of some example neurons obtained from our best model (Fig. 5, orange at 4472 images) in the Appendix, Fig. 4.
Consistent with previous work (Cadena et al., 2019b), pre-trained and random VGG16 cores per- formed similarly (Fig. gray, dotted vs. dash-dotted). Both VGG cores performed worse than directly trained data-driven core (Fig. 5 blue vs. gray dotted and dash-dotted). Our core with random weights (64 features) performs worst, demonstrating that training on neural data extracts characteristic features. Scaling up this random core to VGG size (128) does not match its performance which could be due to the interaction of the initialization with architectural differences.
Lastly, we investigated the effect of constraining all neurons to share the transformation from cortical location receptive field location by temporarily deactivating this feature (Fig. 5, 11-S: no cortex dashed orange), and found that this constraint was particularly useful for small numbers of images. An equivalent figure for Fig. for the factorized readout, table with a detailed overview over the most important results in numerical form, as well as a comparison with other performance metrics can be found in the Appendix.
4 RELATED WORK
The idea of a common cortical feature representation is wide-spread in sensory and systems neu- roscience, going back to the idea of V1 as a bank of Gabor filters or edge detectors (Jones et al., 1987; Olshausen & Field, 1996). substantial body of recent work focuses on feature representa- tions learned by training deep networks on vision tasks such as object recognition (Cadena et al., 2019a; Yamins & DiCarlo, 2016; Güçlü & van Gerven, 2014; Kriegeskorte. 2015; Khaligh- Razavi & Kriegeskorte, 2014). The brain-score! initiative compares different representations, resulting from pre-training on different tasks or different network architectures, with regards to performance of multiple neural prediction tasks (Schrimpf et al., 2018). In contrast to that, we focused whether multi-task learning between thousands of neurons leads to a generalizing representation.
While task-driven representations perform comparably to data-driven representations in primates (Ca- dena et al.. 201 10)),. Cadena et al. (2019b) recently demonstrated that they show almost no difference in predictive performance for mice. Our results corroborate this finding (Fig. 5) and show that
hhttps://wwe brai --ssoor..rrrr////
8

---

Published as a conference paper at ICLR 2021
data-driven representation outperforms task-driven representation by a substantial margin, even when tested on equal grounds with transfer learning.
Sharing representation between neurons is commonly used to learn data system identification networks (Cadena et al., 2019b;a; Batty et al., 2016; Sinz et al., 2018; Antolik et al., 2016; Klindt et al., 2017). Klindt et al. (2017) investigated the effect of the number of neurons and training examples onto the predictive performance of a data-driven network. However, this experiment was experimeer done on simulated data only and did not explore the generalization (transfer) performance of the learned representation. To the best of our knowledge, we are the first to systematically investigate the ability of data-driven representations to capture general characteristic features of visual cortex.
DISCUSSION
Machine learning applications in biology are often faced with limited amount of data. Especially, for recent deep learning approaches this poses a challenging problem. One promising way to approach is multi-task learning by training shared nonlinear representation on multiple tasks or subjects. This increases the data volume and can help to extract inductive biases to achieve better generalization. Here, we investigated particular instance of this problem: Modeling the responses of thousands of cortical neurons as a function of natural visual stimuli. We demonstrated that nonlinear data-driven representations, trained via massive multi-task learning through parameter sharing among thousands of neurons from mouse primary visual cortex, generalize to other neurons and mice, and significantly outperform common task-driven alternatives that are predictive for monkey V1 (Cadena et al., 2019b; Yamins & DiCarlo, 2016). As noted by Cadena et al. (2019b) already, this does not imply that all task-trained representations
necessarily suboptimal but rather indicates that we have not found the right task yet. But so far, our network sets a new state-of-the-ar for neural response prediction in direct training as well as transfer learning.
Our transfer results strongly suggests that data trained cores can capture features that are characteristic of mouse primary visual cortex, and the fact that the Gaussian readout can predict novel neurons from this core with relatively few training example corroborates this idea. In addition, the good transfer learning performance indicates that iuber-subject variability, which could affect the success of multi-task learning, does not seem to be major problem for this application. For that reason, we believe that our data-driven core is the most characteristic representation to date to predict mouse primary visual cortex and that it could be great tool in new experiments where data is scarce and/or training time is limited, such as the inception loops introduced by Walker et al. (2019) and Bashivan et al. (2019). To facilitate this, we share the weights of the trained representation together with its code online to allow others to predict neural responses with it. Other possible applications include for example the analysis of the learned feature representations investigate the operations in visual processing, or using the core to support vision tasks like image categorization (Li et al., 2019). Additionally, we also share the dataset that we evaluate our core on (Fig, 1, blue) so that other representations can be tested and compared with ours on the same data?.
Our results are based on "deconvolved" calcium traces of neural activity integrated over 500 ms. Whether data driven cores can also provide generalizing features for shorter time scales, recordings with electrodes, to dynamic responses, or of higher areas remains be seen in future studies.
ACKNOWLEDGMENTS
We thank A.K. Schalkamp, A. Nix, C. Blessing, S. Safarani, E. Froudarakis, and N. Patel for comments and discussions. KKL is funded by the German Federal Ministry of Education and Research through the Tübingen AI Center (FKZ: 01 011$18039A). FHS is supported by the Barl-Zeiss-Stiftung and acknowledges the support of the DFG Cluster of Excellence "Machine Learning- - New Perspectives for Science", EXC 2064/1, project number 390727645. SAC and ASE are supported by the German Research Foundation (DFG grant EC-479/1-I] This work was supported by an AWS Machine Learning research award to FHS. MB was supported by the International Max Planck Research School for Intelligent Systems. Supported by the Intelligence Advanced Research Projects Activity via Department of Interior/Interior Business Center contract number D16PC00003.
"https://gin, www.giii..nddogg///g.e g-m.ee.orgg/ajja///ro220

---

Published as a conference paper at ICLR 2021
REFERENCES
J. Antolík, S. B. Hofer, J. A. Bednar, and T. D. Mrsic-flogel. Model Constrained by Visual Hierarchy Improves Prediction of Neural Responses Natural Scenes. PLoS Comput Biol, pp. 1-22, 2016. ISSN 1553-7358. doi: 10. 10.1371/joural..pci. 1004927.
P. Bashivan, K. Kar, and J. DiCarlo. Neural Population Control via Deep ANN Image Synthesis. pp. 1-33, 2019. doi: 202477/ccm.2018.122
E. Batty, Merel, N. Brackbill, A. Heitman, A. Sher, A. Litke, E. J. Chichilnisky, and L. Paninski. Multilayer network models of primate retinal ganglion cells. Number Nips, 2016.
S. A. Cadena, G. H. Denfield, E. Y. Walker, L. A. Gatys, A. S. Tolias, M. Bethge, and A. S. Ecker. Deep convolutional models improve predictions of macaque VI responses to natural images. PLoS Comput Biol, pp. 201764, 2019a. doi: 10. 10.1101/201764. URL https:/ /www www.xii.orgg./o//oooo..ooo...... wwt.::/ee.......eee.e.........e.
content/early. 201/
A. Cadena, F.H. Sinz, Muhammad, E. Froudarakis, E. Cobos, E. Y. Walker, Reimer, Bethge, and A. S. Beker. How well do deep neural networks trained on object recognition characterize the mouse visual system? In NeurlPS 2019 Workshop Neuro AI, 2019b.
M. Carandini, J. B. Demb, V. Mante, J. Tolhurst, Y. Dan, Olshausen, L. Gallant, and C. Rust. Do We Know What the Early Visual System Does? J. Neurosci., 25(46): 11 2005. doi: 10. N0.522/NEURRSSCL377666
François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings 30th IEEE Conference Computer Vision and Pattern Recognition, CVPR 2017, 2017. ISBN 9781538604571. doi: 10.1109/CVPR.2017.1199
Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs). pp. 1-14, 2015. ISSN 09226389. doi: 10.3233/
188-1-61499-611
1760. URL http::pp... arxiv. org/ abs / habs/1111 1511. 07289.
A. S. Ecker, F. H. Sinz, E. Froudarakis, P. G. Fahey, S. A. Cadena, E. Y. Walker, E. Cobos, J. Reimer, S. Tolias, and M. Bethge. A rotation-equivariant convolutional neural network model of primary visual cortex. pp. 5-7, 2018. doi: 10011809..0 10504v1. URL http: aaxii. org/ /bs/ 1809. 10504.
Marina E. Garrett, Ian Nauhaus, James H. Marshel, Edward M. Callaway, Marina E. Garrett, James H. Marshel, Ian Nauhaus, and Marina E. Garrett, Topography and areal organization of mouse visual cortex. Journal of Neuroscience, 34(37):12587-12600 sep 2014. ISSN 15292401. doi: 10. 1523//NEUROSCC.112 14.20014 URL /twwtst////w.......t.t.... jneurosci wwtp://ee/rr/oooo////////// 37/12587https: www. junerrnsiiii wwweuu/ssc..oo//ooootet/////eeee wttp///ottet////////////o/
abstract.
Umut Güşlü and Marcel A. van Gerven. Deep Neural Networks Reveal Gradient in the Com- plexity of Neural Representations across the Brain's 's Ventral Visual Pathway. Technical Re-
port 27, 2014. URL htRp://///axx wwtp://wwwwwrrra/aaat.eaaasaeast orq haabb///111 64229 //dx. doi www.doo.oooooooo org/ 10. 1523/ N0.15533/NEEURON JNEUROSCI
2tt.-:11....11.:p:/..... httptttttttt::tt//ttttt //tppiv///////v///aa/aa hwarrg/bbs/1i/ 6422v1.
Sergey loffe and Christian Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. Technical report, 2015.
Jones, Palmer L A, and I.. A. Palmer. The two-dimensional spatial structure simple receptive fields in cat striate cortex. Journal of Neurophysiology, 1987. ISSN 0022-3077.
Ovidiu Jurjut, Petya Georgieva, Laura Busse, and Steffen Katzner. Learning enhances sensory processing in mouse VI before improving behavior. Journal of Neuroscience, 37(27):6460- 6474, jul 2017. ISSN 15292401. doi: 10. (0.5522NNEURRSCC..48........ 16.--17 //sites. www.le........oooo...oo...e www.///o....///////////.e/aaaaat a/nyu.
URL https:
Alexander J.E. Kell, Daniel L.K. Yamins, Erica N. Shook, Sam V. Norman- Harman-Hainner, and Josh H. McDermott. A Task-Optimized Neural Network Replicates Human Auditory Behavior, Predicts Brain Responses, and Reveals Cortical Processing Hierarchy. Neuron, 2018. ISSN 10974199. doi: 10.10166/.neumon 200.....0.0...0..0 0.03.444.
10

---

Published as a conference paper at ICLR 2021
S.-M. Shaligh-Razav and N. Kriegeskorte. Deep Supervised but Not Unsupervised Models May Explain IT Cortical Representation. PLoS computational biology, 10(11), 2014. doi: 10.iaa//ouumal..cbb. 1003915.
Tim C. Kietzmann, Courtney J. Spoerer, Lynn K.A. Sörensen, Radoslaw M. Cichy, Olaf Hauk, and Nikolaus Kriegeskorte. Recurrence is required to capture the representational dynamics of the human visual system. Proceedings of the National Academy of Sciences of the United States of America, 1 116(43) 10 2019. ISSN 10916490. doi: 10.10733pnas.190555 1905544116.
W. F. Kindel, Elijah D. Christensen, and Joel Zylberberg. Using deep learning to reveal the neural code for images in primary visual cortex. Technical report, 2017. URL http: / /////aaa /aarii.voorrggo................e org/ abs/ / 6 06208.
William F Kindel, Elijah D Christensen, and Joel Zylberberg. Using deep learning to probe the neural code for images in primary visual cortex. Journal of Vision, 19(4):29, 10 2019. doi: 10.1167/19.4.29. URL https:://ppp d' mens ons. httpeeaaawlpppllaeeooppppppbpple wtt.s/..aaar.ooooeeetteeoooo//ee 19 29. pdf....f.
gub.1113/52449tttsss///oo. jourmal/jov/939940/
Diederik P. Kingma and Jimmy Lei Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015 Conference Track Proceedings, 2015.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. 2nd International Conference on Learning Representations, ICLR 2014 Conference Track Proceedings, 2014.
D. A. Klindt, A. S. Ecker, T. Euler, and M. Bethge. Neural system identification for large populations separating "what" and "where". In Advances in Neural Information Processing Systems, number Nips, pp. 6-6, 2017. ISBN 778-1-5108-6096- doi: 10.12751 77775//mmn.c2207.00000 URL http: //arxiiv/
http//bs/1i/ /1711 02653.
N. Kriegeskorte. Deep neural networks a new framework for modelling biological vision and brain information processing. Annual Reviews of Vision Science, 2015. doi: 10.1101/029876.
Jonas Kubilius, Martin Schrimpf, Aran Nayebi, Daniel Bear, Daniel L.. K. Yamins, and James J. DiCarlo. CORnet: Modeling the Neural Mechanisms of Core Object Recognition. bioRxiv, pp. 408385, 2018. doi: 10.1401/4 URL https:://// www. bwwrxiv org/ content/10. 2101/40838500
Zhe Li. Wieland Brendel, Edgar Y Walker, Erick Cobos, Taliah Muhammad, Jacob Reimer, Matthias Bethge, Fabian H Sinz, Xaq Pitkow, and Andreas S Tolias. Learning From Brains How to Regularize Machines. 2019. URL http: htttp::/arxxv.orggg absgg//bb/11111 1t11/911110000....11
A. Nayebi, D. Bear, J. Kubilius, K. Kar, S. Ganguli, D. Sussillo, J. DiCarlo, and D. L. K. Yamins. Task-Driven Convolutional Recurrent Models the Visual System. (NeurIPS):1-14, 2018. URL httpt: 1807. 00053.
peattps://axiv... /ttp//aaxvvvvv.///a//////..aa../ http:////axxioooo/////////ee/// tttp:///t........oo.tttttptp// http::::/aaxvi...ooa.ooooo org/abs/ 00053. www.ttp:tttpp.m
B. A. Olshausen and D. J. Field. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, (866658)6607766609, 1996. doi: 10.1038/381607a0
Eftychios A. Pnevmatikakis, Daniel Soudry, Yuanjun Gao, Timothy A. Machado, Josh Merel, David Pfau, Thomas Reardon, Yu Mu, Clay Lacefield, Weijian Yang, Misha Ahrens, Randy Bruno, Thomas M. Jessell, Darcy S. Peterka, Rafael Yuste, and Liam Paninski. Simultaneous Denoising, Deconvolution, and Demixing of Calcium Imaging Data. Neuron, 2016. ISSN 10974199. doi: n0.1016/j.neuron.22 2015. 11.03....
Carlos R. Ponce, Will Xiao, Peter F. Schade, Till S. Hartmann, Gabriel Kreiman, and Margaret S. Livingstone. Evolving Images for Visual Neurons Using a Deep Generative Network Reveals Coding Principles and Neuronal Preferences. Cell, 177(4):999- 1009,, 2019, ISSN 10974172. doi: 10.1006j..eel..0000 0..000. URL http: /tpp:/dxx..o doi. http///xdd...oo//0..... /t/ wetpe//....ell........ 2019. 04.005.
11

---

Published as a conference paper at ICLR 2021
Lutz Prechelt. Automatic early stopping using validation: Quantifying the criteria. Neural Networks, 1144:761-777, 6 1998. ISSN 08936080. doi: (0006///899360000980000000
M. Schrimpf, J. Kubilius, H. Hong, N. J. Majaj, R. Rajalingham, E. B. Issa, K. Kar, P. Bashivan, J. Prescott-Roy, K. Schmidt, D. L. K. Yamins, and J. DiCarlo. Brain-Score: Which Artificial Neural Network for Object Recognition most Brain-Like? bioRxiv, pp. 407007, 2018. doi: 10.1 10.1101/407007. URL /enttps:/wwwer http:/://wwwwwwwwww/eotte wworxiv...vvvi.................. 407007.
Almut Schüzz and Günther Palm. Density of neurons and synapses in the cerebral cortex of the mouse. Journal of Comparative Neurology, 286(4):442-455, 1989. ISSN 10969861. doi: 10.1002//me URL https:// //ppmed ncbi nlm. nih, gov/2778101/.
Karen Simonyan and Andrew Zisserman. VERY DEEP CONVOLUTIONAL NETWORKS FOR LARGE-SCALE IMAGE RECOGNITION. Technical report, 2015. URL http: /www.
robots. ox uk/.
Sinz, A. S. Ecker, Fahey, E. Walker, Cobos, E. Froudarakis, Yatsenko, Pitkow, J. Reimer, and A. Tolias. Stimulus domain transfer in recurrent models for large scale cortical population prediction on video. Advances in Neural Information Processing Systems 31. 2018. doi: 10.1101/45267..
Fabian H. Sinz, Xaq Pitkow, Jacob Reimer, Matthias Bethge, and Andreas S. Tolias. Engineering a Less Artificial Intelligence, 2019. ISSN 10974199.
Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical Bayesian Optimization of Machine Learning Algorithms. Technical report, 2012.
Nicholas James Sofroniew, Daniel Flickinger, Jonathan King, and Karel Svoboda. A large field of view two-photon mesoscope with subcellular resolution for vivo imaging. elife, 2016. ISSN 2050084X. doi: 10.7554/eLifc.14472.
Ghislain St- vves and Thomas Naselaris. The feature- receptive field: an interpretable encoding model for complex feature spaces. The feature-weighted receptive field: interpretable encoding model for complex spaces, 2017. ISSN 1095-9572. doi: 10. 101/126318.
Jumpei Ukita, Takashi Yoshida, and Kenichi Ohki. Characterisation of nonlinear receptive fields of visual neurons by convolutional neural network. Scientific Reports, 91):3779, 2019. doi: 10.1038/s41
B. Vintch, J. A. Movshon, and E. P. Simoncelli. A Convolutional Subunit Model for Neuronal Responses in Macaque V1. The Journal of neuroscience the official journal of the Society for Neu- roscience, 35(44):14829 2015. 2015. ISSN 1529-2401. doi: 10. 10..522/NEURRSS122155113...
URL.
201 http:://wwww..aaarrrror.o.o///// // //www, http://pwbmee....
E. Y. Walker, F. H. Sinz, E. Cobos, T. Muhammad, E. Froudarakis, G. Fahey, A. Ecker, J. Reimer, X. Pitkow, and A. S. Tolias. Inception loops discover what excites neurons most using deep predictive models. Nature Neuroscience, 2019. ISSN 15461726. doi: (0.10088641593--0--00511 URL wwtp:tttp//.. / //xx ieri.org// 038 s/1//415599 10:41593-000 01 9-0517--
D. L. K. Yamins and J. DiCarlo. Using goal-driven deep learning models to understand sensory cortex. Nature Neuroscience, 19(3):356-366 2016. ISSN 1097-6256. doi: 10. 10.1038/nn.4244 URL http:/
/ttp:/aauurreeer///m// com/ do finder////eeee 11.10.. 1..1038/n..4244.
D. L. K. Yamins, H. Hong. C. F. Cadieu, E. A. Solomon, D. Seibert, and J. Di- Carlo. Performance-optimnzze hierarchical models predict neural responses in higher visual cortex. Proceedings of the National Academy of Sciences of the United States of America, 111(23):8619-24, 2014. ISSN 1091-6490. doi: 10.1073/pnas. 1403112111. URL http::/wwwww....eeeccaaaaataattt
fogi?
Yimeng Zhang, Tai Sing, Lee Ming, Li Fang, Liu Shiming, T.-S. Lee, M. Li, F. Liu, and S. Tang. Convolutional neural network models of V1 responses complex patterns. Technical report, 2018.
12


Reviews:
[{"review_id": "v0OBKdMoHsl", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summarize what the paper claims to contribute. \nThe paper introduces a deep-network-based approach to regression of responses to natural stimuli in mouse primary visual cortex. There is closely related work in the literature, but this paper achieves very good performance, partly through a new way of accounting for neurons’ receptive-field positions. The paper also provides a helpful analysis of prediction performance versus numbers of images and neurons used to train the model, and shows that the already excellent performance is not saturated with respect to the number of images. The work also shows that features learned by a core network generalize well across different mice. \n\nList strong and weak points of the paper.\nStrong points: \n-\tEmpirical modelling of neural responses has a long tradition, and the results in this paper are state-of-the-art. \n-\tThorough and insightful positioning in the recent literature. \n-\tExpert execution in terms of details of the technical work. \n-\tThe method of parameterizing the receptive field location is well-motivated and effective. \n-\tThe analyses are interesting and provide useful insights. \n\nWeak points: \nI wouldn’t characterize any part of the paper as weak, but here are some minor suggestions to further strengthen it: \n-\tSay more about how the model can be used, or what insights might arise from it (there is only a short comment on inception loops). \n-\tSay more about limitations as a model of neural responses, particularly with respect to dynamics. While the method is impressive with respect to short-time-window responses, system identification methods have long been used to study temporal responses. I think a short comment on this scope limitation would help to further contextualize the paper. \n-\tAn additional way to contextualize the results might be relative to the total number of neurons in L2/3 of VISp (I believe ~200K). Does this number have any significance relative to the dimension of the core-network output, or the number of recorded neurons? \n-\tConsider adding a sentence on ethics oversight regarding the animal experiments. \n-\tConsider adding a few further details of the experiments. \n\nClearly state your recommendation (accept or reject) with one or two key reasons for this choice. \nI recommend that the paper be accepted. The paper addresses a long-standing problem very well. It introduces a new method that is well justified and effective. Overall, the performance is impressive, and the analyses are well done. \n\nAsk questions you would like answered by the authors to help you clarify your understanding of the paper and provide the additional evidence you need to be confident in your assessment. \nWhat are the kernel sizes in the core?\nWhich hyperparameters are adjusted in the hyperparameter search?\n\nProvide additional feedback with the aim to improve the paper. \nI was confused by the following sentences: \n“… both readouts assume that the receptive field of each neuron is the same across features” \n“… readout has c + 7 parameters per neuron …” (I only see c+6.)\n“Fig 5 for the factorized readout …” (I didn’t get it until reading it four times and looking for these results in Figure 5 twice.) \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "gfNaT2yfPBY", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors adopt a data-driven approach to neural system identification. They train a neural network consisting of a \"core\" and a \"readout\" in an end-to-end fashion to learn stimulus (visual inputs) -- response (single neuron activity) pairs. Since the core is shared across neurons, these stimulus-response pairs can be learnt in a massively parallel manner. In particular, they propose a novel readout mechanism that is parameter efficient and drives the core to learn better and generalizable features of the visual inputs. They find that their representations are more suited to predict neural responses in the mouse visual cortex when compared to representations derived from task-driven learning, especially in the context of transfer to previously unseen animals. Lastly, they also observe that the combination of their core+readout is more sample efficient than other naive alternatives.\n\nPros:\nOne of the major positives about this paper is the presented dataset. It seems to be relatively large and well-curated. This can certaily support several follow-up studies.\n\nThe authors identify that \"global\" use of features (i.e. the full hXwXc representational tensor) in the readout is a wasteful strategy (in terms of learned parameters per neuron) and instead adopt a local approach where they only select specific feature columns per neuron to drive the readout. Though this is of minor technical novelty, this constraint forces the core to learn appropriate representations while allowing the entire module to be more data efficient, given the big reduction in the number of free parameters.\n\nThe sample efficiency studies are neat and informative. The dissociations gleaned from diff-core/best-readout vs best-core/diff-readout scenarios are useful. Though it needs more work to fully justify this claim, their demonstration that transferred representations seem to be more effective than direct training is surprising and interesting.\n\nCons:\nThough this study is certainly valuable, the manuscript needs several clarifications before it can be publication-ready.\n\n(i) The authors seek to develop better core representations indirectly by controlling the readout mechanism. This is fine, but there is little justification as to why they chose the current \"core\" architecture. This choice contains arbitary decisions (such as including depth-separable convolutions) that are not justified. Was there a systematic procedure behind a search that led to this architecture? Were other non-standard architecures tested?\n\n(ii) Figure 2 currently seems to be adding very little value and needs to be improved. Given that the proposed readout mechanism was a major contribution of this paper, the authors could have used the Fig. 2 space to visually depict this readout procedure, on top of the readout position network. The arrow to a neuron is also a bit misleading.\n\n(iii) One of my main concerns is with respect to the liberal use of the term \"generalization\". The authors repeatedly state that train-val-test splits were based on neurons and not images. This, coupled with the fact that their readouts leverage retinotopy, it is surprising that the authors never discuss the spatial segregation of the \"held-out\" neurons (say H) from the neurons in the training set (say T). If most H neurons were spatially proximate to T neurons, then this amounts to an \"interpolation\" regime for the network as opposed to \"extrapolation\". If my understanding here is wrong, could the authors please clarify why?\n\n(iv) The authors report that transfered \"core\" representations work better than direct-training in their generalization experiments. This result is surprising and needs to be more strongly justified computationally. Is it possible that a sub-optimal training regime was used for direct-training? Is this anyhow related to issue (iii) raised above?\n\n(v) The authors report that task-driven cores (such as VGG-16 pretrained on imagenet) perform badly in generalizaing across animals. Is this due to impoverished data regimes? Or are there more systemic issues? Also, VGG-16 isn't the best ventral stream model that best fits neural data. Do the authors think that this claim would hold for more recent task-driven systems, like CORnet-S for example.\n\n(vi) Though not necessary for this manuscript per se, it would be helpful to test the usefulness of the generalizable core representations presented here for visual tasks supported by early visual areas. Perhaps some commentary on this would be nice.\n\nMinor:\n\"Code for the analyses and the weights of the best generalizing representation will be shared in the final version of the paper\". The authors do not commit to making the dataset public. Is this oversight or intentional?\n\nhow sensitive to number neurons in a scan?\n\nClarity: (Fig. 4 caption) \"a fully trained core\": I think the authors are referring to a core trained on all available data, which is different from \"fully training\" a network as this alludes to loss saturation. Also language like \"a sub-optimal\" core is vague and misleading.", "extra_scores": {}, "extra_text": {}}, {"review_id": "D3DKskzm8j", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents an experimental study on predicting the responses of mice V1 neurons with computational models. The paper advances a few contributions:\n\n1. Confirm that task-driven models based on object recognition, are outperformed by data-driven models for predicting single neuron responses.\n2. Show that training a shared model of neural responses on data from several animals and several neurons leads to models that transfer well to data from new neurons and new animals.\n3. Introduce a novel readout mechanism that allows models to be shared fully across neurons which in turn helps transfer learning.\n\nI think this paper is interesting and it should be presented at ICLR. I am not an expert in this specific sub-field so I am not qualified to make suggestions or evaluate the experimental design. I will leave here a few suggestions that I hope you will consider for a camera ready version.\n\n1. Is the claim that, in mice, task-driven models are outperformed by data-driven models fully justified? I have no trouble believing that object recognition is not the right task for mice, but there is a fair chance we just haven't found a good task yet, and that we might find it in the future. If this is correct, it might be worth mentioning in the introduction or discussion.\n\n2. I think the Introduction could do a better job of anticipating the implications of this study, similarly, the discussion is a bit dry and does little more than just repeating the results. As I mentioned, I am not especially familiar with the literature in this particular subfield, so I had a hard time imagining what I should learn about visual systems in general from your study. What can we do with this new information?\n\n3. Would it be possible to visualize the receptive fields you learn? Maybe some unit maximization technique could be sufficient. I think it would be cool to see a few.\n\n4. It might make sense to move the training regimes for direct training, within- and across-animal transfer to the method section and explain the data splitting, training and evaluation after those have been introduced. The reader will know why you are designing your datasets and training a certain way, which might make it easier to follow.\n\nThank you again for sharing these cool ideas and results, I hope my suggestions help.\nAll the best!", "extra_scores": {}, "extra_text": {}}, {"review_id": "BvTAyPeetGw", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors train a neural net to predict responses of mouse V1 L2/3 neurons to visual stimulation. The NN has a \"core\" that is shared between all neurons, and a neuron-specific readout. They train the core on multiple animals and find that it can generalize well: it can be used in a new animal and (with sufficient training of the readouts) achieve high performance. They also use a neat approach of constraining the readout weights (receptive field location) using the known retinotopy of V1. Finally, they show that their network outperforms task-trained ones at predicting V1 responses.\n\nThis is nice work overall. I have a few suggestions:\n\n1) It might be worth considering other measures of performance, different from the normalized correlation coefficient. Recent work shows that this measure can have unintended bias, being substantially affected by trial-to-trial variability.\nSee \"The unbiased estimation of the fraction of variance explained by a model\" from Pospisal and Bair (https://doi.org/10.1101/2020.10.30.361253) for details, and a potential solution.\n\n2) 2-photon imaging can have issues at detecting single spikes (see this preprint, for example: Relationship between spiking activity and simultaneously recorded fluorescence signals in transgenic mice expressing GCaMP6,  https://doi.org/10.1101/788802). So the neural dataset could in principle show more multi-spike events than single-spike ones, or have other issues. This is inevitable of course with calcium imaging, but it makes the problematic to compare with previous work that used electrical recordings. E.g., I don't think it is possible to prove better performance for this work than the prior ones, because of this difference in recording methods. A good follow-up work should try this method on electrical recordings from (say) monkey, and compare with performance from the Cadena, Yamins, Kindel, Klindt, Batty, etc. studies.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper has received four positive reviews. The main intellectual contribution of the paper is the introduction of a novel readout mechanism that allows models to be shared fully across neurons which in turn helps transfer learning across neurons and even across animals. The reviewers commented on the technical strength of the paper. At the same time, the main contribution remains relatively incremental from a technical standpoint, and while the approach may be of value to future work, the impact of the current study on neuroscience (which is the target here) is quite limited. Nonetheless, there seems to be sufficient enthusiasm from the reviewers to recommend this paper be accepted.
Decision: Accept (Spotlight)