Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The authors train a neural net to predict responses of mouse V1 L2/3 neurons to visual stimulation. The NN has a "core" that is shared between all neurons, and a neuron-specific readout. They train the core on multiple animals and find that it can generalize well: it can be used in a new animal and (with sufficient training of the readouts) achieve high performance. They also use a neat approach of constraining the readout weights (receptive field location) using the known retinotopy of V1. Finally, they show that their network outperforms task-trained ones at predicting V1 responses.

This is nice work overall. I have a few suggestions:

1) It might be worth considering other measures of performance, different from the normalized correlation coefficient. Recent work shows that this measure can have unintended bias, being substantially affected by trial-to-trial variability.
See "The unbiased estimation of the fraction of variance explained by a model" from Pospisal and Bair (https://doi.org/10.1101/2020.10.30.361253) for details, and a potential solution.

2) 2-photon imaging can have issues at detecting single spikes (see this preprint, for example: Relationship between spiking activity and simultaneously recorded fluorescence signals in transgenic mice expressing GCaMP6,  https://doi.org/10.1101/788802). So the neural dataset could in principle show more multi-spike events than single-spike ones, or have other issues. This is inevitable of course with calcium imaging, but it makes the problematic to compare with previous work that used electrical recordings. E.g., I don't think it is possible to prove better performance for this work than the prior ones, because of this difference in recording methods. A good follow-up work should try this method on electrical recordings from (say) monkey, and compare with performance from the Cadena, Yamins, Kindel, Klindt, Batty, etc. studies.

review_title: Predicting V1 responses with less training data

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Summarize what the paper claims to contribute. 
The paper introduces a deep-network-based approach to regression of responses to natural stimuli in mouse primary visual cortex. There is closely related work in the literature, but this paper achieves very good performance, partly through a new way of accounting for neurons’ receptive-field positions. The paper also provides a helpful analysis of prediction performance versus numbers of images and neurons used to train the model, and shows that the already excellent performance is not saturated with respect to the number of images. The work also shows that features learned by a core network generalize well across different mice. 

List strong and weak points of the paper.
Strong points: 
-	Empirical modelling of neural responses has a long tradition, and the results in this paper are state-of-the-art. 
-	Thorough and insightful positioning in the recent literature. 
-	Expert execution in terms of details of the technical work. 
-	The method of parameterizing the receptive field location is well-motivated and effective. 
-	The analyses are interesting and provide useful insights. 

Weak points: 
I wouldn’t characterize any part of the paper as weak, but here are some minor suggestions to further strengthen it: 
-	Say more about how the model can be used, or what insights might arise from it (there is only a short comment on inception loops). 
-	Say more about limitations as a model of neural responses, particularly with respect to dynamics. While the method is impressive with respect to short-time-window responses, system identification methods have long been used to study temporal responses. I think a short comment on this scope limitation would help to further contextualize the paper. 
-	An additional way to contextualize the results might be relative to the total number of neurons in L2/3 of VISp (I believe ~200K). Does this number have any significance relative to the dimension of the core-network output, or the number of recorded neurons? 
-	Consider adding a sentence on ethics oversight regarding the animal experiments. 
-	Consider adding a few further details of the experiments. 

Clearly state your recommendation (accept or reject) with one or two key reasons for this choice. 
I recommend that the paper be accepted. The paper addresses a long-standing problem very well. It introduces a new method that is well justified and effective. Overall, the performance is impressive, and the analyses are well done. 

Ask questions you would like answered by the authors to help you clarify your understanding of the paper and provide the additional evidence you need to be confident in your assessment. 
What are the kernel sizes in the core?
Which hyperparameters are adjusted in the hyperparameter search?

Provide additional feedback with the aim to improve the paper. 
I was confused by the following sentences: 
“… both readouts assume that the receptive field of each neuron is the same across features” 
“… readout has c + 7 parameters per neuron …” (I only see c+6.)
“Fig 5 for the factorized readout …” (I didn’t get it until reading it four times and looking for these results in Figure 5 twice.) 

review_title: Innovative state-of-the-art predictor of mouse visual responses

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors adopt a data-driven approach to neural system identification. They train a neural network consisting of a "core" and a "readout" in an end-to-end fashion to learn stimulus (visual inputs) -- response (single neuron activity) pairs. Since the core is shared across neurons, these stimulus-response pairs can be learnt in a massively parallel manner. In particular, they propose a novel readout mechanism that is parameter efficient and drives the core to learn better and generalizable features of the visual inputs. They find that their representations are more suited to predict neural responses in the mouse visual cortex when compared to representations derived from task-driven learning, especially in the context of transfer to previously unseen animals. Lastly, they also observe that the combination of their core+readout is more sample efficient than other naive alternatives.

Pros:
One of the major positives about this paper is the presented dataset. It seems to be relatively large and well-curated. This can certaily support several follow-up studies.

The authors identify that "global" use of features (i.e. the full hXwXc representational tensor) in the readout is a wasteful strategy (in terms of learned parameters per neuron) and instead adopt a local approach where they only select specific feature columns per neuron to drive the readout. Though this is of minor technical novelty, this constraint forces the core to learn appropriate representations while allowing the entire module to be more data efficient, given the big reduction in the number of free parameters.

The sample efficiency studies are neat and informative. The dissociations gleaned from diff-core/best-readout vs best-core/diff-readout scenarios are useful. Though it needs more work to fully justify this claim, their demonstration that transferred representations seem to be more effective than direct training is surprising and interesting.

Cons:
Though this study is certainly valuable, the manuscript needs several clarifications before it can be publication-ready.

(i) The authors seek to develop better core representations indirectly by controlling the readout mechanism. This is fine, but there is little justification as to why they chose the current "core" architecture. This choice contains arbitary decisions (such as including depth-separable convolutions) that are not justified. Was there a systematic procedure behind a search that led to this architecture? Were other non-standard architecures tested?

(ii) Figure 2 currently seems to be adding very little value and needs to be improved. Given that the proposed readout mechanism was a major contribution of this paper, the authors could have used the Fig. 2 space to visually depict this readout procedure, on top of the readout position network. The arrow to a neuron is also a bit misleading.

(iii) One of my main concerns is with respect to the liberal use of the term "generalization". The authors repeatedly state that train-val-test splits were based on neurons and not images. This, coupled with the fact that their readouts leverage retinotopy, it is surprising that the authors never discuss the spatial segregation of the "held-out" neurons (say H) from the neurons in the training set (say T). If most H neurons were spatially proximate to T neurons, then this amounts to an "interpolation" regime for the network as opposed to "extrapolation". If my understanding here is wrong, could the authors please clarify why?

(iv) The authors report that transfered "core" representations work better than direct-training in their generalization experiments. This result is surprising and needs to be more strongly justified computationally. Is it possible that a sub-optimal training regime was used for direct-training? Is this anyhow related to issue (iii) raised above?

(v) The authors report that task-driven cores (such as VGG-16 pretrained on imagenet) perform badly in generalizaing across animals. Is this due to impoverished data regimes? Or are there more systemic issues? Also, VGG-16 isn't the best ventral stream model that best fits neural data. Do the authors think that this claim would hold for more recent task-driven systems, like CORnet-S for example.

(vi) Though not necessary for this manuscript per se, it would be helpful to test the usefulness of the generalizable core representations presented here for visual tasks supported by early visual areas. Perhaps some commentary on this would be nice.

Minor:
"Code for the analyses and the weights of the best generalizing representation will be shared in the final version of the paper". The authors do not commit to making the dataset public. Is this oversight or intentional?

how sensitive to number neurons in a scan?

Clarity: (Fig. 4 caption) "a fully trained core": I think the authors are referring to a core trained on all available data, which is different from "fully training" a network as this alludes to loss saturation. Also language like "a sub-optimal" core is vague and misleading.
review_title: Review for "Generalization in data-driven models of primary visual cortex"

final_score: rating: 7: Good paper, accept
review_content: The paper presents an experimental study on predicting the responses of mice V1 neurons with computational models. The paper advances a few contributions:

1. Confirm that task-driven models based on object recognition, are outperformed by data-driven models for predicting single neuron responses.
2. Show that training a shared model of neural responses on data from several animals and several neurons leads to models that transfer well to data from new neurons and new animals.
3. Introduce a novel readout mechanism that allows models to be shared fully across neurons which in turn helps transfer learning.

I think this paper is interesting and it should be presented at ICLR. I am not an expert in this specific sub-field so I am not qualified to make suggestions or evaluate the experimental design. I will leave here a few suggestions that I hope you will consider for a camera ready version.

1. Is the claim that, in mice, task-driven models are outperformed by data-driven models fully justified? I have no trouble believing that object recognition is not the right task for mice, but there is a fair chance we just haven't found a good task yet, and that we might find it in the future. If this is correct, it might be worth mentioning in the introduction or discussion.

2. I think the Introduction could do a better job of anticipating the implications of this study, similarly, the discussion is a bit dry and does little more than just repeating the results. As I mentioned, I am not especially familiar with the literature in this particular subfield, so I had a hard time imagining what I should learn about visual systems in general from your study. What can we do with this new information?

3. Would it be possible to visualize the receptive fields you learn? Maybe some unit maximization technique could be sufficient. I think it would be cool to see a few.

4. It might make sense to move the training regimes for direct training, within- and across-animal transfer to the method section and explain the data splitting, training and evaluation after those have been introduced. The reader will know why you are designing your datasets and training a certain way, which might make it easier to follow.

Thank you again for sharing these cool ideas and results, I hope my suggestions help.
All the best!
review_title: Interesting findings

Meta-review: This paper has received four positive reviews. The main intellectual contribution of the paper is the introduction of a novel readout mechanism that allows models to be shared fully across neurons which in turn helps transfer learning across neurons and even across animals. The reviewers commented on the technical strength of the paper. At the same time, the main contribution remains relatively incremental from a technical standpoint, and while the approach may be of value to future work, the impact of the current study on neuroscience (which is the target here) is quite limited. Nonetheless, there seems to be sufficient enthusiasm from the reviewers to recommend this paper be accepted.
Decision: Accept (Spotlight)