Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: Motivated by the literature showing that the primate visual system has two pathways ("ventral" and "dorsal") with distinct functional specializations, the authors try to build a model of a visual system that also has representationally and functionally distinct regions. They turn to calcium imaging data from mouse visual cortex, which allows them to compare learned representations of ANN to the neural representations found in distinct cortical areas of animals shown the same images. (There aren't publicly available datasets with simultaneous recordings from ventral and dorsal areas in primates, as far as I know.) The authors find that a two-pathway 3D ResNet model trained on the contrastive predictive coding (CPC) task and a dataset of natural videos has layers that overall explain more of the mouse calcium responses in more cortical areas than several baseline models, including the same architecture trained on the same dataset but with a supervised task. In addition, the two pathways in the self-supervised model transfer better to different tasks (object recognition and motion discrimination), which is reminiscent of some of the functional specialization between primate ventral and dorsal pathways.
main_review: I like this work overall and think the authors provide enough evidence to support their general conclusion: that it's **possible** for representationally and functionally specialized layers, resembling different regions of visual cortex, to emerge in a single computational model with the right architecture, trained on the right task. I'm not sure about two of their main specific claims, though: (1) that the particular architecture/loss function they identify here is the best model of mouse visual cortex that they could have reasonably identified, and (2) that their results have much to do with the distinction between the primate ventral and dorsal pathways. I'll discuss both of these below, but want to say up front that I think they can be remediated with some more careful model comparisons/controls and a clearer elaboration of the authors' central hypotheses. I'll start with (2) since it's a "higher level" response:

_Does the authors' work say something about "dorsal" and "ventral" visual pathways?_

As noted in the authors' introduction, ANNs optimized for object recognition (and recently in a self-supervised manner) are the best current models of multiple cortical areas in the primate ventral visual pathway. While there are a smaller number of cases where people have compared task-optimzied ANN representations to the primate dorsal pathway, the overall approach seems sound: these models predict neural activity or fMRI responses better than prior computational models, with some evidence that different tasks better account for dorsal areas than ventral areas and vice versa [A]. It would be nice, then, if there were single ANN models that could account for neural representations across both pathways in a "physical way" -- i.e., distinct layers of the single ANN could be convincingly mapped to distinct areas of visual cortex, extending the results of numerous studies for the ventral pathway alone (and primary visual cortex.) 

But there's another problem that seems just as important to tackle, and much closer to what the authors are actually doing here: explaining the neural activity of _rodent_ visual cortex using task-optimized ANNs. To date, success has been mixed: there's some indication that object-recognition optimized deep CNNs do no better than randomly initialized models at predicting mouse visual cortex responses [B], and the top-layer features of such models do not seem to account for rat neural representations as well as early- and mid-layer features [C]. (I'm surprised that at least the first of these papers isn't discussed in the Related Work section.) Since the task-optimized ANN theory of visual representations isn't meant to be primate-specific, it predicts that **some** combination of architecture, task, and dataset should yield good predictions of rodent visual cortex (assuming task optimization methods succeed.) It seems reasonable to explore tasks that are more ethologically relevant to mice than complex invariant object recognition, which may be a primate-specific ability.

All this is to say: I'm surprised by the framing of this work as mainly addressing the first of these problems, not the second. I don't know that there's good reason to think of the various mouse visual areas as analogous to primate ventral and dorsal pathways (or whether there's evolutionary homology); the couple of references the authors give here might try to justify this categorization, but it's not clearly explained in this paper. In looking at Figure 1, I wouldn't immediately conclude that there are two distinct types of visual area in mice; one might just as easily propose that they all belong to a single hierarchy, with AM near the top and P/LM near the bottom. So, independent of the actual results (on which I'll comment below), I'm worried that the main claim here isn't well aligned with what the paper shows; comparing to the work in [A], for instance, I'm still more inclined to think that primate ventral and dorsal pathways are better explained by adaptation for **different** tasks rather than a single task with a branched (and then merging) architecture.

If the authors were to present this work more as addressing the second problem above -- actually explaining multiple regions of rodent visual cortex with a single task-optimized model -- I think it could be a clearer contribution to the literature on explaining neural data with task-optimized ANNs. If the authors are very attached to the dorsal/ventral distinction in mice, though, I think they need to justify the connection between mouse vision and (far better studied) primate vision much better than they do here.

_How close is the 2-stream CPC model to the "correct" architecture/task for explaining mouse visual cortex?_

Beyond the concern about framing (which I don't think is semantic or trivial -- it makes a big difference for ideas about the evolution of vision in mammals!) I have some questions about the actual experiments that I hope the authors could address in a revision. I think, overall, their results are pretty solid -- and pretty consistent with concurrent work on modeling mouse visual cortex [D]. Arriving at similar conclusions (multiple pathways, non-supervised object recognition tasks) despite many differences in implementation is a reassuring sign.

1. In the authors' initial model comparison (Figure 2), an ImageNet trained model ("Object Categ") did better than the untrained model at explaining responses in several the visual areas. This is a different result and conclusion than the one in [B]. Could the authors comment on what might have led to this difference?

2. The set of baselines compared to the 1- and 2-stream CPC models is pretty limited -- as the authors themselves mention in the "Limitations" section. I think the paper would be much stronger if there were a broader exploration of architectures trained on the same loss function and/or a broader set of loss functions tested for the same architectures, with the dataset held constant. For instance, the 2-stream model does seem better than the 1-stream model in that it has a better match to the AM representation in its higher layers, but (as per the discussion above) it's not at all clear from anatomy or prior work that 2 is the "correct" number of pathways here. Why not try a 3-, 4-, etc.-stream model? Or a PredNet-like model that makes predictions in pixel space rather than latent space? I appreciate that there are resource contraints here, but I don't think we can draw very strong conclusions about which architectures and/or losses are really the best match for mouse visual cortex until there's somewhat broader sampling of model space. (Especially if the authors are attached to the dorsal-ventral distinction, it would be good to see that 2 is the "correct" number of pathways.)

3. The object categorization model, if it's a pretrained ImageNet model, was likely trained at a different image resolution than the CPC models (unless I missed this somewhere.) It's also not the closest comparison to the CPC models that the authors could make. What about a self-supervised 2-stream CNN trained on a static contrastive loss, on the same dataset and input resolution? If that model also failed to predict RSMs in the "dorsal" areas, it would make for a much stronger argument that dynamic prediction is a key part of the ethologically relevant task.

4. It's very interesting that the two streams learn such different representations (judging by Figures 3 and 5) without any asymmetry between the two pathways' architectures other than different initial weights (though it's reminiscent of the original AlexNet.) It would be nice to get a qualitative flavor of how they differ. Do they learn very different convolutional filters at early layers or later layers? Are they really selective for shape versus motion as the authors suggest (the "dorsal" pathway is only a little worse at the CIFAR-10 task.) Anything to better understand how the symmetry is "broken" would be interesting to see here. Is it possible to bias one pathway to "become" the more dorsal one by changing the architecture slightly? 

5. The authors use CPC as a proxy task for learning representations, but also try to argue (correctly, I think) that predicting the future of a visual scene is an ethologically important task. But we don't actually see in the paper whether the models are any good at this prediction problem, or whether the 2-stream architecture is better (gets lower loss) than the 1-stream architecture. If prediction is important and this one task/loss is supposed to explain neural processes across multiple visual areas, I would expect the authors to think that the 2-stream architecture is actually better at prediction than the 1-stream architecture! Otherwise, it doesn't seem like the 2-stream + CPC task provides an explanation of the structure of mouse visual cortex in the way that the authors want it to. So I'm eager to see whether this is the case.

6. Likewise, how does the 1-stream architecture do on transferring to the two tasks in Figure 5? 

[A] Dwivedi, Kshitij, et al. "Unveiling functions of the visual cortex using task-specific deep neural networks." bioRxiv (2021): 2020-11.

[B] Cadena, Santiago A., et al. "How well do deep neural networks trained on object recognition characterize the mouse visual system?." (2019).

[C] Vinken, Kasper, and Hans Op de Beeck. "Using deep neural networks to evaluate object vision tasks in rats." PLoS computational biology 17.3 (2021): e1008714.

[D] Nayebi, Aran, et al. "Unsupervised Models of Mouse Visual Cortex." bioRxiv (2021).
limitations_and_societal_impact: I think what the authors wrote is fine, though to me the "Limitations" section reads more like "Here's what we would have done if we had more time" rather than "Here are the questions that our experiments weren't set up to address and are beyond the scope of the work." I think most of the things they suggest in the Limitations section are good ideas, and don't seem prohibitively expensive to try! I don't think I can reasonably expect them to try all of them in the week-long rebuttal period, but including at least some of them in a revision would make the paper much stronger.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper uses contrastive predictive coding to train deep networks on UCF101 videos. It then uses representational similarity to compare their representations to those of mouse calcium-imaging data from the Allen Brain Observatory. The key result is that a two-stream network is a good match to mouse visual cortex data. One of the (structurally identical) streams develops a representation that is a good match to dorsal mouse areas, and the other matches ventral mouse areas. Deeper areas correspond better to areas higher in the mouse visual hierarchy. Furthermore, the ventral stream of the model better supports transfer to CIFAR-10, while the dorsal stream better supports discrimination of random-dot motion direction. 
main_review: Originality: 
There have been a few papers recently that compared deep-network representations to Allen Brain Observatory data. A recent one (Nayebi, A., Kong, N. C., Zhuang, C., Gardner, J. L., Norcia, A. M., & Yamins, D. L. (2021). Unsupervised Models of Mouse Visual Cortex. bioRxiv.) also uses contrastive objectives and considers networks with multiple streams. This paper and that one agree that contrastive objectives work well for modelling mouse data, but there is surprisingly little overlap otherwise. This paper focuses mainly on the issue of dorsal/ventral stream differentiation with a single cost function, while Nayebi et al. didn’t specifically address that point (although their StreamNets may have exhibited this differentiation). 

Quality: 
The technical approach is sound and the analysis is well-informed and appropriately thorough. In a couple of cases the description of the results seems more categorical than the results themselves. For example, line 223 says “unlike ResNet-1p, which fails to model the dorsal areas”. But ResNet-1p isn’t a total failure at the dorsal stream. It’s max RSM for VISal is even a little higher than that of ResNet-2p. 

Clarity: 
The paper is clear and well organized. The authors shared their code in case any details were missed.  

Significance: 
It is not too surprising that a network with two streams would develop some differentiation (Krizhevsky et al.’s version of AlexNet did this). However, it is significant that an unsupervised two-stream model with no other prodding would develop representations that are convincingly dorsal and ventral. The paper also contributes substantially to the understanding of what may lead to mouse visual representations, specifically.

Minor: 
The paper was very clean. I only have a few suggestions: 
-	Remind the reader in the caption of Figure 3 that red is dorsal. 
-	Switch red and blue in Figure S4, to match the convention in the main paper. 
-	Give parameters of the Gabor control model
-	Use vector graphics for the figures where possible. The file size is 10x that of the other papers I’m reviewing, and it doesn't scroll smoothly. 

limitations_and_societal_impact: The work has a minor societal impact. The impact and limitations were thoughtfully addressed in the last section of the paper.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors address the challenge of training a single deep neural network, using a single loss function, which captures the specialized neural responses in both the ventral (“what”) and dorsal (“where”) pathways of mouse visual cortex. The authors demonstrate that a deep neural network comprised of two parallel convolutional hierarchies, trained with a single self-supervised predictive objective, captures neural responses in the ventral and dorsal streams better than a neural network comprised of a single convolutional hierarchy (which captures only the ventral stream), and better than either architecture trained with a supervised objective. Furthermore, by evaluating their two-pathway model on “what” and “where” downstream tasks —object categorization and motion detection— the authors show that their model has learned functionally specialized representations in each of its two pathways.
main_review: That the architectural choice to split a neural network into two parallel pathways encourages functionally specialized representations, even when both pathways are trained on the same objective, is on its own a remarkable and intriguing observation (although the authors do not appear to be the first to make this observation; see [1] for examples where branched architectures e.g. Alexnet exhibit specialization in their branches e.g. Gabor filters in one branch vs color detectors in another). This observation could inspire future works on the theory of deep learning, help to understand how representations are learned in neural networks, and aid in model interpretability. And it would be a resounding success if a simple anatomical constraint like this could boost the performance on specific tasks or encourage representations which better fit the brain (the authors demonstrate the latter under their representational similarity metric RSM).

However, I have one lingering question after reading the paper: has the 2-pathway architectural choice encouraged *new* functional specialization, or has it simply isolated functional specialization that already existed in the 1-pathway network? I would have liked to see a couple more analyses to investigate this possibility. (1) I would have liked to see the performance of the 1-pathway network on the two downstream tasks. Does the 1-pathway network already contain the specialization in its representations necessary to perform both downstream tasks successfully? Or does the 2-pathway constraint actually encourage new specialization that allows it to perform better on the downstream tasks? (2) I would have liked to see at least one other metric for comparing representations in the trained model to those in the brain. While the fact that the best layer in the 1-pathway network barely outperforms raw pixels at capturing the dorsal stream under the RSM is pretty convincing, naively I can imagine a scenario where the 1-pathway network contains both ventral-like and dorsal-like units, but the variance due to the ventral-like units swamps the variance due to the dorsal-like units, and hence the ventral-like features dominate the RSM as computed by Pearson correlation (whereas the 2-pathway network overcomes this by sorting the units into distinct subpopulations). Perhaps the authors could try another metric --such as regressing the target neural responses with a small number of source units, or under an L1 penalty, or CCA or something similar-- to determine whether dorsal-like features are present in the 1-pathway network (all analyses that have been used to compare models and brains in the past). (3) Finally, is there a clustering that could be performed, either on neural activity or connectivity, by which units in the 1-pathway network could be sorted into two functionally specialized subsets -- one which captures ventral-like features, and one which captures dorsal-like features?

The paper includes several other interesting insights, including recapitulating (to some extent) the anatomical hierarchy of mouse visual cortex, and indicating an apparent tradeoff between ventral-like specialization and dorsal-like specialization. Overall I think this work is a very interesting contribution, the analyses are well-performed, and the manuscript is clearly written.

[1] Voss, et al., "Branch Specialization", Distill, 2021.
limitations_and_societal_impact: The authors more than adequately address the limitations and potential societal impacts of their work, and suggest a few interesting directions for future work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors show that a two-stream 3d ResNet architecture trained using a contrastive predictive coding (CPC) objective develops a representation that resembles the ventral/dorsal stream split in visual cortex. Although I do not buy into all claims of the paper, I am generally very supportive of it. The authors make their main point and the paper is well written. With some adjustments of the claims it will be a great paper.


### Update after discussion period

Given the results from the control experiments reported by the authors, I support publication of the paper and increased my score to 7. I do ask the authors to include those results in the final version of the paper.
main_review: ## Strengths
+ Simple but innovative idea
+ Demonstrates a single network architecture and objective function can generate a ventral/dorsal split as in visual cortex
+ Well written and easy to read

## Weaknesses
- Unclear if claimed superiority of CPC over supervised pre-training holds
- Unclear how robust the emergence of two streams is


## Detailed comments on weaknesses

### 1. Comparison to supervised pre-training

Using the CPC objective for training, the authors demonstrate nicely that the two-stream architecture results in more dorsal-like representations in its second stream than the one-stream architecture. However, I am concerned about their claim that the CPC objective is necessary.

- The supervised baseline might be problematic due to the low resolution of the videos. What can be recognized at 64x64 pixels? What's the performance of the classifier on the task it was trained on? If the task cannot really be solved, I'd be worried that the network simply doesn't learn useful representations in the first place, because the semantic labels are not much related to the image information present at the low resolution. Following the authors' reasoning, supervised pre-training should perform well on the ventral stream areas. However, predictive performance of the supervised model is quite weak for all brain areas. This result seems to suggest that comparison to the supervised model is not very meaningful. Wouldn't it make more sense to compare to a well-trained action classification net that uses higher-resolution movies and take care of upsampling the videos shown to the mouse accordingly?

- (Related to the previous point.) Previous work [20] has demonstrated dorsal-like hierarchical representations in a 3d CNN trained on UCF101 (Tran et al. 2014). The authors cite that paper in their related work section, but do not further comment on it. Given that their claim that supervised training is not sufficient goes directly against published work, I would have expected at least some kind of discussion/explanation, and I think the burden of proof for such a claim is on the authors.

- There is an existing two-stream video recognition architecture (SlowFast networks: https://arxiv.org/abs/1812.03982), which the authors don't seem to be aware of (at least it's not cited in the related work). Given that it's publicly available, trained in a supervised fashion and features two streams, I think it should be included in the comparisons in Figs. 2–4.

In summary, if the authors want to make the point that the CPC objective is necessary and supervised training does not work, they need to provide substantially more evidence for this claim. An alternative solution could be to remove this claim and focus on making the point that CPC + two-stream architecture are a sufficient condition for achieving a ventral/dorsal split, but not claim anything about necessity.


### Robustness of the emergence of two streams

As I understand the paper, the authors trained exactly one instance of the two-stream architecture using CPC and are basing their claims on this single instance. I wonder how robust this result is, though. Do these two streams always emerge or were the authors "lucky"? It would be good to have at least a second, independent verification with a different seed – and, perhaps, slightly different hyperparameters – to verify that the finding is at least somewhat robust.


### Minor comments

- How was the input resolution for the ANNs determined? Depending on the resolution one picks, different layers may be optimal simply because of the receptive field sizes of neurons. It would be good to clarify this point.

- Fig. 2+3: What do the error bars depict?

- Fig. 2–4: Why are the gray points not identical? I thought they depict the RSM similarity of the  input (pixels) with the respective brain area? Those shouldn't depend on the model, should they? The same goes for the untrained model in Fig. 3+4(c): Shouldn't the two also be identical? After all, it's the same architecture.
limitations_and_societal_impact: Yes
review_title: null

Meta-review: This is a clear accept.  All the reviewers were clearly positive, and I largely agree with their judgements.   I think the reviews speak for themselves, and I don't have much to add here. 
Decision: Accept (Spotlight)