Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper evaluated the ability of artificial neural networks with different learning objectives to explain single-neuron responses in the primate dorsal visual pathway. A 3D ResNet trained to predict action motion parameters in simulation videos matched dorsal stream responses better than networks trained for action recognition, suggesting that estimating self-motion may be a main role of neurons in the dorsal stream.
main_review: The authors evaluated a range of neural networks, including one trained to estimate self-motion parameters (DorsalNet), in terms of their match to primate neural data. DorsalNet matched dorsal stream data better than networks trained for action recognition or latent state prediction. The approach of comparing networks with different learning objectives is powerful, and different training sets are also represented across the neural networks tested.

The convergence of results across experiments and stimuli is another strength of the paper. However, the fact that recordings from different brain areas were collected in different datasets is also a challenge when directly comparing results across areas, which could be mentioned more clearly in the paper. This seems particularly important given the small difference between the DorsalNet and MotionNet performance in MST (a single dataset available).

The intermediate representations in layers 1-3 of DorsalNet are interesting in terms of their match to the hypothesized features that drive dorsal stream responses. Figure 2 suggests that the DorsalNet match to dorsal neural data peaks at ~layer 3, and the paper focuses on layers 1-3. I wonder how representations in later layers compare to the intermediate ones, since the decrease in performance in later layers is so small, especially in MST.

Discussion: Although the results do suggest a separation between action recognition and motion estimation, the paragraph in lines 295-305 seems to discuss conclusions that don’t necessarily follow from the data at hand. Since the paper did not look at action recognition with/without motion in the ventral stream, the conclusions here could be rephrased to be more focused.

Finally, it is somewhat unclear in what sense dorsalnet is self-supervised. Although the discussion explains that the self-supervision refers to the goal from the agent’s perspective, the abstract describes the network as having been “trained in a self-supervised manner” – perhaps this could be rephrased to be more specific/accurate. 

However, the paper addresses the question of goals in the dorsal stream in an interesting way and paves the way for more direct confirmatory experiments using DNN-generated stimuli/simulation videos. 

------------------------------------------------
After author response:

I thank the authors for their thorough response. Interesting to see the results using sparse regression, as well as the layer representation results. I look forward to reading the updated paper!


limitations_and_societal_impact: The authors have discussed the main limitations of the work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose to use estimation of self-motion from short video snippets as an objective function to train visual representations resembling those of the dorsal stream of the primate visual system. They show that the first layers of small ResNets trained with this objective develop representations similar to areas V1, MT and MST. According to the authors, alternative objectives such as action classification or contrastive predictive coding or classical baselines like a Gabor filter bank did achieve the same performance on the task of predicting neural responses in the dorsal stream areas – a claim I do not fully agree with; see below. The paper proposes an original idea. It is fairly well executed and well written. I do have some reservations and questions, but I'm overall somewhat supportive of the paper – in particular if the authors can address my concerns.


### Update after discussion period
The authors' response addressed some of my concerns. Unfortunately they promised to report on the control regarding scale but did not post the results, so it's hard to assess this point. I therefore keep my score as borderline accept and hope that the authors will report the results in the final version of the paper.
main_review: ## Strengths
+ Original idea
+ Proposes an image-computable model of a non-trivial task associated with the dorsal stream
+ Model can be used to create in silico predictions that can be tested experimentally
+ Well written and easy to read

## Weaknesses
- Performance difference to the MotionNet baseline is weak at best
- Task seems too simplistic for a model of the dorsal stream
- Relative scale of receptive fields and input stimuli not considered


## Detailed comments on weaknesses

### Performance difference to MotionNet

The claim that inference of self-motion is a crucial task for modeling the dorsal stream doesn't seem to be fully supported by the data. Neither in MST (the most high-level area considered) nor in one of the two MT datasets does the authors' DorsalNet outperform the much simpler MotionNet that is trained on inferring object motion. Don't these results argue against self-motion being important and rather suggest that the dorsal stream tries to resolve ambiguous local motion by inferring which regions belong to the same objects? I'm not super familiar with MotionNet, so please correct me if I'm interpreting those results wrong.


### Task too simplistic?

While estimating an agent's self-motion is certainly one important objective (and likely one of the dorsal stream), calling DorsalNet a model that explains "the properties of dorsal stream neurons" seems like a bit of an overstatement given that two of the three MT/MST datasets were with random dot kinematograms, where motion is the only cue. In the authors' defense, the MT dataset with natural movies is the only one where their model actually shows an advantage, which could be taken to suggest that they have a point. However, on this dataset CPC on the same stimulus performs almost equally well, suggesting that video sequences that feature optic flow related to self-motion might be just as important as the training objective. 

So while I think the authors are on the right track with their approach, the publicly available data does not seem to allow strong conclusions at this point. I would suggest toning down the claims and providing a more nuanced discussion of the possible interpretations rather than claiming strong support of their hypothesis, which I don't see.


### Relative scale of receptive fields and input stimuli

Since the models are image-computable and different datasets from different brain areas and labs are combined, I was a bit surprised to not see any discussion of scale. How did the authors decide how large one pixel in the models should be in terms of degrees of visual angle? Neurons have different receptive field sizes depending on eccentricity and brain area. Depending on which input resolution one picks, the models probably perform better or worse, and the conclusion about which layer (or model) predicts neural activity best can change (see, e.g., the work by Cadena et al., PLoS Comp Bio 2019). Unless the authors considered this issue and I missed it, I think it is important to clearly describe the reasoning here and potentially perform some controls to ensure that the inputs were provided at the optimal resolution or the conclusions don't change as a function of input resolution.



## Minor comments

- I wouldn't call the training paradigm "self-supervised". It's using ground truth information from the rendering engine as labels. Self-supervised techniques usually derive their labels from the images themselves by augmentation, cropping other pretext tasks.

- The discussion talks about the "curriculum," but means the dataset. Curriculum is usually rather used to refer to a sequence of training examples that vary over time w.r.t some feature (e.g. complexity), so I was a bit surprised at first until I realized how it's meant.

- It's not clear what the bold numbers in Table 3 mean. Given the SEM values in parentheses it doesn't seem to signify statistical significance. It would be great to clarify.


limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In this paper, the authors develop a new goal-driven model of the primate dorsal pathway based on a ResNet architecture trained to infer self motion parameters. They show that its units show properties reminiscent of properties of neurons found along the dorsal hierarchy. They benchmark this model extensively against other models including object recognition trained ones.
main_review: In general, this is a well-written paper, with a clear structure and well-done experiments. A few details could be improved (see below), but the main point of criticism one could voice against this paper is that of novelty and advance over previous work. Given the hype about the ventral stream goal driven models, the present paper seems like a relatively minor extensions. However, I think it is still worthwhile doing, and doing it well (as in the present work). The authors argue for the importance of self-motion cues in the dataset and the task being self motion inference, yet the quantitative advantage over MotionNet (their ref 26) seems small (Table 3, last two lines). Maybe the authors could elaborate on why they still think their task and network are superior in modeling dorsal stream neurons than MotionNets (which seems to use simple displaced images as stimuli and seems to be a bit shallower). 

Additional comments:
- Why was a ResNet used as a base model?
- For the alignment between networks and neurons, why was L2 penalized regression used and not L1 regularized? 
- How many video segments were generated?
- Line 96: "the" missing in front of Unreal
- The authors call their approach self-supervised in the abstract. Why? To me this seems just a DNN with known ground truth. 
- The tuning index in line 170 is likely higher than usual due to the lack of noise
- Why do the correlation values in Fig. 2 go beyond 1? Rather use non-parametric CIs. 
- The results in Line 203ff are not obvious from Fig. 2, maybe mark the maximum
- For Table 3, what is the number of experiments and what is the SE over?

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper positions itself in an important gap in the computational neuroscience literature which uses artificial neural networks as models of biological vision. The paper uses first-person perspective video stimuli to train a 3d ResNet model to predict self-motion parameters (namely head rotation and velocity) with contrastive predictive coding. The paper examines layer-by-layer the properties of the learned representations and compares them qualitatively and quantitatively to a database of neural recordings from the dorsal stream of non-human primates, recorded while those primates were viewing videos, or classical psychophysical visual stimuli. The authors find that their model learns representations that are more similar to the primates than baseline models or the same model trained on a different task. Finally the authors find that the networks which are functionally achieve  the best self-motion estimation performance are also those that match biological dorsal stream neural data the best. 
main_review: This paper is an original and complete contribution to the field which helps to address an important gap in models of biological perception. The approach to train on simulated egocentric video footage and estimate motion is novel for computational neuroscience work, and the layer-by-layer connections between the resulting representations in the ANN and primate NNs were well explored.
Finally the discovery that the ANN models that perform best at predicting self-motion also form the representations most closely aligned with those in the primate dorsal stream is an exciting addition to the literature. 

The work was also clearly presented, although as a very minor point more detailed figure legends would be helpful to communicate the main point of each figure more readily to the reader. Note also as an aside a missing 'TODO' link on line 203 which can be easily fixed.
limitations_and_societal_impact: Yes.
review_title: null

Meta-review: This paper received 4 accepts (including a marginal accept). The paper makes an original and complete contribution to the field and addresses an important gap in biological vision. I would add that unlike much of the current work in computational models of vision where the goal appears to simply quantitatively fit model responses to neural data, this paper performs extensive in-sillico electrophysiology on the model responses to highlight qualitative properties using classic stimuli that have been used to characterize the dorsal stream. The AC recommends accepting as a spotlight. 
Decision: Accept (Spotlight)