Venue: NeurIPS 2022 Workshop SVRHM

Reviews:

final_score: rating: 7: Good paper, accept
review_content: In this paper, the authors suggest that in ANNs trained with self-supervised learning and with topographic constraints, specialized modules and pathways similar to those in the visual system emerge. It is an intriguing finding and is consistent with a few other previous studies. The paper is very well written and I enjoyed reading it. I have a few comments that I explain below:

1- It is not intuitive how SimCLR can learn dorsal-like representations. Based on our current understanding of the ventral and the dorsal pathways (e.g. classic studies by Goodale and Milner), the main difference between the two pathways can be understood in terms of their representational invariances. For example, the ventral representations need to be viewpoint invariant (for e.g. object categorization), while the dorsal representations actually should be viewpoint dependent to properly guide actions, such as grasping. The augmentations used in SimCLR encourage learning viewpoint invariant representations; hence the good performance on object categorization. Because of these augmentations, it's hard to understand how dorsal-like representations could emerge in a model trained with SimCLR. Did the implemented SimCLR use the same augmentations in the original SimCLR paper? I am interested to hear the authors' explanation. 

2- No measure of alignment between the ANNs and the brain, besides the topographic alignment, has been reported in the paper (a regression-based alignment for example). It'd be nice to see comparisons with an untrained ANN, an ANN only trained with the spatial correlation loss (very large $\alpha$), and also between the supervised and self-supervised ANNs. Is there any significant difference between the alignment scores of the three specialized streams? 

3- The way the $\alpha$ value should be chosen isn't clear. In some of the plots, it seems that increasing $\alpha$ leads to an increase in the number of clusters, but the "correct" number of clusters is, in general, not known a priori. Why did the authors choose $\alpha=0.25$? Why the same $\alpha$ value was used for comparing the supervised and self-supervised models? One could imagine that a larger or smaller alpha value could potentially lead to better clustering with the supervised model. Also, unlike the ventral and lateral clusters, the parietal cluster isn't as clearly formed, which is also consistent with my first comment re the dorsal pathway.


review_title: Interesting findings

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper compares the representations learned by topographic DNNs to high-level visual cortex, which here is split into three ROIs: ventral stream, lateral stream, dorsal stream. They find that ANNs trained with self-supervision self-organize in a topographic layer with different regions corresponding to each stream.

Overall, I like this paper, I think the methods are innovative (clever use of a smoothed Munkres algorithm for matching voxel to topographic network), and it’s an important subject, so I think it should be accepted. The criticisms which follow do not need to addressed for this conference, but if this turns into a full-fledged paper I would like to see this addressed:

1. The three-stream classification is a fairly new idea that hasn’t been adopted in the non-human primate and mouse literature and is still fairly controversial in the human literature. I think the authors will need, in a future paper, to figure out what is it about the representations that make them more lateral- or ventral- or dorsal-like, and confirm or infirm what is believed to be in each representation. They’ve tried without much success to do this with different tasks, and broadly speaking that didn’t work. They might want to try visualization of the high-level units, e.g. in the style of Olah et al.; these can be excellent diagnostic tools. They may also try to use probe stimuli, either from a bank of images or generated with DALL-E, Imagen, Stable Diffusion, etc.
2. As the authors point out in Appendix A.8, the stimuli are static images. This may be ok for ventral stream investigations, but it doesn’t make sense for dorsal and lateral stream investigations. The authors may be able to use existing datasets, such as the studyforrest dataset, which likely contains a sufficient number of objects, motion, and social relationships to map out these three streams.
review_title: Review of SVRHM paper on topographic DNNs

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: This paper proposes a model to explain how the organization of the visual system into three streams may emerge. Using self-supervised learning and a constraint encouraging local response correlations in a CNN and comparing its activities with the large scale NSD neuroimaging dataset, the authors present a model where different regions of a layer better match the activity of different visual streams. The paper is well-written, the analyses seem rigorous and the topic is timely. I recommend this contribution for SVRHM and provide some comments below in the spirit of helping improve the paper.


Comments:
1. In principle, the spatial loss is used to enforce similar activity patterns in neighbouring units. However, how it is computed is not clear to me. As far as I can tell, the authors correlate two vectors of size N_unit_pairs for each layer: C, which contains the correlations in activity patterns for each pair, and D, which contains the spatial distances between each pair. The spatial loss is low if these vectors correlate. This would mean that the loss would be low if large distance was correlated to similar activity patterns? This seems like the opposite of the intended purpose. I take it I misunderstood, but maybe the authors can clarify? In addition, it was not clear to me on which input data the loss is computed. Is it for each batch?

2. Typically, the ventral stream is modelled hierarchically, with different layers modelling different brain areas. Here, we only see the results for the last layer, mapped to large parts of visual cortex, which is quite different in spirit since there is no hierarchy, and it is not obvious how we should interpret what th earlier layers are doing. It would be interesting to see if different layers better explain different regions in terms of topography and activities, etc. It would also be interesting to see if removing the spatial loss decreases the match of the last layer with the brain's stream topography.

3.It would be interesting to report the performance of networks on their main objective for different values of alpha. Does alpha hurt performance? Or can it in fact help?

review_title: A good paper on the emergence of visual streams

final_score: rating: 7: Good paper, accept
review_content: The authors used DCNN with topographic restriction during self-supervised training to build neural network model study the correspondence between artificial network unit and brain voxels. Interestingly, the authors show topographic DCNN also show different visual streams similar to human brain’s. The is a great showcase on how to use neural network models to shed light on how the brain develop to show such functional organizations by anatomical contraints.

Pros: 
1.	Instead of linear regression, this paper shows a voxel to unit mapping, which is a novel method
2.	Using self-supervised learning provides a better insight on how different visual streams can be naturally emerged through contrastive learning
3.	This method provides a lot of opportunities to further model and understand the functional organization of the brain

Suggestions:
1.	The paper does not provide any performance metric of the model in performing any vision task (instead of the match between brains). I am wondering if the performance differs a lot with different spatial loss weighting
2.	In the method, two different tasks were mentioned to train the topographic DCNN. However, in the result session, the patterns between self-supervised and supervised task are not the same. However, the discussion between the results is not thorough. Readers would love to understand more about how the different training tasks would lead to different results.
3.	Typo in Figure 1 legend: missing “:” in the second last line.

review_title: A good use of topographic DCNN model to study functional organization of the brain

Decision: Accept(Oral)