Venue: NeurIPS 2021 Workshop SVRHM

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: In this paper, the authors show that the multimodal models better explain fMRI responses in the hippocampus. They use a publicly available fMRI dataset and select a subset of voxels in multiple ROIs based on beta values. Responses in the visual, fusiform, hippocampus, and parahippocampus are then compared to visual, language, and multimodal DNN models.  The results show that in fusiform and hippocampus multimodal models explain responses better than visual and language models.

### Quality
The paper is of high quality due to the following reasons: 
1. Clear formulation of the hypothesis and suitable controls within the multimodal models as well as language and visual models 
2. Clever use of already existing publicly available datasets and models to validate the hypothesis
3. Suitable statistical tests
4. Robustness of results with respect to the number of voxels and choice of distance metrics.

While overall the results validate the key hypothesis of the paper, there are several other interesting results that might need more discussion. For example, in figure 2A in hippocampus BIT-M seems to be significantly better than other visual models but it shows almost 0 correlation with visual voxels. Why this would be the case? BiT-M is trained on a larger dataset and has been shown to learn more generalizable representation, so it is a bit unclear why it would show almost zero correlation with the visual voxels.

I would also encourage authors to look at layers other than average pooling for a more thorough comparison across different layers of the models and see where the differences start emerging.

### Clarity
The paper is written clearly and the text is easy to follow.

### Originality
The idea of applying multimodal models to explain responses in the hippocampus is novel and has not been done before. This paper also shows how existing datasets and models can be exploited to test new hypotheses.

### Significance
The overall findings are significant for the community interested in applying the computational models to test new hypotheses about the functions of the human brain. The study clearly demonstrates the potential of this approach and thus can be extended beyond the hippocampus to other brain areas.

review_title: Excellent work demonstrating multimodality in hippocampus activity using computational models

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: # Summary

Here, the authors tackle the question of whether multimodal models can explain brain activity in regions where it is known to be concept cells better than unimodal models (visual models in particular). They use a publicly available fMRI dataset and via representational similarity analysis, they compare the brain activity and models.

## Strengths

- The paper poses a very interesting question on a relevant topic
- The variety of the SOTA models considered. Not only visual models were included in the comparison, but also language models that were not trained in a visual task but were adapted for it.
- The description and discussion regarding the voxels selection and the noise-celling calculation for each particular ROI
- The clarity of the paper in general

## Weaknesses

- Given the variety of models considered it would be good to have more details on how the latent representations for each model are built
- Results of the statistical tests are not informed, only the type of test performed and whether it was significant or not
- Limitations to compare between specific models

Overall, I find this article highly relevant for the SVRHM workshop. It provides valuable insights on the correlates of multimodal models and brain activity, helping to bring machine learning and neuroscience closer together.
review_title: Interesting analysis on the correlates between multimodal models and brain activity

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The authors review several DNN models and compare their internal representations with fMRI activity of different brain areas using  Representational Similarity Analysis. In particular, they are interested in evaluating how well representations in new multi-modal DNNs match up with signals in the Hippocampus as both have been shown to include 'concept' cells; cells that respond  to a concept regardless of input type (images, text, audio etc.) 

Overall the paper is well written. The data, their methodology  and their results are clearly explained. The scope of the work is small but it is well executed.

Comments:

[1] Many models where used in the analysis. Although all are referenced, the authors do not specify where the code for these models is from. Did they implement them themselves or get them from somewhere?  How did they guarantee they work correctly? 

[2] In Figure 2, it is intriguing that multi-modal networks show better alignment with fMRI data from visual areas compared to models trained only on images. Which visual cortices does the fMRI data correspond to? Does it include more than just visual areas? Can the authors expand on this.

[3] Line 58, How do results using the Pearson's correlation coefficient compare to the results obtained by using the other metric? 

[4] Lines 67-76, there are a lot of abbreviations that are not explained and are hard to follow.

[5] The example on line 35 needs a reference.



review_title: Clear, precise and well written paper. Use Representational Similarity Analysis to show that multi-modal DNNs learn representations that are more similar to representations in the Hippocampus compared to uni-modal DNNs.

Decision: Accept (Oral)