Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposed and encoding model using large fMRI datasets to predict the brain response to visual images in different regions of the ventral stream of the visual cortex. They test the model with different paradigms such as: generalization to out-of-distribution images, generalization to other subjects, spatial tuning in the early visual areas, separability of objects in higher visual areas, alignment with human perception, and shape bias. They then visualized the preferred images by different voxels predicted by the model. The main novelty here are the shape bias and the ability of the model to achieve these above things concurrently. Each one of these was observed already in previous models but not together. The paper is quite dense and the authors attempted their best to fit within the page limit.
strengths_and_weaknesses: Strengths:
- The paper succeeds in having a strong prediction accuracy for OOD images. While, many other models have managed to do that, it is not an easy task.
- Shape bias is one novelty over the traditional DNN models for CV.
- The model shows multiple similarities with the ventral stream properties instead of focusing on one or two.

Weaknesses:
- The paper is quite dense leading the authors to omit many necessary information and to have figures too small and cluttered.
- Literature review in this field is lacking many important relevant references that discuss the idea of the similarity of hierarchy between humans and DNNs. To name a few:
    - Horikawa, Tomoyasu, and Yukiyasu Kamitani. "Generic decoding of seen and imagined objects using hierarchical visual features." Nature communications 8.1 (2017): 1-15.
    - Abdelhack, Mohamed, and Yukiyasu Kamitani. "Sharpening of hierarchical visual feature representations of blurred images." Eneuro 5.3 (2018).
   - Tsunoda, Kazushige, et al. "Complex objects are represented in macaque inferotemporal cortex by the combination of feature columns." Nature neuroscience 4.8 (2001): 832-838.
   - Schrimpf, Martin, et al. "Brain-score: Which artificial neural network for object recognition is most brain-like?." BioRxiv (2020): 407007.
   - Schrimpf, Martin, et al. "Integrative benchmarking to advance neurally mechanistic models of human intelligence." Neuron 108.3 (2020): 413-423.
   - Federer, Callie, et al. "Improved object recognition using neural networks trained to mimic the brain’s statistical properties." Neural Networks 131 (2020): 103-114.


- The paper tests many different conditions with their model leading the paper to be increasingly complex since each condition is done via a separate analysis. One suggestion is to have a summary table of test and how it is measured and baselines so one can understand the big picture.

- Each of the baseline model is trained very differently which makes them not really good baselines

- Viewing figure 6A does not support the statement in the text that preferred stimuli increase in complexity
questions: - Do you have any explanation for the aggregated distribution of points in figure 4A which seems to be relatively ordinal in comparison to the predicted?


limitations: - Encoding models are inherently limited by their dependence on the images solely unless other mental or prior information are added
- Testing on a new dataset limits their ability to test the model against other models that focus on classification.
- The appeal of encoding models is to understand how the model is able to achieve these feats that are similar to human visual cortex which is not addressed here.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work builds end-to-end voxel-level encoding models for fMRI recordings of a popular publicly available dataset of natural scene images (NSD). The authors perform extensive comparisons against networks optimized for image classification on different types of data, and show that their proposed response-optimized networks are better able to generalize to heldout subjects, are more sample-efficient, and better generalize to out of distribution images. The work also offers several interpretability analysis of the representations learned by the response-optimized networks and shows that they appear to align with human similarity judgments and have more shape bias than a task-optimized network.
strengths_and_weaknesses: Strengths:

This work is very rich. It has a good technical contribution and the scientific interpretation is top-notch. I found the following highlights about the proposed method especially important:
- Good generalization to held-out subjects 
- Good generalization to out of distribution images
- Increased shape-bias
- An improved sample efficiency

Weakness:

- No discussion of related work, this works needs to be addressed in order for this work to be published. Works that specifically integrate brain recordings during the training of models are important to mention and discuss with respect to the current work. Here are a few that I'm familiar with, and some of them even use the same image dataset:
    - Seeliger et al. 2021, PLoS Comp Bio https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1008558
    - St-Yves et al. 2022, bioRxiv https://www.biorxiv.org/content/10.1101/2022.01.21.477293v1
    - Khosla and Wehbe, 2022 bioRXiv https://www.biorxiv.org/content/10.1101/2022.03.16.484578v1
    - In language: Schwartz et al. 2019 NeurIPS: https://proceedings.neurips.cc/paper/2019/hash/2b8501af7b64d1aaae7dd832805f0709-Abstract.html

- Using an AlexNet that is trained on ImageNet as that task-optimized model baseline doesn’t seem like the best choice. The NDS images are from the Coco dataset which has a different distribution from ImageNet, and therefore the test NSD images are much more out of domain for the task-optimized model than they are for the response-optimized model. A more fair comparison as a task-optimized model is to either train the same CNN architecture (minus the linear layer on top) to classify the NDS images, or at least fine-tune AlexNet on the training NDS images. Similarly, in the generalization to ImageNet experiments, the authors compare to a baseline model that has been randomly initialized. A stronger (and fairer) baseline here would be a task-optimized model that has the same architecture as the response-optimized and was trained with the same NSD images, but with a different objective (image classification, rather than brain prediction).

- The organization can be improved. Currently the Materials and Methods section is very long (ends on page 5) and details all experiments. By the time that the corresponding results come around, the reader has forgotten the necessary details from the experiment. I suggest describing the materials and main models in this section, and then moving the description of each analysis to the corresponding place in the results. 

questions: Major:
1. I would like to hear the author’s response to the point I raised under Weaknesses about a fairer comparison to a task-optimized model that is actually trained on the NSD training images.
2. Please discuss previous related work that has incorporated brain responses during training of models (see under Weaknesses for a few that I am aware of)
3. The paper contains a mix of encoding model results and RSA. Can the authors comment on why this was done, as opposed to only using encoding/decoding analyses?
4. Can the authors clarify how many images were used for train, validation, and test? L112 says that 1k were used for test and 37k for train and validation, but there are 37k images in total.

Minor:
- Related work to the finding that predictive models can denoise brain data: Ravishankar et al. 2021, Frontiers Comp Neuro showed that predictive models can help with improving both encoding and decoding performance (for single-trial MEG) https://www.frontiersin.org/articles/10.3389/fncom.2021.737324/full
- Please include a 1 sentence description in the main paper of the type of noise ceiling computation that was done.
- Lots of details relegated to supplementary with the text (see Suppl. Methods). The supplementary is long so please include specific pointers to where in the supplementary this information can be found.
- Calling the object category vector “semantic” is strange because all of the model representations contain semantic information. I would change the name to something more descriptive. Also another term to describe this “binary vector” is a k-hot encoding.
- Clarify whether the subjects were in a common space (e.g. MNI).
- The supplementary can benefit from proofreading (I spotted several typos from a quick look)

Suggestions for future/follow-up work:
- The PC interpretation in Fig 5D is nice, but it would be good to do something a bit more principled. If you define a set of semantic norms for each THINGS image (those can be sampled from already available datasets), you may be able to actually show what semantic dimensions are highly associated with each ROI.

limitations: This work directly integrates brain recording measurements into the training of deep learning models. It should discuss any possible undesirable biases that can arise from this (for instance, any additional societal bias).
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Here, the authors leverage the Natural Scenes Dataset to train up a “hypothesis-agnostic” encoding model. In contrast to recent task-optimized models, they train a series of CNNs (one per visual area) and linear readouts (one per voxel) to predict neural responses to a large set of images. Then, generally speaking, the authors test two things about these models: (1) whether they generalize in a non-trivial way to other subjects/datasets and (2) whether the model learns interesting features in the data that may not be present in task-optimized models.
strengths_and_weaknesses: Strengths

1)	The writing is clear and concise in most places.  

2)	The approach to the problem – using an encoding model with a shared convolutional core – is relatively novel.

3)	Some of the results impressive, especially when the authors focus on generalization and sample efficiency.

Weaknesses

1)	It is difficult to contextualize the encoding model, especially the results of Figure 1, without a more systematic set of baseline models. While it is not necessary to comprehensively survey the field a la CORnet (Kubulius et al. 2019), it is essential to know if the gains the authors’ see are unique to their proposed encoding model. Here, I would include top performing models for ImageNet (e.g. DenseNet and ResNet) along with the older AlexNet that they have already included.

2)	As mentioned above, the authors argue that their encoding model (1) generalizes well and (2) learns non-trivial features and response properties. They present compelling evidence for (1), but (2) is less clear. For example, Figure 4 presents measured/predicted retinotopic parameters, but without the task-optimized network for comparison. Are these features unique to their encoding model? Even if the encoding model is a better match, it is explicitly optimized to better match neural data, so it hard to know whether this is truly surprising or not.

    1.	Same point holds for Figure 5 – I would expect this result in the task-optimized model as well. 

    1.	More broadly, I think the point about generalization is interesting – maybe their encoding model does learn a set of features that are generically useful in a visual system. However, the fact that the model learns certain features is potentially circular since the model is optimized to mirror neural responses. In other words, it would be more surprising for a task-optimized network to have neural-like responses, while here the model is explicitly optimized to have neural-like responses.

3)	Related to the previous point, I’m a little confused about the shape bias analysis. Presumably the neural responses that the model is fit to have a shape bias – though I could not find this analysis – in which case the model should inherit a shape bias via the fitting procedure. Perhaps I am misunderstanding this, but it seems that the results in Figure 6 are potentially circular. 


Minor

1)	Why is the prediction performance pooled across visual areas higher than the prediction performance for each visual area on its own?

2)	Include units in all axes for convenience (e.g. RDM similarity is Kendall’s Tau, I would include in the y axis where relevant, e.g. Supp. Fig. 3A).

3)	Should the task-optimized network  also be in Figure 6B middle – Silhouettes transfer performance?

questions: 1. Why was only AlexNet used as a non-trivial baseline model, since there are other CNNs already known to be better fits to neural data?
1. Why were pRFs not estimated for the task-optimized model?
1. Is it wrong to expect the task-optimized model to look very similar to what is shown in Figure 5? Should this be unique to the proposed encoding model?
1. Hopefully I didn't miss this, but did the authors measure shape bias in the neural data used to fit their encoding model? Presumably this bias is simply inherited by the encoding model? 

limitations: One major limitation that I found missing in the discussion is that many of the important features found in their encoding model – e.g. correspondence to perceived image similarity – are also found in the neural data, which the models are fit to. The discussion would benefit from clarifying what unique contribution their encoding model brings to the table.
review_title: null

Meta-review: metareview: The authors leverage a large fMRI dataset to fit  “hypothesis-agnostic”/data-driven encoding models to brain data (as opposed to task-optimized ones -- as done in most of the related work). While there is general agreement that this is not a groundbreaking advance (there is already similar work published), the reviewers agreed that the paper was technically strong (good generalization to held-out subjects, good generalization to OOD images,  increased shape-bias, improved sample efficiency). The interpretation of results for neuroscience was rated as "top-notch".  The paper received a rare unanimous clear accept. The AC recommend acceptance.
confidence: Certain
Decision: Accept