Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper uses self-supervised (and supervised) models trained with speech or auditory scene data to predict fMRI activity in the auditory cortices of English-, French-, and Mandarin-speaking participants. It shows that this can be done at a level higher than some baseline models (e.g. random untrained models) and the models recapitulate some aspects of the neural organization such as hierarchical processing. 
strengths_and_weaknesses: Strengths:

The experiments and analyses were done competently and carefully with a large number of participants and the results seem to be sound.

Weaknesses: please see the questions and limitations sections.
questions: The authors claim 600 hours of data is comparable to the amount of auditory experience humans receive during their development. But 600 hours is only 25 days, which seems short compared to the developmentally relevant time scales for humans, so could you please elaborate on this comparison? The Discussion section mentions some tasks that can seemingly be done by humans with a few hundred hours of speech, but those tasks are very low level tasks which do not represent all facets of human speech perception. Relatedly, what would the performance (e.g. neural predictivity etc.) look like if you used pretrained self-supervised models trained on much more data, say, even more data than in a human lifetime (thus not necessarily restricting yourself to 600 hours).
limitations: I do have to note the correlations shown in Fig. 2B-C are very low (R). This may reflect the intrinsic SNR limitations of the fMRI signal. This makes me wonder if Fig. 1C is cherry-picked. A more representative example would be much better here (with an R similar to the average R). Relatedly, what is the noise ceiling performance in these recordings? That is, if I tried to predict the fMRI of one participant from the fMRIs of other participants, what would the performance look like?

Re supervised vs self-supervised: I do have to note phoneme recognition is a very low-level task, perhaps a higher level classification task would give better results for the supervised model (which is already pretty close to the self-supervised one, to begin with).

ABX accuracy of the models is much higher than human accuracy, which suggests these models might not be behaviorally adequate models of human speech processing (and possibly of human auditory processing more generally).

I understand that the differences in Fig 4C are significant, but they are tiny differences. Shouldn’t I interpret this as saying the non-native model, for example, is actually a pretty good model of the native auditory cortex? Whatever makes the auditory cortices of, say, Mandarin speakers different from the auditory cortices of French speakers doesn’t seem to be captured by these models (or perhaps more likely by fMRI itself).

Finally, I also do have to note most of the results here (hierarchy, very slightly better prediction of native vs. non-native fMRI responses, etc.) are not really too surprising given what we know from earlier literature and prior expectations.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In this work, the authors build speech encoding models using wav2vec 2.0 and experiment with 4 different regimens:
1. Untrained network
2. Network trained on non-speech sounds
3. Network trained on non-native speech
4. Network trained on native speech

Their main findings are:
1. Representations from wav2vec 2.0 predict brain activity well in several regions. Although significantly different, the untrained networks have comparable effect sizes to the trained models.
2. Diff regions across cortex are best predicted by diff layers of the model such that low-level regions like the auditory cortex are best predicted by the lower layers.
3. Much like humans, the models do better at a phoneme discrimination task on the native language than non-native. The non-speech networks do worse, followed by the untrained networks.
strengths_and_weaknesses: Edit after rebuttal: I comment the authors efforts to do a new analyses and the thorough responses to the review. Consequently, I have increased my score by 1. However, at this point I cannot recommend the paper for acceptance as some things stand out to me (more detail in follow-up comments):
- There is no established link between the different types of models tested, the layer-wise selectivity and the probing result. To this end, it is hard to interpret what the posited function of each region is or,
- what new things did we learn about the brain/deep speech networks that wasn't know previously? The paper seems largely confirmatory and the methods employed are not novel.
- some of the prediction correlations are surprisingly small even in areas robustly engaged during speech perception. Furthermore, some of the claims on selectivity for native vs. non-native vs. non-speech info is based on extremely small, possibly insignificant differences. This make interpretation hard.

###################################################################################################

Strengths:
1. Clarity: The paper is well written and easy to read. 

Weaknesses:
1. Significance/Originality: The central premise of this paper lies in exploring self-supervised architectures for building encoding models. The results presented include 1) encoding model performance maps 2) layer wise selectivity maps 3) encoding performance viewed through a cascade of model selectivity (native > non-native....> chance). However, due to several reasons highlighted in detail below (like no discussion/analysis on what information is represented at different layers and how does this explain patterns of variation in layer selectivity, for example) and due to the identical results+analyses in prior work on supervised models and concurrent work on self-supervised models, the novelty of this work is not clear. 
- Kell et al., Neuron 2018 -  exploring layer selectivity for diff supervised task architectures + specificity for speech vs. non-speech
- Millet et al., arXiv 2021 - exploring many of the analyses presented here but with supervised models
- Vaidya et al., ICML 2022 - exploring 4 self-supervised models, layer selectivity and probing to infer information at each layer
2. Under-explored/Limited analyses: Although the paper presents encoding model performance in every major analysis, the trends observed are not discussed further. For example, what information do different layers capture that could drive selectivity for lower layers in low-level brain regions vs. selectivity for deeper layers in high-level regions? Does this information emerge as a consequence of training? Does it emerge only when models are trained on speech? How about specificity of information for the native language? Without digging deeper into these questions, I am not sure of the significance of the results or how they inform our understanding of both speech models and the brain.
3. Overstating claims:
- Claims of supervision or lack thereof that match human ability:
    - Firstly, while the effect sizes are significantly different, these differences are still considerably small between the random/non-speech/non-native/native models.
    -  It is interesting to compare supervised vs. unsupervised models in terms of the information they capture and consequently, their ability to predict brain data. However:
        - given the implausibility of the learning mechanisms employed in these networks (as noted in the paper too; Lines 301-312)
        - the fact that they do not follow the same patterns of errors/do not mimic human behavior
        - the fact that the paper currently does not explore the types of information captured (what makes one model perform better than the other? how is this related to the purported functions of different regions?)

        , why is this a realistic model of speech perception? I agree that the model uses far less training data than LMs (which, contrary to how it is stated in the paper, are self-supervised too) or supervised speech models. However, without a serious/exhaustive analysis on the types of speech tasks these models do well on, there ability to mimic human speech errors etc., is it fair to state that we are moving towards brain-like models? (more detailed arguments in Guest & Martin, 2021) For example, lines 259-261: “Here, we test whether self-supervised learning on a limited amount of speech suffices to **yield a model functionally equivalent to speech perception in the human brain.** “ 

- in lines 290-300, the authors discuss several studies on the functional specificity of different brain regions during speech processing, functional gradients in PFC etc.. However, how is this being inferred form the data? I may have missed the link but iiuc this is not supported by the evidence presented. 
- "the functional hierarchy of its transformer layers aligns with the cortical hierarchy of speech in the brain, and **reveals the whole-brain organisation of speech processing with an unprecedented clarity**" - Why is the layer wise selectivity important? Not to belabour the emphasis on what is captured by these representations, but without a discussion/analysis to reason why these differences might arise across regions, I am not sure what to make of these results or how they clarify the organization of speech processing across cortex. Furthermore, it appears that the upper-middle layers have high correlations in several regions across cortex, albeit not the highest. What do the authors make of this?
- Please cite related work appropriately. The authors have missed several relevant citations from PIs like Jack Gallant, Leila Wehbe, Alex Huth, Tom Mitchell, Christopher Honey, Jonathan Brennan, Alona Fyshe and more. For example, papers that were the first to build language encoding models or present the style of analyses reported here are missing.
questions: 1. Selectivity for deeper layers in primary AC: In figure 3, why is primary auditory (or what is possibly Heschl’s gyrus) marked in red, ie., selective for deep layers? It would be useful to visualize the correlations for different layers in a few voxels to see if the curve is relatively flat and if this is an artifact of using a circular mean.
2. What do the individual participant maps look like for figure 3? Is the group-averaging introducing smoothness in the gradients that is not visible in individual maps ( due to the circular mean estimate being noisy)?
3. How were models trained for the ABX task? What are the specifications of the dataset and the task? Which layers were used to do this?
4. ABX task results: I imagine that the authors ran clustering for the English/French/non-speech data independently to get different quantization targets for each. Given that these targets roughly correspond to phonemes, is it surprising that the English model was better at distinguishing English phonemes than French? Similarly, is it surprising that English models outperform non-speech or untrained models that don’t have English quantization targets or have not learned how to use them respectively?
5. In Fig. 4D, it is surprising that only auditory cortex seems to care about language/speech specific features although we know that a large part of cortex processes natural speech. Can the authors comment on why this might be and perhaps visualize the individual correlations for all 5 conditions in a few example voxels per color?
6. What is the amount of data per participant in each language & stimulus set? Iiic, the Enligsh datasets have far more data per participant than French. How does the paper circumvent differences in the quality of model fits due to different amount of data being used? (especially for cross-language comparisons)
limitations: -NA-
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper provides a study to investigate how Wav2Vec 2.0, as a self-supervised method, learns brain like representation. Different sets of experiments were conducted to examine the cortical hierarchy of speech processing aligns its function. It also provides observation on brain's reaction to non-speech, non-native and native sounds compared to that of Wav2vec 2.0.
strengths_and_weaknesses: Strengths: 
Interesting observations and an extensive set of results
Weakness:
No machine learning innovation 
questions: - I believe the results are interesting to be shared, but I do not see any machine learning innovation in this paper. Basically the authors have used an already proposed self-supervised method to compare with the brain action. I believe this could be a better fit for some speech related journals. 
- R is some times used as Neural predictivity score and sometimes as Pearson correlation coefficient. Please make it clear through the paper.
- R is some times very small, e.g. Figure 4 B. is small R still something to rely on and make conclusion for?
- There are some related work, e.g. https://www.science.org/doi/abs/10.1126/science.1245994, that could be mentioned in the paper.
- Although the results are promising, please make the conclusion linked to the current set up and do not strongly generalize it to all cases.
limitations: no limitation
review_title: null

Meta-review: metareview: This paper compares learned self-supervised speech representations to brain fMRI representations for more than 400 subjects speaking English, French, and Mandarin. Through the rebuttal period, the authors and reviewers interacted extensively to discuss the contribution, results, and analysis provided in the paper. Most of the reviewers' concerns have been addressed by improvements to the analysis and presentation of the paper. 
One main concern was a concurrent research work that appeared on arxiv about one week after the NeurIPS submission deadline. The novelty of this paper should not be impacted by that other work, given the timing of both papers. 
confidence: Certain
Decision: Accept