Toward a realistic model of speech processing in the brain with self-supervised learning
Authors: Juliette MILLET
Charlotte Caucheteux
Pierre Orhan
Yves Boubenec
Alexandre Gramfort
Ewan Dunbar
Christophe Pallier
Jean-Remi King
Venue: NeurIPS 2022 Accept (2022)
Keywords: Neuroscience
Deep Learning
Speech Processing
Self-supervised learning
fMRI

Abstract: Several deep neural networks have recently been shown to generate activations similar to those of the brain in response to the same input. These algorithms, however, remain largely implausible: they require (1) extraordinarily large amounts of data, (2) unobtainable supervised labels, (3) textual rather than raw sensory input, and / or (4) implausibly large memory (e.g. thousands of contextual words). These elements highlight the need to identify algorithms that, under these limitations, would suffice to account for both behavioral and brain responses. Focusing on speech processing, we here hypothesize that self-supervised algorithms trained on the raw waveform constitute a promising candidate. Specifically, we compare a recent self-supervised model, wav2vec 2.0, to the brain activity of 412 English, French, and Mandarin individuals recorded with functional Magnetic Resonance Imaging (fMRI), while they listened to approximately one hour of audio books. First, we show that this algorithm learns brain-like representations with as little as 600 hours of unlabelled speech -- a quantity comparable to what infants can be exposed to during language acquisition. Second, its functional hierarchy aligns with the cortical hierarchy of speech processing. Third, different training regimes reveal a functional specialization akin to the cortex: wav2vec 2.0 learns sound-generic, speech-specific and language-specific representations similar to those of the prefrontal and temporal cortices. Fourth, we confirm the similarity of this specialization with the behavior of 386 additional participants. These elements, resulting from the largest neuroimaging benchmark to date, show how self-supervised learning can account for a rich organization of speech processing in the brain, and thus delineate a path to identify the laws of language acquisition which shape the human brain.

Toward a realistic model of speech processing in the brain with self-supervised learning
Juliette Millet" 1,2.3 Alexandre Gramfort
Charlotte Caucheteux" 1,4 Pierre Orhan?
Yves Boubenec"
Ewan Dunbar
Christophe Pallier
Jean-Rémi King 1.2
"These authors contributed equally
Mett AI, Paris, France
Ecole Normale Supéricure, PSL University, Paris, France LPI Université de Paris cité, Paris, France
Université Paris-Saclay, Inria, CEA, Palaiseau, France
University of Toronto, Toronto, Canada
Cognitive Neuroimaging Unit, INSERM, Gif-sur- France
{jumi, ccacheeeu,,, janremi (anremi)Cmeta, Cmeta. com
Abstract
Several deep neural networks have recently been shown generate activations similar to those of the brain in response to the same input. These algorithms, how- ever, remain largely implausible: they require (1) extraordinarily large amounts data, (2) unobtainable supervised labels, (3) textual rather than raw sensory in- put, and / (4) implausibly large memory (e.g. thousands of contextual words). These elements highlight the need to identify algorithms that, under these limita- tions, would suffice account for both behavioral and brain responses. Focusing speech processing, we here hypothesize that self-supervised algorithms trained the raw waveform constitute promising candidate. Specifically, we compare a recent self-supervised model, wav2vec 2.0, to the brain activity of 412 English, French, and Mandarin individuals recorded with functional Magnetic Resonance Imaging (fMRI), while they listened to approximately one hour of audio books. First, we show that this algorithm learns brain like representations with as lit- tle as 600 hours of unlabelled speech a quantity comparable to what infants can be exposed to during language acquisition. Second, its functional hierarchy aligns with the cortical hierarchy of speech processing. Third, different training regimes reveal functional specialization akin to the cortex: wav2vee 2.0 learns sound-generie, speech-specific and language-specift, representations similar those of the prefrontal and temporal cortices. Fourth, we confirm the similarity of this specialization with the behavior of 386 additional participants. These el- ements, resulting from the largest neuroimaging benchmark to date, show how self- supervieed learning can account for rich organization of speech processing in the brain, and thus delineate a path to identify the laws of language acquisition which shape the human brain.
Introduction
The performance of deep neural networks has taken off over the past decade. Algorithms trained on object classification, text translation, and speech recognition are starting to reach human-level performance [Xu et al. 2020). Furthermore, the representations generated by these algorithms have
repeatedly been shown correlate with those of the brain [Kriegekkotte] 2015 Yamins and DiCarlo,
2016 Kietzmann et al. 2018 Kell and McDermoti, 2019 Cichy and Kaiser 2019
Toneva and
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Wehbe, 2019, 2019, Millet and King 2021 Caucheteux and King 2022 suggesting that these algorithms
converge to brain-like like computations.
Such convergence, however, should not obscure the major differences that remain between these deep learning models and the brain. In particular, the above comparisons derive from models trained with (1) extraordinarily large amounts of data (40GB for GPT-2 [Radford et al. 2019). the equivalent of multiple lifetimes of reading),
(2) supervised labels, which is rarely the case for humans (e.g. (Yamins and DiCarlo 2016), (3) data in a textual rather than raw sensory format, and/or (4) considerable memory (e.g.. language models typically have parallel access thousands of context words to process text). These differences highlight the pressing necessity to identify architectures and learning objectives which, subject these four constraints, would be sufficient account for both behavior and brain responses.
Here, we hypothesize that the latest self-supervised architectures trained on raw sensory data consti- tute promising candidates [Borgholt et al.) 2022 Bardes et al. 20221] Baevski et al. 2020). We focus on wav2vec 2.0 [Baevski al., 2020]. an architecture that stacks convolutional and transformer lay- ers to predict a quantization of the latent representations of speech waveforms. We train wav2vec 2.0 on 600 h of effective speech a quantity roughly comparable what infants are exposed to dur- ing early language acquisition (speech only makes up small fraction of infants' daily experience) [Dupoux
2018
Hart and Risley
1992 Gilkerson et al., 2017].
We use standard encoding analyses Naselaris et al., 2011 Huth et al. 2016 Yamins and DiCarlo 2016 Kell et al. 2018] (Figure I to compare this model to the brains of 412 healthy volunteers (351 English speakers, 28 French speakers, and 33 Mandarin speakers) recorded with functional magnetic resonance imaging (fMRI) while they passively listened to approximately one hour of audio books
in their native language [Nastase et al. 2020 Li et al. 2021 (8.5 hours of distinct audio materials
in total).
To better understand the similarities between wav2vec 2.0 and the brain, we compare brain activity to each layer of this model, well to several variants, namely (1) random (untrained) wav2vec 2.0 model, (2) a model trained on 600 h of non-speech sounds, (3) a model trained on 600 h of non- native speech (for example, model trained on English speech and mapped onto the brain responses to French-speaking participants). (4) a model trained on 600 h of native speech (for example, a model trained on English speech and mapped onto the brain responses to English participants), and (5) model trained directly on speech-to-text (i.e., a supervised learning scheme) on the native language of the participants.
Our results provide four main contributions. First, self-supervised learning leads wav2vee 2.0 to learn latent representations of the speech waveform similar to those of the human brain. Second, the functional hierarchy of its transformer layers aligns with the cortical hierarchy of speech in the brain, and reveals the whole-brain organisation of speech processing with unprecedented clarity. Third, the auditory-, speech., and language-specific representations learned by the model converge to those of the human brain. Fourth, behavioral comparisons to 386 supplementary participants' results on speech sound discrimination task confirm this common language specialization.
2 Methods
2.1 Models
We train several variants of wav2vec 2.0 datasets using two different learning objectives (a self-supervised and supervised objective).
[Baevski et al. 2020 from scratch on different speech
2.1.1 Architecture
Wav2vec 2.0 consists of three main modules. First, feature encoder composed of seven blocks of temporal convolutions (output dimension 512) transforms the speech input S (raw mono waveform at 16 k6z) into a latent representation (output dimension of 512, frequency 49 Hz, stride of 20 ms between each frame, receptive field of 25 ms). Second, quantization module discretizes into g, a dictionary of discrete and latent representations of sounds. Third, = is input to a "context network" consisting of 12 transformer blocks (model dimension 768, inner dimension 3072, and attention heads), which together yield contextualized embedding c, of the same dimensionality of g.

---

Wav2Vec 2.0
fMRI
Transformer,
n=412
W
Transformer;
q
X
Y
Convolution,
French n=28
Convolutions
English n=351
for Hame D
M Ham p a
Once upon time
Once upon time
Mandarin n=33
Viail 2002: Rm-0.5
Vioaa W27132 R:1229
Visel #3342:: R.0045
tamp pam
Speech 8h30
Sentences n=9,000
300
22.0
2000
130
200
Time
Time 00
Time
R corr(Y, W (X))
Figure 1: Comparing speech representations in brains and deep neural networks. We an- alyze the brain activity of 412 participants recorded with functional Magnetic Resonance Imaging (fMRI) while they passively listened to audio books in their native language (French, English or Mandarin). B. After training wav2vec 2.0 [Baevski et al. 2020) with self-supervised learning (L) over h of unlabelled, effective speech, we extract its activations in response to the audio books that were presented to the participants. We assess the similarity between the activations of the model X and brain activity Y with a standard encoding model W [Nastase et al. 2020] evaluated with a cross-va volsdaeddd Pearson correlation R. C. Examples of the true BOLD response (black) and the pre- dicted BOLD response (red) estimated from a linear projection of the model's 's activations in three voxels randomly selected from the 10th percentile of best voxels identified by the noise ceiling anal- ysis for the first 200 of a representative story in the test set.
2.1.2 Learning objective
Self-supervised learning. In this training paradigm, the model optimizes losses. The first loss is contrastive and requires the model to predict the quantized representation of some masked input using c, from a finite of quantized representations drawn from the input sample. The second loss ensures that the quantized representations are diverse. See Section A.2 and [aevski et al. 2020 for details.
Supervised learning. In this training paradigm, the quantization module discarded and linear layer mapping to phonemes is added at the end of the pipeline. The model is randomly initialized and all layers (including the feature encoder) are trained using a Connectionist Temporal Classifi- cation (CTC) [Graves] 2012) loss to perform phone recognition. For both training paradigms, extract the activations each layer from both the feature encoder (outputting 2) and the context network (outputting c). We extract the representations of the convolutional and transformer blocks using an input window of 10s of raw waveform (stride = 5).
2.1.3 Training
Datasets. We successively train different wav2vec 2.0 models using each of four datasets: (i)
the French and (ii) English Common Voice corpora Ardila et al. 2020]. (iii) the MAGICDATA Mandarin Chinese Read Speech Corpus [Co. 2019). and (iv) a non-speech subset of the Audioset dataset [Gemmeke et al. 2017) which contains recordings of various acoustic scenes.

---

A
B A2
STG
STS
0.1
Untrained
Supervised
p-10
Self-supervised
02
.04
Figure 2: Sell-supervised learning suffices for wav2vec 2.0 to generate brain-like representa- tions of speech. A. Brain score (R) assessed for each subject and voxel independently, and here averaged across subjects for clarity. Only scores significantly above chance level, assessed using two-sided Wilcoxon test across subjects after correction for multiple comparison are color-coded (p < 10-11). B. scores for the same wav2vec2 model, averaged across subjects and voxels in four brain areas typically involved during speech processing (the primary and secondary auditory cortices, the superior temporal gyrus, the superior temporal sulcus, and the infero-frontal gyrus). In grey, the brain score obtained with randomly initialized wav2vec 2.0 architecture. Error bars are the standard errors of the mean (SEM) across subjects. The stars indicate a significant difference between the random and trained model (all p < 10 +) C. R. scores of wav2vec 2.0 without training (top), trained with a supervised (middle) and self-supervised learning rule (bottom), on the same 600 hours of effective speech. Scores averaged across subjects and voxels and error bars SEM across subjects.
Preprocessing. All the audio datasets were randomly subsampled to have approximate size of 600 hours, downsampled to 16 and converted to mono with the Sox softward We ran- domly split the datasets into training (80%), validation (10%) and test set (10%). The audio recordings we use from the Audioset dataset are filtered so that they do not contain speech or any sounds produced by humans, such as laughter singing. For the speech datasets, corresponding annotations (in the supervised settings). We phonemize these annotations using eS- peakNÇ The number of different phoneme symbols in these annotations is similar for French (32), English (39), and Mandarin Chinese (33).
also their
Implementation.
We train all of our models using the fairseq implementation wav2vec 2.0 us- ing default hyperparameters. We also analyze a model whose parameters were randomly initialised ("untrained"" model).
We use self-supervised learning to train four models: three on the speech datasets (French, English, and Mandarin) and one on the acoustic scenes dataset. In each case, the training was performed using the same configuration file (namely, the base configuration provided in the fairseq repository for pretraining wav2vec 2.0 on LibriSpeech [Panayotov et al. 2015)). We train the models for 400k updates and select the ones with the best validation loss.
We also use the supervised training paradigm to train three models, on the French, English, and Mandarin datasets, respectively. Each training was performed using the same configuration file, which was identical to the configuration provided in the fairseq repository for tinettunng wav2vec 2.0 on the 960 hour Voxpopuli corpus [Wang et al. 2021], except that parameters were not frozen (freeze_ firrteffintueeeeemm _upettes_______eeeeeee 0) and learning was performed on all parameters of the models using the CTC loss (feature_ge) _tt_earrrrddddd__aaaaeee__ _mmmm=== 0.1). We train the models for 400k updates and we use the ones with the best word error rate (WER) the validation set. The French model obtains 13.9 WER, the English model 28.6 WER, and the Mandarin model 4.6 WER, on their respective test sets.

---

A
Best layer
8
12
16
B
C
18
A1 and A2
STG
STS
IFG
70%
trs
Layer
Share of voxels
convs
0%
R
15
15
Best layer
Figure 3: The functional hierarchy of wav2vee 2.0 maps onto the speech hierarchy in the brain. A. We compute the R score for each layer of wav2vee 2.0 separately and estimate, for each voxel, the layer with highest brain score average across subjects. Only the voxels with significant brain scores are displayed (p < 10 11)). While the first transformer layers (blue) map onto the low-level auditory cortices (A1 and A2), the deeper layers (orange and red) map onto brain regions associated with higher-level processes (e.g. STS and IFG). B. Laser-wise R scores averaged across all voxels. Error bars SEM across subjects. C. Proportion of voxels with most predictive layer (x-axis) in four regions typically involved in speech processing. While most voxels in the primary cortex are best predicted by the first layers of the transformer, higher-level brain areas are best predicted by deeper layers.
2.2 Functional MRI
We analyse a composite set of fMRI recordings aggregated from the Little Prince and the Narratives public datasets Nastase et al. (222].
Li et al.
2021
Narratives. This datase contains the IMRI recordings of 345 native Eiglish-speakiin partici- pants listening to English narratives (4.6 hours of unique audio in total). The participants listened to different stories varying from 7 to 98 min (mean = 26 min). Following [Nastase et al. 2020), we (1) focus on fifteen representative stories and ignore the narratives that have been modified by scram- bling and (2) exclude eight participants because of noisy recordings. Overall, this selection results curated dataset 303 participants listening to fifteen stories ranging from min to 56 min, for total of hours of unique audio (36,018 words from vocabulary of 4,004 unique words).
The Little Prince. This LataseFFoontain fMRI recordings of English native speakers, 33 Man- darin native speakers, and 28 French native speakers listening to The Little Prince in their respective native language. The experiment itself was divided into nine runs of approximately 10 min of pas- sive listening. For each language condition, the story read by single native speaker. The English, Mandarin, and French audiobooks last 94, 90 and 97 minutes respectively.
5

---

C
Humans
Models
Al and A2
STS
IFG
Motor
Native
Non-Native
Non-Speech
Random
ABX acc
ABX acc
.19
.07
R .04
R. .02
Native better than non-native speech
Non-native speech better than non-speach
Non-speech better than random model
Random model better than chance
Figure 4: The specialization of wav2vec 2.0's representations follows and clarifies the acoustic, speech, and language regions in the brain. A. We first evaluate humans' language specificity by quantifying their ability to perceive phonemes their native or non-native languages (Section 2.4 in ABX matching-ttossmppe task [Schatz] 2016) (higher is better). As expected, humans better at matching phonemes of their native language. B. Then, we train four wav2vee 2.0 models with superviud learning four datasets non-speech acoustic scenes, English, and French, and compute their ABX accuracy on the same speech datasets as humans. The 'random' model is wav2vec 2.0 without any training. C. Brain score (R) of each model (with added model trained on Mandarin), averaged across voxels, in four regions of the brain (Section 2.2). D. Acoustic, speech and language specificity for each voxel. For instance, model its native R score is higher than its 'non-native' R score (p < .05). Only the voxels with significant R scores for the untrained model displayed across phone pairs in A and B, and across subjects in C. The stars indicates significant difference between conditions (Section 2.3
voxel is considered specific to the 'native'
10-18). Error bars the SEM
Preprocessing. For Narratives, we did not perform additional preprocessing: we use the public preprocessing of the dataset already projected on the surface space ("fsaverage6") without spatial smoothing (labelled "afni-nosmoothh in the data repository). In contrast, the Little Prince dataset is only provided in a volumetric space. Consequently, for each language condition separately, we subselected the cortical voxels by computing a brain mask using the average of all participants' fMRI data realigned onto a common template brain via Freesurfer [Fischl] [012]. These voxels are then projected onto a brain surface using nilearn's 's vol_to hol_to_sur's function with defaults parameters Abraham al. 2014) and a 'fsaverage6' template surface (Fischl 2012]. For both Narratives and The Little Prince, fMRI signals are normalized across the time dimension to have mean of 0 and variance of 1, for each participant, surface voxel and session independently.
Brain parcellation. For the purposes certain analyses, we group the IMRI voxels into regions of interest using the Destricux Atlas [Destricux et al. 2010). This parcellation results in 75 brain regions in each hemisphere. For simplicity, label the regions as follows: Al and A2 represents Heschl gyrus, which is the anatomical location of the primary and secondary auditory cortices, STG and STS are the superior temporal gyrus and sulcus, and IFG is the inferior frontal gyrus.
2.3 Brain score (R)
To quantify the similarity between the network's activations X and the brain recordings Y, we use standard linear encoding model (Huth et al. (2066] Yamins and DiCarlo (0T6) each subject, we split the data into train and test sets using five-fold cross-validation setting. For each train split,

---

linear mapping W is fitted to predict the brain response Yirain given Xtrain W combines temporal alignment function with fixed weight, and a trained penalized linear regression.
Temporal alignment. The sampling frequency of the model's activations (between 49 and 200 Hz) differs from the sampling frequency of iMRI BOLD signals (0.5 Hz). Furthermore, the BOLD signals have delayed responses spanning over several seconds. Thus, we first convolve the model activations with a standard hemodynamic response function (HRF) using nistats [Abraham et al. 2014] serpppttererreooot function with the 'glover' model and default parameters. This results in the convolved activations (rain with the same sampling frequency as the fMRI Yirain (sce A.3)
Penalised linear regression. Once temporally aligned, fit an é2-penalised linear regression that predicts the brain signals Yurain given the activations Xtrain- We use the RidgeCV function from scikit-learn [Pedregosa al.] 201 with the penalization hyperparameter varying between 10 and 10t (20 values scaled logarihhmiclll) chosen independently for each dimension with nested cross-validation over the training set (see A.4p.
Evaluation. We evaluate the linear mapping W on the held out sets by measuring Pearson's 's cor- relation between predicted and actual brain responses: R = corr (Y W X Finally, we av- erage the correlation scores across test splits to obtain the final "brain score". To report the average layer k- with the highest brain score for each voxel (Figure 3). while being robust to regression- to-the-mean biases, we first find the best layer k, for each participant and each voxel indepen- dently and then compute a circular mean across the N = 412 participants and the 19 layers:
k' = angle
E exp
N
2irk.
K+1
Statisties. We assess the reliability of brain scores with second-level analyses across participants thanks to a Wilcoxon signed-rank test across participants. Thus, the resulting p-values are not af- fected by IMRI auto-correlation within participants. We perform statistical correction for multi- ple comparisons with Benjamini-Hochbery False Discovery Rate (FDR) across voxels [Benjamini 2010
2.4 Behavioral experiment
To compare the phonetic representations of our models to those of humans, we compare the forced- choice discrimination judgements of online participants an analogous method applied to wav2vec 2.0 [Schatz] 2016). Specifically, for each triplet of sound "ABX", participants judged whether the stimulus X was more similar A B. Analogously, we computed the Euclidean distance in the most discriminative layer of wav2vec 2.0 (here transformer layer 5) to determine whether X was closer A or B. Additional data, analyses and model-human comparison can be found in (Millet and Dunbar 2022]. We focus on the French and English stimuli, which represent R 6,000 ABX triplets (testing 508 English and 524 French phone pairs), with 386 participants in total (193 from each language group).
In Figure AA report the ABX accuracy of English- and French-speakinn participants in both their native and non-native language (either English or French). We first average results per phone pair, and then average over phone pairs obtain the ABX discrimination accuracy. Similarly, in Figure B, we compute the ABX accuracy of our wav2vec 2.0 models on the same evaluation sets the participants, using the parameters deseribed in (Millet and Dunbar 2022]. . English and French models are evaluated on the same ("native') or different ("non-native") language stimuli as their training. The random and non-speech models are evaluated on both French and English speech stimuli.
3 Results
Wav2vec 2.0 maps onto brain responses to speech. We estimate whether the activations of wav2vec 2.0 models linearly map onto the human brain activity of 412 individuals listening to audio books in the fMRI scanner. For this, we first independently train three models with 600 h of

---

French, English, or Mandarin, respectively, and compute the brain scores (R) with the correspond- ing participants. Specifically, we (1) convolve the activations (X) of the model with a hemodynamic response function (HRF). (2) train a fz-penalized linear regression on a training split to map them to brain activity Y, and (3) compute the Pearson correlation coefficient between (i) the true fMRI activity and (ii) the predicted activations on test split. The models' activations significantly predict brain activity in nearly all cortical areas, reaching the highest It scores in the primary and secondary auditory cortices (Figure A B). These scores are significantly higher than those obtained with randomly initialised model (p < 10-50 on average across voxels), and this comparison is robust across language groups (all p 10 10-5).
Comparison of self-supervised to supervised models. Does self-supervision reach representa- tions that are as brain-like as those obtained with supervised learning? To address this issue, we trained wav2vec 2.0 with an alternative, supervised objective, namely, predicting phonetic anno- tations from the same 600 hours of effective speech sounds. We then implemented the R score analyses described above. The results show that self-supervised learning in fact leads to modestly but significantly better scores than supervised learning (Figure C): AR 0.002, p < 10
The hierarchy of wav2vec 2.0 maps onto the hierarchy of the cortex. To compare the speech hierarchy in the brain with the functional hierarchy learned by wav2vec 2.0, we evaluate the R score of each layer of the model (Figure D. First, we observe that convolutional layers are less predictive than transformer layers. Second, within the transformers, the hierarchy of representations aligns with the expected cortical hierarchy (Hickok and Poeppel, 2007]: while low-level areas (AI, A2) best predicted by the first transformer layers, higher level areas (IFG, STS) are best predicted by deeper layers. Remarkably, this hierarchy extends to supplementary motor and motor areas in both hemispheres (Figure A).
Language specificity in phone discrimination tasks. The acoustic features underlying speech (fricatives, vowels, and so on) may also characterize non-speech sounds (the sound of tree leaves in the wind, of a stone falling, and so on). Does the model show commonalities merely with general auditory processing in the brain, or does it capture speech-specific processing? If so, does it show commonalities with brain representations that are specific to the native language of the participants, or merely to general speech processing? We first evaluate the specialization of humans' perception to their native language using an ABX behavioral task (Section 2.4.. Specifically, we compare 386 French and English participants on their ability to distinguish native and non-native phones. As expected (Bohn 2017 Kuhl al. 2005 participants were better at discriminating native sounds than non-native ones (across phone pairs: p < 10-18, Figure AA). Second, applying the same test to our self-supervised French and English models shows that, like humans, models best discriminate sounds from their 'native" language (i.e., the French model better distinguishes French stimuli than English ones, across phone pairs, and vice versa: p < 0.05). Interestingly, the ABX accuracy of the model is significantly higher than participants'. : This quantitative difference may be partially explained by the fact that participants and online participants in particular undergo fluctuating attention, and adopt strategies which can negatively impact performance [Humphreys] 1939). Fi- nally, as expected, the random and acoustic models obtain the worst ABX accuracy. Overall, These results confirm that 600 h of self-supervised learning on effective speech suffices for wav2vec 2.0 to learn language-specifie representations (Figure 4 B).
Wav2vec 2.0 and the brain learn language specific representations. Next, we compare the brain scores of random, non-speech, non-native and native models (Figure D).. First, our re- sults show that the non-speech model attains higher R scores than the random model (on average across voxels, AR = 0.006, 10 10-31) confirming the importance of learning to generate brain- like representations. Second, non-native models attain higher scores than the non-speech model (AR = 0.002,p p = 10-9), confirming that wav2vec 2.0 learns speech-specific representations of sounds when trained on speech. Finally, the native model attains higher R scores than non-native models (AR- = 0.002, p = 10-15).
8

---

4 Discussion
Human infants acquire language with little to no supervision: A few hundred hours of speech suffices for their young brain to learn to discretize phonemes, segment morphemes, and assemble words in the language(s) of their social group [Dupoux] 2018 Gilkerson et al. 2017]. However, the learning principle that allows this unique feat remains, to date, unknown.
Here, we test whether self-supervised learning applied to a limited amount of speech effectively accounts for the organization of speech processing in the human brain as measured with fMRL. For this, we train several variants of wav2vec 2.0 [Bacvski et al. 2020] with three curated datasets of French, English, and Mandarin, and compare their activations to those of large group of French, English, and Mandarin speakers recorded with fMRI while passively listening to audio stories. Our results show that this self-supervised model learns (i) representations that linearly map onto a markably distributed set of cortical regions (Figure - (ii) a computational hierarchy that aligns with the cortical hierarchy (Figure 3) and (iii) features specific to the language of the participants (Figure 4
Towards a paglogically-plausibee learning principle. These results extend recent findings on the similarities between the brain and variety of deep learning models trained with biologically-
implausible objectives and data. First, fMRI [Kell et al.) 2018, Millet and King 2021 Thompson et al. 2021], elecroencephallogrphyy (Huang et al. 2018 and multi- or single-unit responses to sounds Koumura et al., 2019, [eegus et al. [022] have been shown to be linearly predicted by the activations of deep convolutional networks trained on supervised auditory tasks. For example, [Millet and King [021] showed that a supervised speech-to-text model better accounted for brain re- sponses speech in I02 individuals when it was trained on speech recognition rather than auditory scene classification. Similarly, [Kell et al. 2018 showed that eight participants listening to brief speech and non non-speech sounds demonstrated fMRI responses in the temporal lobe that aligned with those of a deep convolutional neural network trained on a binary auditory classification task. Our results, based on up to 50 times more fMRI recordings of the entire cortex show that such represen- tational similarities hold with self-supervised objective [Lerner et al.
2011 Berezutskaya et al.,
2017 Caucheteux et al.] 2021c]b). [b]. Second, growing series of MEG
Toneva and Wehbe 2019 2016 Pereira et al. 2018
Caucheteux and King 2022]. fMRI [Mitchell et al., 2008 Qian et al.)
Schwartz et al.J 2019 Antonello et al. 2021] Jain and Huth 2018) and electro-physiology studies
[Schrimpf et al.) |021 Goldstein et al. 2022 showed that text-based language models trained on very large corpora generate brain-like representations too. While these results suggest elements of
convergence between language models and the brain [Caucheteux and King, 2022]. they also remain biologically implausible: not only are these algorithms pre-equipped with abstract linguistic units such characters and words, but they trained corpora that in their lifetime. In contrast, wav vve 2.0 is here trained with a reasonable amount of raw speech waveforms (Hart and Risley 1992 Gilkerson al., 2017 Dupoux 2018). The functional similar- ity between wav2vec 2.0 and the brain thus opens the way to clarify how humans learn to process speech.
would ever be able to read
The emergence of a brain-like hierarchy of speech processing. The present study reveals the hierarchical organization of speech processing with remarkable clarity. First, the functional hierar- chy learnt by wav2vee 2.0 is aligned with the anatomy: e.g. the superior temporal sulcus and the temporal pole are known to project to the ventral and dorsal part of the inferofrootal gyrus, respec- tively (Petkov al. 2015). Second, the identification of functional gradients within the prefrontal cortex, and down to the motor areas typically associated with larynx and mouth control Dichter al. 2018 reinforces the relevance of motor processes to speech perception [Kelii et al. 2010]]
Mugler et al. 2014, (hammaa et 202T). Finally. the existence of multiple levels of representations around the inferofrontal cortex is consistent with the idea that Broca's area may be responsible for
merging linguistic units [Chomsky] 2000 Friederici 1999 Hagoort 2005 PPeppel et al.) 2012). It
should be noted, however, that our results aggregate a large cohort of individuals which could mask a more modular organization at the individual level.
Interpreting the neural representations of speech. Interpreting neural representations is no- toriously difficult challenge to both Al and neuroscience. Here, we first investigate language speci- ficity and show that the neural representations specific to the native models are primarily represented

---

in the superior temporal sulcus and middle temporal gyrus (Figure 4D): areas known to represent phonetic features [Mesgarami et al. 2014). However, these effect are relatively modest (Figure F the random model and the non-speech model reach, in STS and STG, 67% and 87% of the brain scores obtained by the "native" model, respectively. While this high baseline initially surprised us, this phenomenon could be explained by the fact that the auditory cortex is continuously bombarded by 11-am and should thus be tuned to non-speech input. Second, our probing analyses show that the models trained with self-supervised learning learn relevant acoustic and linguistic representations (Supplementary Figure ST). This result, consistent with Vaidya et al. [2022] and [Stephenson et al. (019), suggests that the difference of brain scores observed between the random, non-native and na- temporal, phonetic, word and sentence-level representations, respectively. These elements of interpretation remain, however, scarce, and a systematic interpretation of the representations shared between wav2vec 2.0 and the brain remains necessary.
tive models (Figure 4 may be partly driven by the corresponding spectro-temporal,
Scope of the study. It is important to stress that the scope of the present study could be broad- ened in several ways. First, our study focuses on adult speakers. whose cultural and educational background is not representatie of the population [Henrich et al. 2010]. Second, we focus on the passive listening of three languages. Third, we focus on one self-supervised learning architecture [Bevvki et al. 2020]. and its functional alignment with IMRI, whose temporal resolution is notori- ously limited. Generalizing the present approach to more languages MMalk-MMralee larger spectrum of children and adult participants recorded with a variety of electrophysiologiee and neuroimaging devices will thus be essential to confirm, precise, and/or mitigate the present findings.
al. [022]]
The remaining gap between brain and speech models. Several major gaps can be evidenced between wav2vec 2.0 and the brain. First, the transformer layers are not temporally constrained: each layer can access all elements within the contextual window. This differs from the necessarily recurrent nature of processing the brain. Second, wav2vee 2.0 behaves differently to humans in specific tasks. In particular, it is overly-sensitive to band-pass filtering, non-robustly exploit fine temporal structures [Weerts et al.) 2021] and fails to display the expected categorical responses [Millet et al.) [021]. Third, recent studies show that wav2vee 2.0 encodes significantly less semantic information than text-based models [Pasad et al.) 2021 Vaidya et al. 2022]. While our analyses suggest that learning allows wav2vec 2.0 to capture some lexical features in its deep layers (Figure ST] Table S4). remains unclear whether these layers also capture complex syntactic structures, such recursive syntactic trees [Lakretz et al. 20221 Caucheteux et al. [22]la]. We speculate that these limitations may be due to the time scales of wav2vee 2.0 which, unlike humans, learns very short-term representations of speech. In any case, these differences likely explain why the brain scores of wav2vec 2.0 remain substantially lower than our noise-ceiling (19% on average, and up to 74% in Heschl's ' gyrus and suleus, Table ST Figure (2).
Overall, the complexity of the human brain is often thought to be incompatible with a simple theory: "Even if there were enough data available about the contents of each brain area, there probably would not ready set of equations to describe them, their relationships, and the ways they change over time" [Gallant 2013]. J By showing how the equations of self self-superiieed learning give rise to brain-like processes, this work contributes to challenge this view.
Acknowledgments
This project was funded, in part, by the Sireenoourt-cchelllee Foundation, the Philippe Foundation, and FrontCog grant ANR-I7-EURE-0017 to JRK for his work at PSL.
References
Alexandre Abraham, Fabian Pedregosa, Michael Fickenberg, Philippe Gervais, Andreas Mueller, Jean Kossaifi, Alexandre Gramfort, Bertrand Thirion, and Gaël Varoquaux. Machine learning for neuroimaging with scikit-learn. Frontiers in neuroinformatics, 8:14, 2014.
Richard Antonello, Javier S Turek, Vy Vo, and Alexander Huth. Low-dimensional structure in the space of language representations reflected brain responses. Advances Neural Information Processing Systems, 34, 2021.
10

---

Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber. Common voice: A
massively-muliiingaal speech corpus. In IREC, 2020.
Alexei Baevski, H. Zhou, Abdel rahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. ArXiv, abs/2006.11477, 2020.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: for self-supervised learning. arXiv preprint arXiv:210S.04990. 2021.
regularizationnn
Gasper Begus, Alan Zhou, and Christina Zhao. Encoding of speech in convolutional layers and the brain stem based on language experience. bioRxiv, 2022.
Yoav Benjamini. Discovering the false discovery rate. Journal of the Royal Statistical Society: series B (statistical methodology). 72(4):405-41,, 2010.
Julia Berezutskaya, Zachary Freudenburg, Umut Güęlü, Marcel AJ van Gerven, and Nick Ram- sey. Neural tuning to low-level features of speech throughout the perisylvian cortex. Journal of Neuroscience, 37(33):7906-7922 2017.
Ocke-Schwen Bohn. Cross-language and second language speech perception. The handbook of psycholinguistics, pages 213-239, 2017.
Lasse Borgholt, Jakob Drachmann Havtorn, Joakim Edin, Lars Maaløe, and Christian Igel. A brief overview of unsupervised neural speech representation learning. arXiv preprint arXiv:2203.01829, 2022.
Charlotte Caucheteux and Jean-Rémi King. Brains and algorithms partially converge in natural language processing. Communications Biology, 5(1):1-10, 2022.
Charlotte Caucheteux, Alexandre Gramfort, and Jean-Rent King. Disentangling syntax and seman- tics the brain with deep networks. In Proceedings of the 38th International Conference on Machine Learning, pages 1336-1344 1348. PMLR, July 2021a. ISSN: 2640-3498.
Charlotte Caucheteux, Alexandre Gramfort, and Jean-Remi King. Long-range and hierarchical lan- guage predictions in brains and algorithms. arXiv:2111.142.1 [cs, q-bio], November 2021b. arXiv: 2111.14232.
Charlotte Caucheteux, Alexandre Gramfort, and Jean-Remi King. Model-based analysis of brain activity reveals the hierarchy language in 305 subjects. Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3635-3644, Punta Cana, Dominican Republic, November 2021c. Association for Computational Linguistics.
Charlotte Caucheteux, Alexandre Gramfort, and Jean-Rémi King. Deep language algorithms predict semantic comprehension from brain activity. Scientific Reports, 12(1):16327, September 2022. ISSN 2045-2322, doi: 10. Group.
Number: Publisher: Nature Publishing
Noam Chomsky, Linguistics and brain science. Image, language, brain, pages 13-28, 2000.
Radoslaw M Cichy and Daniel Kaiser. Deep neural networks scientific models. Trends in cogni- live sciences, 23(4):00--37,, 2019.
Beijing Magic Data Technology Co. speech. http: http://www.. /www. itaggiddsaattehh.cm///neeeeetet http:///weeee.iiooooommoaottaaae info/idi /////i//111111 2019.
Magic data chinese mandarin conversational www./oomeenseeeeeeee
Christophe
Destrieux, Bruce Fischl, Anders Dale, and Eric Halgren. Automatic parcellation of human cortical gyri and sulci using standard anatomical nomenclature. Neurolmage, 53(1):1-15, October 2010. ISSN 19.. doi: 10. wwt.://eeurommeee21000....
Benjamin K Dichter, Jonathan D Breshears, Matthew Leonard, and Edward Chang. The control of vocal pitch in human laryngeal motor cortex. Cell, 174(1):21-31, 2018.

---

Emmanuel Dupoux. Cognitive science in the era of artificial intelligence: A roadmap for reverse-
engineering the infant language-leamer. Cognition. 173:43-59, 2018.
Bruce Fischl. Freesurfer. Neuroimage, 62(2):774-781, 2012.
Angela D Friederici. The neurobiology of language comprehension. In Language comprehension:
biological perspective, pages 265-304. Springer, 1999.
Jack Gallant. in 'reading minds'. Nature, 502(7472):428, 2013.
Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 776-780. IEEE, 2017.
Jill Gilkerson, Jeffrey A Richards, Steven F Warren, Judith K Montgomery, Charles R Greenwood, D Kimbrough Oller, John HL. Hansen, and Terrance D Paul. Mapping the early language envi- ronment using all-day recordings and automated analysis. American journal of speech-language pathology, 26(2):248 265, 2017.
Ariel Goldstein, Zaid Zada, Eliav Buchnik, Mariano Schain, Amy Price, Bobbi Aubrey, Samuel A Nastase, Amir Feder, Dotan Emanuel, Alon Cohen, et al. Shared computational principles for language processing in humans and deep language models. Nature neuroscience, 25(3);369-380, 2022.
Alex Graves. Connectionist temporal classification. In Supervised Sequence Labelling with Recur- rent Neural Networks, pages 61 93.9 Springer, 2012.
Peter Hagoort. On broca, brain, and binding: new framework. Trends in cognitive sciences, 9(9): 416-423, 2005.
Betty Hart and Todd Risley. American parenting language-learning children: Persisting dif- ferences in family-child interactions observed in natural home environments. Developmental psychology, 28(6):1096, 1992.
Joseph Henrich, Steven J. Heine, and Ara Norenzayan. The weirdest people in the world? Be- havioral and Brain Sciences, 33(2-3):61-8) June 2010. ISSN 01 0140-525X, 1469-1825. doi: 10. 10.10177501445555099912222
Gregory Hickok and David Poeppel. The cortical organization of speech processing. Nature reviews neuroscience, 8(5):393-402, 2007.
Nicholas Huang, Malcolm Slaney, and Mounya Elhilali. Connecting deep neural networks phys- ical, perceptual, and electrophysiological auditory signals. Frontiers in neuroscience, 12:532, 2018.
L. G. Humphreys. Acquisition and extinction of verbal expectations in situation analogous to conditioning. Journal of Experimental Psychology, 25:294-301, 1939. ISSN 0022-1015. doi: 10.1037/h0053555. Place: US Publisher: American Psychological Association.
Alexander G. Huth, Wendy A. de Heer, Thomas L. Griffiths, Frédéric E. Theunissen, and Jack L. Gallant. Natural speech reveals the semantic maps that tile human cerebral cortex. Nature, 532 (7600):453 458, April 2016. ISSN 0028-0836, 1476-4687. doi: 10. 1038/nature 17637.
Shailee Jain and Alexander Huth. Incorporating context into language encoding models for fmri. Advances in neural information processing systems, 31, 2018.
Alexander JE Kell and Josh H McDermott. Deep neural network models of sensory systems: win- dows onto the role of task constraints. Current opinion in neurobiology, 55: 55:121-132, 2019.
Alexander JE Kell, Daniel LK Yamins, Erica N Shook, Sam V Norman-Haignere, and Josh H Mc- Dermott. task-optimized neural network replicates human auditory behavior, predicts brain responses, and reveals cortical processing hierarchy. Newron, 98(3))633--44 2018.
12

---

Spencer Kellis, Kai Miller, Kyle Thomson. Richard Brown, Paul House, and Bradley Greger. De- coding spoken words using local field potentials recorded from the cortical surface. Journal of neural engineering, 7(5):056007, 2010.
Tim Kietzmann, Patrick McClure, and Nikolaus Kriegeskorte, Deep neural networks in computa- tional neuroscience. BioRviv, page 133504, 2018.
Takuya Koumura, Hiroki Terashima, and Shigeto Furukawa. Cascaded tuning to amplitude modu- lation for natural sound recognition. Journal of Neuroscience, 39(28):5517-533, 2019.
Nikolaus Kriegeskorte. Deep neural networks: a new framework for modeling biological vision and brain information processing. Annual review of vision science, 1:417-446, 2015.
Patricia K Kuhl, Barbara T Conboy, Denise Padden, Tobey Nelson, and Jessica Pruitt. Early speech perception and later language development: Implications for the" critical period". Language learning and development, 1(3-4))237-2661 2005.
Yair Lakretz, Théo Desbordes, Jean-Rémm King, Benoît Crabbé, Maxime Oquab, and Stanislas De- haene. Can RNNs learn Recursive Nested Subject- Verb Agreements? arXiv:2101.02258 [cs], January 2021. URL htt.::ttt://rttt/ org/ 22001 02258
arXiv: 2101.02258.
Y. Lerner, J. Honey. J. Silbert, and U. Hasson. Topographic Mapping of Hierarchy of Temporal Receptive Windows Using a Narrated Story. Journal of Neuroscience, 31(8):2906- 2915, February 2011. ISSN 0270-6474,
doi: 10.1523/NEUROSCC33664 10.
Jixing Li, Shohini Bhattasali, Shulin Zhang, Berta Franzluebbers, Wen-Ming Luh, R Nathan Spreng, Jonathan Brennan, Yiming Yang. Christophe Pallier, and John T Hale. petit prince: A multilingual fmri corpus using ecological stimuli. bioRxiv, 2021.
Saima Malik-Moraleda, Dima Ayyash, Jeanne Gallée, Josef Affourtit, Malte Hoffmann, Zachary Mineroff, Olessia Jouravlev, and Evelina Fedorenko. An investigation across 45 languages and language families reveals a universal language network. Nature Neuroscience, 25(8):0014 1019, August 2022. ISSN 1546-1726, doi: 01138///11593022--0111 Nature Publishing Group.
Number: 8 Publisher:
Nima Mesgarani, Connie Cheung, Keith Johnson, and Edward F. Chang. Phonetic Feature Encoding in Human Superior Temporal Gyrus. Science, 343(6174) 77):1006610014 February 2014. ISSN 0036-8075, 1095-9203, doi: 10.1 126/scc/sce 129osciienee124599444 1245994. Publisher: American Association for the Advancement of Science Section: Report.
Juliette Millet and Ewan Dunbar. Do self-supervise speech models develop human-like perception biases? AAAI 2022 Workshop, Self-Supervised Learning for Audio and Speech Processing, 2022.
Juliette Millet and lean-Remi King. Inductive biases, pretraining and fine-tuning jointly account for brain responses to speech. arXiv preprint arXiv:2103.01032, 2021.
Juliette Millet, Ioana Chitoran, and Ewan Dunbar. Predicting non-native speech perception using the perceptual assimilation model and of-the-ar acoustic models. In Proceedings of the 25th Conference on Computational Natural Language Learning, pages 661-673, Online, November 2021. Association for Computational Linguisties. doi: 10. 11183865111000 .ooll...o.
Tom M Mitchell, Svetlana V Shinkareva, Andrew Carlson, Kai-Min Chang, Vicente L Malave, Robert Mason, and Marcel Adam Just. Predicting human brain activity associated with the meanings of nouns. science, 320(5880): 1991-1995, 2008
Emily M Mugler, James L Patton, Robert D Flint, Zachary A Wright, Stephan Schuele, Joshua Rosenow, Jerry J Shih, Dean Krusienski, and Mare W Slutzky. Direct classification of all american english phonemes using signals from functional speech motor cortex. Journal of neural engineering, 11(3)0335015, 2014.
Thomas Naselaris, Kendrick N Kay, Shinji Nishimoto, and Jack Gallant, Encoding and decoding in fmri. Neuroimage, 4612):400-410, 2011.
13

---

Samuel A Nastase, Yun-Fei Liu, Hanna Hillman, Asieh Zadbood, Liat Hasenfratz. Neggin Ke- shavarzian, Janice Chen, Christopher J Honey, Yaara Yeshurun, Mor Regev, al. Narratives: fmri data for evaluating models of naturalistic language comprehension. preprint. Neuroscience, December, pages 2020-06, 2020.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acouslics, speech and signal processing (ICASSP), pages $206-52100 IEEE, 2015.
Ankita Pasad, Ju-Chieh Chou, and Karen Livescu. Layer-wise analysis of a self-supervised speech representation model. arXiv preprint arXiv:2107.04734 2021.
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Seikit-learn: Machine learning in python. the Journal of machine Learning research, 12:2825-2830, 2011.
Francisco Pereira, Bin Lou, Brianna Pritchett, Samuel Ritter, Samuel J Gershman, Nancy Kan- wisher, Matthew Botvinick, and Evelina Fedorenko. Toward universal decoder of linguistic meaning from brain activation. Nature communications, 9(1):1-13, 2018.
Christopher Petkov, Yukiko Kikuchi, Alice E. Milne, Mortimer Mishkin, Josef P. Rauschecker, and Nikos K. Logothetis. Different forms of effective connectivity in primate frontotemporal pathways. Nature Communications, 6(1):6000, January 2015. ISSN 2041-1723. doi: 10. 10.1038/ ncomms7000. Number: 1 Publisher: Nature Publishing Group.
David Poeppel, Karen Emmorey, Gregory Hickok, and Liina Pylkkänen. Towards new neurobiol- ogy of language. Journal of Neuroscience, (114)):44125144111, 2012.
Peng Qian, Xipeng Qiu, and Xuanjing Huang. Bridging Istm architecture and the neural dynamics during reading. arXiv preprint 1eXiv:664.00663 2016.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Thomas Schatz. ABX-discriminaabliiy measures and applications. PhD thesis, Université Paris 6 (UPMC), 2016.
Martin Schrimpf, Idan Asher Blank, Greta Tuckute, Carina Kauf, Eghbal Hosseini, Nancy Kan- wisher, Joshua B Tenenbaum, and Evelina Fedorenko. The neural architecture of language: In- tegrative modeling converges on predictive processing. Proceedings of the National Academy of Sciences, 1 118(45), 2021.
Dan Schwartz, Mariya Toneva, and Leila Wehbe. Inducing brain-relevant bias in natural language processing models. Advances in neural information processing systems, 32, 2019.
Shihab Shamma, Prachi Patel, Shoutik Mukherjee, Guilhem Marion, Bahar Khalighinejad, Cong Han, Jose Herrero, Stephan Bickel, Ashesh Mehta, and Nima Mesgarani. Learning speech pro- duction and perception through sensorimotor interactions. Cerebral cortex communications, 2(1): tgaa091, 2021.
Cory Stephenson, Jenelle Feather, Suchismita Padhy, Oguz Elibol, Hanlin Tang, Josh McDermott, and Yeon Chung. Untangling in invariant speech recognition. Advances in neural information processing systems, 32, 2019.
Jessica AF Thompson, Yoshua Bengio, Elia Formisano, and Marc Schönwiesner. Training neural networks to recognize speech increased their correspondence to the human auditory pathway but did not yield shared hierarchy of acoustic features. bioRxiv, 2021.
Mariya Toneva and Leila Wehbe. Interpreting and improving natural-language processing (in ma- chines) with natural language-processiing (in the brain). arXiv: 1833 [cs, q-bio), November 2019. arXiv: 1103.1
Aditya R Vaidya, Shailee Jain, and Alexander G Huth. Self-supervised models of audio effectively explain human cortical responses to speech. arXiv preprint arXiv:2205, 14252, 2022.
14

---

Changhan Wang, Morgane Riviere, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux. Voxpopuli: large-scale multilingual speech corpus for representation learning, semi-supervvised learning and interpretation. arXiv preprint arXiv:2101.000390, 2021.
Lotte Weerts, Stuart Rosen, Claudia Clopath, and Dan FM Goodman. The psychometries of auto- matic speech recognition. bioRxiv, 2021.
Qiantong Xu, Alexei Baevski, Tatiana Likhomanenko, Paden Tomasello, Alexis Conneau, Ronan Collobert, Gabriel Synnaeve, and Michael Auli. Self-training and Pre-training Complemen- tary for Speech Recognition, October 2020. arXiv:2010. 1430 [cs, eess].
Daniel LK Yamins and James DiCarlo. Using goal-driven deep learning models to understand sensory cortex. Nature neuroscience, 19(3):356-36,, 2016.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] See Section 3
(b) Did you describe the limitations of your work? [Yes] See Section 4
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] We have, and since we use results of human experiments that were previously made, and that respect these review guidelines, we conform them.
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A] No theoretical results
(b) Did you include complete proofs of all theoretical results? [N/A] No theoretical results
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] The code will
be provided upon request. Otherwise for the rest, see Section 2.2 and 2.1.3
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] See Section 2.1.3
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [Yes] This is the case for all our figures, see Figure 2 and
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No] The models we train are not new: the original papers already give that type of information.
4. If you are using existing assets (e.g., code, data, models) curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] See Section 2.2
2.1.3 2.4
(b) Did you mention the license of the assets? [No] They all under CCO for the En- glish and French training audio data, the Perceptimatic Benchmark, The Little Prince dataset and the Narratives dataset. Magic data is under CC BY-NC-ND 4.0, Audioset is under CC BY 4.0 for the audio recordings, and CC BY-SA 4.0 for the ontology, Narratives' preprocessing code under GNU GENERAL PUBLIC LICENSE and fairseq code is under MIT License.
(c) Did you include any new assets either in the supplemental material or as a URL? [No] The code will be given upon request
(d) Did you discuss whether and how consent was obtained from people whose data you're re using/curating? [Yes] See Section 2.2 and 2.4 More details on the way they
gave their consent can be seen in the original papers/websites that provided the exper- imental results.
15

---

(c) Did you discuss whether the data you are using/curating contains personally identifi- able information or offensive content? [N/A] The data were anonymized before used them.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A] We did not conduct the experiments ourselves
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A] We did not conduct the experiments ourselves
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A] We did not conduct the experiments our- selves
16


Reviews:
[{"review_id": "xZaau6dfjNH", "reviewer": "Reviewer_6WfS", "summary": "In this work, the authors build speech encoding models using wav2vec 2.0 and experiment with 4 different regimens:\n1. Untrained network\n2. Network trained on non-speech sounds\n3. Network trained on non-native speech\n4. Network trained on native speech\n\nTheir main findings are:\n1. Representations from wav2vec 2.0 predict brain activity well in several regions. Although significantly different, the untrained networks have comparable effect sizes to the trained models.\n2. Diff regions across cortex are best predicted by diff layers of the model such that low-level regions like the auditory cortex are best predicted by the lower layers.\n3. Much like humans, the models do better at a phoneme discrimination task on the native language than non-native. The non-speech networks do worse, followed by the untrained networks.", "questions": "1. Selectivity for deeper layers in primary AC: In figure 3, why is primary auditory (or what is possibly Heschl’s gyrus) marked in red, ie., selective for deep layers? It would be useful to visualize the correlations for different layers in a few voxels to see if the curve is relatively flat and if this is an artifact of using a circular mean.\n2. What do the individual participant maps look like for figure 3? Is the group-averaging introducing smoothness in the gradients that is not visible in individual maps ( due to the circular mean estimate being noisy)?\n3. How were models trained for the ABX task? What are the specifications of the dataset and the task? Which layers were used to do this?\n4. ABX task results: I imagine that the authors ran clustering for the English/French/non-speech data independently to get different quantization targets for each. Given that these targets roughly correspond to phonemes, is it surprising that the English model was better at distinguishing English phonemes than French? Similarly, is it surprising that English models outperform non-speech or untrained models that don’t have English quantization targets or have not learned how to use them respectively?\n5. In Fig. 4D, it is surprising that only auditory cortex seems to care about language/speech specific features although we know that a large part of cortex processes natural speech. Can the authors comment on why this might be and perhaps visualize the individual correlations for all 5 conditions in a few example voxels per color?\n6. What is the amount of data per participant in each language & stimulus set? Iiic, the Enligsh datasets have far more data per participant than French. How does the paper circumvent differences in the quality of model fits due to different amount of data being used? (especially for cross-language comparisons)", "limitations": "-NA-", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Edit after rebuttal: I comment the authors efforts to do a new analyses and the thorough responses to the review. Consequently, I have increased my score by 1. However, at this point I cannot recommend the paper for acceptance as some things stand out to me (more detail in follow-up comments):\n- There is no established link between the different types of models tested, the layer-wise selectivity and the probing result. To this end, it is hard to interpret what the posited function of each region is or,\n- what new things did we learn about the brain/deep speech networks that wasn't know previously? The paper seems largely confirmatory and the methods employed are not novel.\n- some of the prediction correlations are surprisingly small even in areas robustly engaged during speech perception. Furthermore, some of the claims on selectivity for native vs. non-native vs. non-speech info is based on extremely small, possibly insignificant differences. This make interpretation hard.\n\n###################################################################################################\n\nStrengths:\n1. Clarity: The paper is well written and easy to read. \n\nWeaknesses:\n1. Significance/Originality: The central premise of this paper lies in exploring self-supervised architectures for building encoding models. The results presented include 1) encoding model performance maps 2) layer wise selectivity maps 3) encoding performance viewed through a cascade of model selectivity (native > non-native....> chance). However, due to several reasons highlighted in detail below (like no discussion/analysis on what information is represented at different layers and how does this explain patterns of variation in layer selectivity, for example) and due to the identical results+analyses in prior work on supervised models and concurrent work on self-supervised models, the novelty of this work is not clear. \n- Kell et al., Neuron 2018 -  exploring layer selectivity for diff supervised task architectures + specificity for speech vs. non-speech\n- Millet et al., arXiv 2021 - exploring many of the analyses presented here but with supervised models\n- Vaidya et al., ICML 2022 - exploring 4 self-supervised models, layer selectivity and probing to infer information at each layer\n2. Under-explored/Limited analyses: Although the paper presents encoding model performance in every major analysis, the trends observed are not discussed further. For example, what information do different layers capture that could drive selectivity for lower layers in low-level brain regions vs. selectivity for deeper layers in high-level regions? Does this information emerge as a consequence of training? Does it emerge only when models are trained on speech? How about specificity of information for the native language? Without digging deeper into these questions, I am not sure of the significance of the results or how they inform our understanding of both speech models and the brain.\n3. Overstating claims:\n- Claims of supervision or lack thereof that match human ability:\n    - Firstly, while the effect sizes are significantly different, these differences are still considerably small between the random/non-speech/non-native/native models.\n    -  It is interesting to compare supervised vs. unsupervised models in terms of the information they capture and consequently, their ability to predict brain data. However:\n        - given the implausibility of the learning mechanisms employed in these networks (as noted in the paper too; Lines 301-312)\n        - the fact that they do not follow the same patterns of errors/do not mimic human behavior\n        - the fact that the paper currently does not explore the types of information captured (what makes one model perform better than the other? how is this related to the purported functions of different regions?)\n\n        , why is this a realistic model of speech perception? I agree that the model uses far less training data than LMs (which, contrary to how it is stated in the paper, are self-supervised too) or supervised speech models. However, without a serious/exhaustive analysis on the types of speech tasks these models do well on, there ability to mimic human speech errors etc., is it fair to state that we are moving towards brain-like models? (more detailed arguments in Guest & Martin, 2021) For example, lines 259-261: “Here, we test whether self-supervised learning on a limited amount of speech suffices to **yield a model functionally equivalent to speech perception in the human brain.** “ \n\n- in lines 290-300, the authors discuss several studies on the functional specificity of different brain regions during speech processing, functional gradients in PFC etc.. However, how is this being inferred form the data? I may have missed the link but iiuc this is not supported by the evidence presented. \n- \"the functional hierarchy of its transformer layers aligns with the cortical hierarchy of speech in the brain, and **reveals the whole-brain organisation of speech processing with an unprecedented clarity**\" - Why is the layer wise selectivity important? Not to belabour the emphasis on what is captured by these representations, but without a discussion/analysis to reason why these differences might arise across regions, I am not sure what to make of these results or how they clarify the organization of speech processing across cortex. Furthermore, it appears that the upper-middle layers have high correlations in several regions across cortex, albeit not the highest. What do the authors make of this?\n- Please cite related work appropriately. The authors have missed several relevant citations from PIs like Jack Gallant, Leila Wehbe, Alex Huth, Tom Mitchell, Christopher Honey, Jonathan Brennan, Alona Fyshe and more. For example, papers that were the first to build language encoding models or present the style of analyses reported here are missing.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "IYqZ2Q_T__w", "reviewer": "Reviewer_1gQj", "summary": "This paper uses self-supervised (and supervised) models trained with speech or auditory scene data to predict fMRI activity in the auditory cortices of English-, French-, and Mandarin-speaking participants. It shows that this can be done at a level higher than some baseline models (e.g. random untrained models) and the models recapitulate some aspects of the neural organization such as hierarchical processing. ", "questions": "The authors claim 600 hours of data is comparable to the amount of auditory experience humans receive during their development. But 600 hours is only 25 days, which seems short compared to the developmentally relevant time scales for humans, so could you please elaborate on this comparison? The Discussion section mentions some tasks that can seemingly be done by humans with a few hundred hours of speech, but those tasks are very low level tasks which do not represent all facets of human speech perception. Relatedly, what would the performance (e.g. neural predictivity etc.) look like if you used pretrained self-supervised models trained on much more data, say, even more data than in a human lifetime (thus not necessarily restricting yourself to 600 hours).", "limitations": "I do have to note the correlations shown in Fig. 2B-C are very low (R). This may reflect the intrinsic SNR limitations of the fMRI signal. This makes me wonder if Fig. 1C is cherry-picked. A more representative example would be much better here (with an R similar to the average R). Relatedly, what is the noise ceiling performance in these recordings? That is, if I tried to predict the fMRI of one participant from the fMRIs of other participants, what would the performance look like?\n\nRe supervised vs self-supervised: I do have to note phoneme recognition is a very low-level task, perhaps a higher level classification task would give better results for the supervised model (which is already pretty close to the self-supervised one, to begin with).\n\nABX accuracy of the models is much higher than human accuracy, which suggests these models might not be behaviorally adequate models of human speech processing (and possibly of human auditory processing more generally).\n\nI understand that the differences in Fig 4C are significant, but they are tiny differences. Shouldn’t I interpret this as saying the non-native model, for example, is actually a pretty good model of the native auditory cortex? Whatever makes the auditory cortices of, say, Mandarin speakers different from the auditory cortices of French speakers doesn’t seem to be captured by these models (or perhaps more likely by fMRI itself).\n\nFinally, I also do have to note most of the results here (hierarchy, very slightly better prediction of native vs. non-native fMRI responses, etc.) are not really too surprising given what we know from earlier literature and prior expectations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe experiments and analyses were done competently and carefully with a large number of participants and the results seem to be sound.\n\nWeaknesses: please see the questions and limitations sections.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "7f3bddenKKb", "reviewer": "Reviewer_HY2z", "summary": "The paper provides a study to investigate how Wav2Vec 2.0, as a self-supervised method, learns brain like representation. Different sets of experiments were conducted to examine the cortical hierarchy of speech processing aligns its function. It also provides observation on brain's reaction to non-speech, non-native and native sounds compared to that of Wav2vec 2.0.", "questions": "- I believe the results are interesting to be shared, but I do not see any machine learning innovation in this paper. Basically the authors have used an already proposed self-supervised method to compare with the brain action. I believe this could be a better fit for some speech related journals. \n- R is some times used as Neural predictivity score and sometimes as Pearson correlation coefficient. Please make it clear through the paper.\n- R is some times very small, e.g. Figure 4 B. is small R still something to rely on and make conclusion for?\n- There are some related work, e.g. https://www.science.org/doi/abs/10.1126/science.1245994, that could be mentioned in the paper.\n- Although the results are promising, please make the conclusion linked to the current set up and do not strongly generalize it to all cases.", "limitations": "no limitation", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \nInteresting observations and an extensive set of results\nWeakness:\nNo machine learning innovation ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept