Improving Semantic Understanding in Speech Language Models via Brain-tuning
Authors: Omer Moussa
Dietrich Klakow
Mariya Toneva
Venue: ICLR 2025 Poster (2025)
Keywords: fMRI
Speech Models
Speech Recognition
Alignment
Brain Alignment
Cognitive Neuroscience
Encoding Models
Transformers
Primary area: applications to neuroscience & cognitive science
TL;DR: Fine-tuning speech models with brain recordings leads to increased semantic understanding and higher performance on downstream tasks.

Abstract: Speech language models align with human brain responses to natural language to an impressive degree. However, current models rely heavily on low-level speech features, indicating they lack brain-relevant semantics which limits their utility as model organisms of semantic processing in the brain. In this work, we address this limitation by inducing brain-relevant bias directly into the models via fine-tuning with fMRI recordings of people listening to natural stories--a process we name brain-tuning. After testing it on 3 different pretrained model families, we show that brain-tuning not only improves overall alignment with new brain recordings in semantic language regions, but also reduces the reliance on low-level speech features for this alignment. Excitingly, we further show that brain-tuning leads to 1) consistent improvements in performance on semantic downstream tasks and 2) a representational space with increased semantic preference. Our results provide converging evidence, for the first time, that incorporating brain signals into the training of language models improves the models’ semantic understanding. We make the code available at https://github.com/bridge-ai-neuro/brain-tuning.

Published as a conference paper at ICLR 2025
IMPROVING SEMANTIC UNDERSTANDING IN SPEECH LANGUAGE MODELS VIA BRAIN-TUNING
Omer Moussa!
Dietrich Klakow?
Mariya Toneva'
Max Planck Institute for Software Systems (omoussa, mt ontonva @oneaa]@mpi @mpi sws. org
Saarland University
dietrich. klakowelsv umi-saarland de
ABSTRACT
Speech language models align with human brain responses to natural language to impressive degree. However, current models rely heavily low-level speech features, indicating they lack brain-rec relevant semantics which limits their utility as model organisms of semantic processing in the brain. In this work, address this limitation by inducing brain-relevant bias directly into the models via fine-tuning with IMRI recordings of people listening to natural stories-a process we name brain-tuning. After testing it on 3 different pretrained model families, we show that brain-tuning not only improves overall alignment with new brain recordings in semantic language regions. but also reduces the reliance on low-level speech features for this alignment. Excitingly, we further show that brain- tunntn leads 1) consistent improvements in performance on semantic downstream tasks and 2) a representational space with increased semantic preference. Our results provide converging evidence, for the first time, that incorporating brain signals into the training language models improves the models' semantic understanding. We make the code available at http::/giiuub..om/brieee -neumrrrrieee---nnuoorraaaiii -tuning.
INTRODUCTION
It is an exciting time for the cognitive neuroscience of language with the rise of language models which have been shown align with (i.e. predict) brain activity evoked by natural language to impressive and unprecedented degrees (Wehbe et al. 2014 & Huth 2018 Toneva & Wehbe Schrimpf al. 2021 Caucheteux King 2022] Goldstein et al. | 2022 Karamolegkou al,,
2019
2023 Researchers aim to use language models as model organisms (Toneva 2021) of reading and listening in the brain to learn more about the underlying information processing that leads to brain-like representations of language.
However, recent work has questioned whether current popular speech language models can fully serve this role, their aligmment with semantic brain regions was shown to be mostly due to low- level speech features, indicating that speech language models lack brain-relevant semantics (Oota et al. 2024a). Given that most large brain recording datasets are of speech-evoked language (LeBel et al. 2023 Nastase et al. 202T Deniz et al.) 2019 Momenian et al.) (0024), having speech models with improved brain-relevan semantics is important provide better model organisms for auditory language processing. The lack of brain-relevant semantics in speech models (Oota et al.] 2024a) may also be related their incomplete downstream semantic understanding. (Choi al. 2024).
To bridge the gap between language understanding in speech models and the human brain, we pro- pose to augment pretrained speech model training directly with brain recordings in a process we call brain-tuning (see Fig la for illustration of the training approach). We then evaluate the resulting brain-tuned speech models in three distinct ways (see Fig Ic for an illustration of the evaluation ap- proach): 1) alignment with new brain recordings in semantic regions of the brain, which we expect to significantly increase brain-tuning successfully induces brain-relevant semantics, 2) effect of low-level features, such as Tri-Phones and Articulation, on the alignment with these semantic re- gions, which we expect to significantly decrease if brain-tuning successfully induces brain- relevant semantics 3) downstream performance tasks that are helped by semantic understanding, which expect to significantly improve if the brain-relevar semantic understanding induced by the brain- tuning is also useful for downstream semantic tasks.

---

Published as a conference paper at ICLR 2025
We brain-tune three popular speech language models using the largest available fMRI dataset, recorded when participants listened to natural stories. Across all models, we find that brain-tuning 1) significantly improves alignment with new fMRI recordings in semantic brain regions, 2) sig- nificantly reduces the impact of low-level features on this alignment, and 3) significantly improves downstream performance on tasks that helped by semantic understanding. We show that these results hold when comparing the brain-tunee models to their pretrained counterparts, and to two additional strong baselines (i.e. brain-tuning with block-permuted IMRI data, and fine-tuning using representations from larger speech model).
Our results provide converging evidence that augmenting speech models with brain signals from listening to natural language improves semantic understanding in speech models. Excitingly, our findings indicate for the first time that improving alignment with semantic understanding in the brain also translates to downstream gains for the models. We will make all models and code publicly available, and hope that the improved speech models our work provides will contribute to better understanding of listening in the brain.
Our main contributions can be summarized as follows:
1. We provide an approach to fine-tune pretrained speech models using fMRI recordings of people listening natural stories, and validate it across three popular model families.
2. We conduct extensive analyses to understand the impact of this fine-tuning on the speech model representations and behavior.
3. For the first time, we show that improving alignment with the brain has substantial and sigmificant downstream benefit for an Al model.
2 RELATED WORK
Our work is most closely related to that of Schwartz et al. (2019), who fine-tune one pretrained text- based language model (BERT (Devlin et al.][2099) using fMRI and MEG recordings of participants reading a chapter of a book. We instead focus on speech models, validate our method across three model families, and conduct comprehensive analyses to reveal that brain-tuning improves semantic understanding in speech language models for the first time. Separately, growing literature inves- tigates the alignment between human brains and pretrained language models. A number of studies have shown a degree of alignment between language-evoked brain activity with text-based language models (Wehbe et al.
2014 Jain & Huth 2018 Toneva & Wehbe 2019] Caucheteux & King. 2022 Jat al. 2019| Abdou al. [202] 1 Schrimpf et al. [222] Toneva al. 2022a b Antonello
ct al. 2021 Oota et al. 2022 Merlin & Toneva 2022 Aw & Toneva 2023
Oota et al. 2024b
Lamarre al. 2022 Antonello et al. 2024), and with speech-based language models (Millet al.,
2022
Vaidya et al.
2022] Tuckute et al.) 2023 Oota et al.) 20233 2024a Chen et al. 2024). Our
approach of traingtuning pretrained language models complementary and can be used in addition to previous methods for analyzing the alignment between language models and brain activity.
METHODS
3.1 SPEECH LANGUAGE MODELS
We build on three popular pretrained transformer-based speech language model families: Wav2vec2.0 (Baevski al. 2020). HuBERT (Hsu et al.) 2021). and Whisper (Radford al. 2023). We chose versions of these models that have comparable sizes (~90M parameters), the same num- ber of encoder layers (12), and the same embedding size (768). Wav2vec2.0 and HuBERT are self-supervised models that are trained to predict representations of masked portions of the input. They both divide the input into tokens of 20ms and then use a CNN feature extractor. We use the base architectures which are trained on ~96 hours of audio. Whisper, unlike Wav2Vec2.0 and Hu- BERT, is trained in a weakly supervised manner, using 680K hours of paired audio-text data and has an encoder-decoder architecture. Contrary to HuBERT and Wey2Vec2.0, Whisper takes fixed 30s input and then converts it to log-mel spectrograms. We fine-tune only the Whisper encoder for two reasons: to keep the model of comparable size to the other models, and 2) since the encoder
2

---

Published as a conference paper at ICLR 2025
TR-aligned
Feature Extractor
Transformer Layers
(MEI Reqporso
Audio
10l 10)h--lpp My
Pooling
FC Layer
Moeking
lpa1
lew3
Freeze
Fine-tune
wikrng
Produced BMRI Responsc
Proposed brain-tuning approach
Bruin
Low-level
Downstream
Get Representations
Estimate Brain Alignment
Alignment
Features Impact
Performance
Speeda Mecel
Expul
Fethire
Extracier
Transforme
Oripial slganicn:: BB)
Premmined
(B,- OC
Models
Lav Lend
Linood Nurationn of LaneLLeed Dosterr
redbaal
Houer
Bette
Beter
(b) Approach to estimate brain alignment and low- level feature impact
(c) Evaluation strategy and expected outcomes
Training and Evaluation Approaches. (a) Brain-tuning approach for a given speech model; (b) Evaluation of brain alignment and low-level feature impact on the brain alignment; (c) Types of evaluation and expected outcomes brain-tuning successfully improves semantic understanding in speech models: increase of alignment with semantic brain regions, decrease of impact of low-level features on this alignment, and increase downstream performance on semantic tasks.
Figure 1:
expected to represent lower-level information than the decoder, it is a good testbed for whether brain-tuning induce semantic understanding.
3.2 NATURALISTIC BRAIN DATASET AND DATA PREPROCESSING
We use the largest public dataset of fMRI recordings (LeBel al. 2024 for brain-tuning. The dataset contains fMRI recordings for 8 participants listening to 27 short stories from the Moth Radio Hour podcast for a total of 6.4 hours audio per participant (11, 543 fMRI images (TRs) with TR 2.0045s). To fine-tune model using fMRI recordings, fMRI recordings and the corresponding audio snippets that were presented to the participants. We follow previously proposed approaches for this (Oota al.) 2024a Vaidya et al. 2022 Antonello et al., [024]] Schwartz et al.) 2019). Specifically, we first partition the audio input by utilizing a sliding window length 7' seconds with stride W seconds. This way, at each time t the audio, a window of length jt - T" seconds is provided as input to the speech model. We use T 16s and W - 0.1s. We next align the stimulus presentation rate with the slower IMRI acquisition rate by downsampling using a 3-lobed Lanczos filter. Lastly, we account for the slowness of the fMRI hemodynamic response by modeling as a finite response filter with 10 seconds TRs). These steps result in an audio-fMRI paired dataset that can be used for brain-tuning or evaluation.
need to build paired dataset of
Estimated noise ceiling. Noise in fMRI data can impair brain-tuning and evaluation, so it is impor- tant to estimate the "noise ceiling" of each voxel in the fMRI recordings. We estimate the voxel-wise noise ceiling for all participants' [MRI data based on the preferred method by the original dataset pa- (LeBel et al. 2023). which leverages aithin-participant repetitions of the same story. This noise ceiling value estimates the amount explainable variance in the brain signal, ranging from 0 to 1. We this estimated noise ceiling to filter noisy voxels and to normalize the brain alignment during evaluation. We use a filtration threshold 0.4, in line with the findings of Antonello et al.] (2024). After filtering voxels with low noise ceiling, there remain 30, 000 50. 000 voxels per participant.
3

---

Published as a conference paper at ICLR 2025
The final brain-tuning voxel set contains voxels from late language regions and the auditory cortex. Note that because the late language regions are much larger than the auditory cortex, the number of included voxels from the late language regions is naturally much greater (as shown in Fig 13
3.3 BRAIN-TUNING SPEECH MODELS
Brain-tuning approach, Given an input audio and its corresponding fMRI response, obtained via the method in Section 3.2 aim to fine-tune pretrained speech model with the IMRI responses (i.e., brain-tune the model). Specifically, we fine-tune the model to reconstruct the fMRI responses corresponding to the voxels with high noise ceiling (> 0.4). The approach is illustrated in Fig la To this end, we add pooling layer and projection head on top of the output tokens. The projection head predicts the fMRI response from the pooled model tokens. More formally, given the Oy....N output tokens, we have function H, that predicts fMRI targets such that (o) o = FC(P(o) oy)), where P an average pooling function and FC a linear function. The training objective reconstruction loss (Le loss) between the outputs of H and the fMRI voxels. We freeze the feature extractor and backpropagate the loss to fine-tune the projection head and the transformer layers.
Training details. We used a base learning rate of x 10-5 and 10 respectively for the transformer layers and the linear projection head. Both had a linear decay scheduler for the learning rate with a warmup period for 10% of the epochs. The 27 fMRI stories are split into training set (24 stories), validation set (2 stories), and a held-out test set (1 story). The training is stopped when the validation loss saturates begins to diverge. Since the number of voxels differs for each participant, this fine-tuning process is done separately for each fMRI participant, We apply this approach to the 3 pretrained models: Wav2vec2.0, HuBERT, and the Whisper encoder.
3.3.1 COMPARISON MODELS
In addition to comparing the brain-tuned models to the corresponding pretrained ones, we further train several additional baselines for comparison. We briefly summarize these baselines and their purpose below, and provide more details about each baseline in Appendix D.2
Random brain-tuned. This baseline aims to test how the addition of any fMRI data impacts model performance. This baseline uses the same fine-tuning process in Fig la but instead of using the matched fMRI responses for the input stimulus, it uses flock-permuted iMRI responses.
Big spoken language model-tuned (BigSLM-tuned). This baseline tests the importance of having fMRI responses as the training targets. We replace the fMRI targets for the input stimuli with representations for the same stimuli obtained from BigSLM. We use Whisper Medium (800M parameters) as the BigSLM and use a concatenation of all its decoder layers' representations.
Stimulus-tuned. This baseline tests whether tuning with the fMRI signal results in additional gains over simply further tuning only using the stimulus audio. Stimulus-tuned models have been pre- viously found outperform pretrained models specifically for brain alignment (Merlin & Toneva 2022). but their performance on downstream tasks has not been investigated.
Text language model-tuned (L.M-tune)). We expect that current text LMs encode richer semantics than current speech LMs, so this baseline tests the importance of added semantics for model perfor- mance. For tuning, we use representations from two pretrained text LMs (GPT2 and LLama2). We leverage LM-tuned models to detect which downstream tasks benefit from more semantics.
In the main paper, we focus on two of these baselines-Random Brain-tuned and BigSLM-tuned and provide results from the remaining baselines in Appendix D.2 Briefly, stimulus-tuned models perform similarly to pretrained models and substantially worse than brain-tuned models on the tested downstream tasks. LM-tuned models improve over the pretrained models on two downstream tasks, the same ones where brain-tuning tuning leads to the biggest gains over the pretrained models. This further supports conclusions that brain-tuning improves semantic understanding in speech models.
3.4 EVALUATION
We evaluate multiple aspects of the brain-tuned models and illustrate our evaluation strategy in Tc. If brain-tuning successfully improves semantic understanding in speech models, expect
4

---

Published as a conference paper at ICLR 2025
that brain-tuner models will align better with semantic language regions in new brain recordings, have impact of lower low-level features on the alignment with these regions, and have improved downstream performance semantic tasks.
3.4.1 BRAIN ALIGNMENT
To compare brain alignment for a model before (i.e., the pretrained version) and after brain-tuning, compute the normalized brain alignment using standard voxel-wise encoding models and report it for language- and speech-related brain regions. For each region, we statistically test whether brain-tuning leads to significantly better alignment.
Normalized brain alignment. We estimate standard voxel-wise encoding models to evaluate the brain alignment of model representation (Antonello et al. 2024 Vaidya al., 2022] Oota et al. 2024a). We carry out this voxel-wise encoding as shown in the original alignment branch in Fig lb The audio data processed as detailed in Section 3.2 then a voxel-wise encoding function h is learned using ridge regression on the training portion of the dataset. The prediction performance of this encoding function is computed over the held-out testing portion of the dataset via Pearson correlation. For voxel v, we define Pe (the alignment for voxel v) as the Pearson correlation between the predictions of h and the corresponding brain responses for this voxel across all held-out data samples. Lastly, we define the normalized brain alignment B for a brain region of V voxels as:
1 Σ
B
NC, Pr
BEV
where NC,, is the noise ceiling for voxel v. This serves as a standardized measure for alignment between model and different brain regions since it computed relative to the estimated explainable variance in the brain region.
Parsing language and primary auditory regions. To make the normalized brain alignment com- parison focused on language and primary auditory regions, we use FreeSurfer v7 to project the participants' data, and then we use the human cerebral cortex parcellation atlas from (Glasser al.) 2016) to parse the regions of interest (ROIs). We focus mainly on the late language regions (e.g.. in- ferior frontal gyrus, angular gyrus, anterior and posterior temporal lobes, and middle frontal gyrus) and the primary auditory regions. The full ROI list and their functions is provided in Appendix A.
Significance testing. To test whether the brain- tunn- models have significantly different alignment than the pretrained ones, we use the Wilcoxon signed-rank test. We indicate significant differences (corresponding to p-value < 0.05) with an asterisk
3.4.2 IMPACT OF LOW-LEVEL FEATURES ON BRAIN ALIGNMENT
Previous work showed that the alignment of pretrained speech models with late language regions is mostly due to low-level features (Oota et al. 2024a), which is undesirable. We further set out to test the impact of low-level features on the brain- models' alignment with the brain. To enable comparisons with previous work, we estimate the low-level feature impact on brain alignment using the same approach as in Oota al. (2024)). Intuitively, the impact of a specific low-level feature is estimated by comparing the brain alignment of a model before and after this low-level feature is computationally removed from the model. If, after removal of the low-level feature, the alignment is significantly lower than the original one, the low-level feature is said to have high impact on the brain alignment. We illustrate this process in Fig Ib and provide details about this method below.
Low-level features. We focus on four low-level speech features: Power Spectrum (the time-varying power spectrum across frequency bands), Di-Phones Tri-Phones (adjacent pairs and triples of phonemes), and Articulation (articulatory characteristics of the phonemes). These features cover different stages of speech and are considered to be non-semantic features. The specifics of obtaining these features from the audio are detailed in (Oota et al. 2024a) and Appendix A.3]
Low-level feature impact. First, given a low-level feature of the input audio, a linear function F learns to predict the representations of the model from this feature. Then, the predicted model representations are subtracted from the true representations, and the brain alignment of this residual is estimated via a standard encoding model (Section 1). We define the low-level impact It as:
B,
B
R 100
(2)
B.
5

---

Published as a conference paper at ICLR 2025
where B, and B, correspond to the original and residual brain alignments. R represents the percent- age drop in alignment due to the removed low-level feature. Large means that much of the original alignment was due to the low-level feature. To test for significant differences between models, perform the same statistical tests as descrihed in Section 3.4.1
3.4.3 DOWNSTREAM TASKS
To test whether improving brain-relevant semanties via brain-tuning also improves semantic under- standing in models, we evaluate our models on a range of downstream tasks at different semantic levels. We also test the semantic vs. phonetic preference of the models' representations.
Downstream tasks. We choose tasks with several semantic difficulties, namely: automatic speech recognition (ASR), phonetic sentence type prediction, sequence understanding, phonemes predic- tion, word identity prediction, and emotion recognition. We use standard datasets for these tasks: TIMIT Garofolo 1993). Crema-D (Cao et al.] 2014). Speech Commands (Warden 2018 and SLURP (Bastianelli et al.] 2020). All datasets were not seen by the model during brain-tuning Appendix B details further information about the datasets and the formulations for each task. We consider emotion recognition to be the least semantic tone and prosodic information are highly predictive emotions in speech (Singh & Gupta 2023). Phonemes and word prediction are mod- erate in semantic difficulty, and the rest high in semantic difficulty as they require understanding beyond
decoding. Additionally, to have a more empirical guide on which task expect model with improved semantic understanding to perform better on, in Appendix D.2 we provide downstream results from the text LM-tuned baselines. We observe that text LM-tuned models benefit two tasks in particular: phonetic sentence type prediction and phonemes prediction, suggesting that these two tasks can benefit most from semantics.
Downstream evaluation. To perform downstream analysis, we add linear projection head f. where the input is the layer representation and the output is task-specific (e.g.. which phonemes were present, which word was present, etc). Each task performance is evaluated on held-out data, provided by each dataset, and an aggregate performance metric is reported. Except for ASR, all tasks use linear probes across layers and are evaluated using the Fi-score on a held-out test set. Since they are classification tasks, we also report additional Naive classifier as comparison baseline that predicts the majority class for the given task. For ASR, we fine-tune the whole transformer model, calculate the Word Error Rate (WER) on the held-out test set, and report (1 WER) as the performance accuracy metric.
Semantic-phonetic preference. Previous work has shown that speech models' representations
consistently more phonetic than semantic across all layers (Choi et al. 2024). They also show that in seemingly semantic task such Intent Classification (IC), the models rely on phonetic not semantic features to do the task. We further test the semantic-phie phonnttic preference of our models, using the same method as Choi al. (2024), which we detail in Appendix C Briefly, the method tests the representation distance between set of words, their phonetic neighbors (e.g. "divine" and "divide"), and their semantic neighbors (e.g. "divine" and "god"). A model for which phonetic neighbors are closer than semantic ones is said to have a phonetic preference.
RESULTS
4.1 BRAIN ALIGNMENT WITH HELDOUT DATA
We estimate the normalized brain aligmment described in Section[..4.1 separately for two important language-related areas of the brain: the late language regions and the primary auditory regions. The late language regions are thought to support semantic language processing, while the primary auditory regions support mostly lower-level processing related to the speech signal (Deniz et al. 2019). For each the three model families, we evaluate the normalized brain alignment for the pretrained and brain-tuned versions, along with the alignments of two main comparison baselines- BigSLM-tune and Random Brain-tuned (see Section 3.
In Fig and 2b show the normalized brain alignment averaged across voxels, layers, and par- ticipants for all models. We observe that muin-tuning significantly improves alignment with late language regions for the self-supervised models (Wav2vec2.0 and HuBERT), with increase of

---

Published as a conference paper at ICLR 2025
500 Pretrained
BigSLM-tuned
Brain-tuned
Sig different from pretrained
Normalised Brain Alignment
Language Regions
lormalised Brain Alignment
Primary Auditery Regions
0.35
0.35
0.30
0.30
0.20
0.20
0.15
0.15
0.05
0.05
Wav2Vec2.D
HuBERT
Whisper
Wav2Ver2.0
HUBERT
Whisper
(a) Normalized alignment for late language regions
(b) Normalized alignment for primary auditory
Difference in Brain Alignment (Brain tuned Pretrained)
ioom
(c) Difference in brain alignment due brain-tuning of Wav2vec2.0
Figure 2: (a), (b) Mean normalised brain alignment for different brain areas. Error bars indi- cate the standard across participants, with * indicating significantly different alignment from pretrained. Brain-tuming significantly improves alignment with late language regions for the self- supervised models. (c) Voxel-wise differences in brain alignment between brain-tuned and pre- trained Wav2vec2.0 for representative participant. Higher alignment is observed in semantic areas.
30% over the corresponding pretrained models. This gain in alignment with late language regions also be seen the level of individual voxels (Fig for Way2vec2.0 and one representative par- ticipant: the brain maps for the remaining participants are shown in Appendix [.2).. In contrast, the two comparison models-BigSI.. M-ttune and Random Brain-tuned (see Appendix Fig 6 for Random Brain-tuned results)-lead to lower brain alignment than the pretrained models. This suggests that the gain from the brain-tuned models is due to incorporating the correct fMRI signal that corresponds to the audio input. We do not observe significant gains for Whisper in the late language regions or for any of the model families in the primary auditory regions.
The result that brain-tuning improves the alignment of two of the pretrained models with semantic late language regions, and with less semantic regions, such as the primary auditory cortices, sug- gests that brain-tuning tuminn may improve the brain-relevant semantics in at least some speech language models. We test this further in the next sections.
4.2 EFFECT OF LOW-LEVEL FEATURES ON BRAIN ALIGNMENT
We further test the dependence lovelevel features of the observed gain in brain alignment due to train-tuning, Fi and b present the impact low-level features on the brain alignment across model families (averaged over voxels, layers, low-level features, and participants).

---

Published as a conference paper at ICLR 2025
500 Pretrained
BigSLM-tuned
Brain-tuned
Sig different from pretrained
Language Reglons
Primary Auditory Regions
Low-led lewel Impact
30
30
Im
Low-leve
15
15
10
10
Whisper
Wav2Vec2.00
Whisper
(a) Low-level impact in late language regions
(h) Low-level impact in primary auditory regions
impact (Brain
MFC
FOOD
IFGOR
(c) Difference in low-level impact due brain-tuning Wav2vec2.0
Figure 3: (a), (b) Mean impact of low-level speech features (percentage drop in brain alignment) for different regions. Error bars indicate the standard error of the mean across participants, and * denotes significantly lower low-level impact than in the pretrained model, All models have significantly lower low-level impact in late language regions. (c) Voxel-wise differences in low-level impact between brain-tuned and pretrained Wav2vec2.0 for representative participant.
We observe that brain-tuning reduces the impact of low-level features on alignment with late lan- guage regions across all model families, including Whisper, which did not originally show improve- ments in the total brain alignment in these regions (Fig(Zh). The comparison models once again do account for the improvements due brain-tuning- Similarly to before, for all models, brain- tuning leads to no significant changes related to the primary auditory regions. These observations also hold on the voxel-level (Fig for Wav2Vec2.0 and one representative participant; see Ap- pendix F33 for the remaining participants). Moreover, brain-tuning can also make up for scale: we smaller model (HuBERT small) is similarly aligned with late language
observe that a regions as a substantially larger pretrained model of the same family (HuBERT large), while having lower impact low-level features on this alignment (see Appendix A.2 for more details).
Overall, these results indicate that brain-tuning's gain in alignment with the late language areas is not merely in the magnitude but also in nature, as the reliance on low-level features for alignment with semantic regions is reduced. Next, we investigate if the induced brain-relevant semantics via brain-tuning also lead to improvement in semantic understanding the models.
4.3 DOWNSTREAM PERFORMANCE
We next investigate the models' semantic understanding via their performance on downstream tasks (detailed in Section 3.4.3, and their semantic-phonetic preference (detailed in Appendix C).
8

---

Published as a conference paper at ICLR 2025
Naive Classifier
Random Brain-tuned
Pretrained
BigSLM-tuned
Brain-tuned
HiSE4T
HHASKT
Wh
Wax2Ww2.
NURERT
opor
(a) Automatic Speech Recognition (ASR)
(b) Phonetic Sentence-type Predic- (c) Sequence Understanding
tion
SLURP
WaseWez..0
MURERI
Wh 0011
WaxdWeG20
HuzoHI
Wh spar
RusERI
Wh SP1
(d) Phonemes Prediction
Command Word Idt. Prediction
(1) Emotions prediction
Figure 4: Downstream task performance for different models. Brain-tuned models' performance is the mean and STE across participants. Brain-tuned models show consistent improvement over the baselines, with the biggest gains in more semantic tasks.
Performance downstream tasks. Fig presents the models' performances popular down- stream tasks that require different levels of semantic understanding. The brain-tuned performance averaged across layers and participants, except for ASR where the performance is only averaged across participants since evaluations are done only once per model. We observe that brain-tuning boosts the performance the pretrained models for all model families across all tested downstream tasks, with the exception of emotion prediction, in which the performance improves slightly for one model family and decreases slightly for the other two (Fig[4r). The brain-tuned models also generally outperform the BigSt.--tuned baselines, with the exception of the emotion prediction task. Most notably, brain-tuning leads to most consistent gains above the BigSL.--tuned and pretrained baselines for those tasks where fine-tuning with representations from text-based language models also helpful (Fig 10; Appendix D.2.. This provides additional evidence that brain-tuning most beneficial for tasks where improved semantic understanding is helpful.
General trends downstream performance. We observe few trends in all downstream tasks: (1) the BigSLM-tuned models perform par better than the pretrained ones, indicating that the BigSee-ttuned model is meaningful baseline. (2) Generally, the pretrained, brain-tuned, and BigSL.M-tuned models perform better than the Random Brain-tuned baselines, which indicates that permuting 1MRI targets does not lead to improvement on downstream tasks. (3) The models perform better than naive classifiers, so substantial gains in performance cannot be attributed to random behavior. (4) More interestingly, the brain-tuned Whisper encoder consistently and substantially outperforms pretrained and BigSLM-tunm versions, which often have close to random or naive performances. This shows that the substantial gain in performance is due to fine-tuning with the matched fMRI data.
Change in semantic-phonetic preference. Finally, when we compare the semantic-phoneti pref- erence for the brain-tuned vs. pretrained models (F) we find that the preference becomes more semantic in the late layers for the brain-tuned models. In contrast, the pretrained models show either no change or a decrease in semantic preferences. This indicates that the brain-tume tuned models have reduced phonetic preference in the later layers. We elaborate these findings in Appendix C
Excitingly, the consistent improvement in downstream semantic tasks and reduced phonetic prefer- ence in late layers in tandem with the improvement in brain alignment with late language regions and the reduced impact of low-level features on this brain alignment. These results provide converg- ing evidence for improved semantics in the brain-tuned models.

---

Published as a conference paper at ICLR 2025
5 DISCUSSION
In this work, we present a method to augment speech model training directly with fMRI recordings of people listening to natural stories and show two converging lines of evidence that this leads to improved semantic understanding in the models.
First, two of the three tested model families have improved alignment with new brain recordings in the semantic language regions after brain-tuning (F For all model families, brain-tuning also significantly reduces the impact of low-level speech features on alignment with late language regions 3). This is marked improvement over pretrained models, which were shown to almost entirely rely on low-level speech features to align with semantic brain regions (Oota et al. 2024a). Furthermore, brain-tuned models align with semantic language regions as well as larger pretrained models while also having a lower low-level feature impact (Appendix Fig[5). This suggests that brain-tuning has the potential to break the brain alignment plateau for speech models, which has been shown to reach saturation around 700M parameters (Antonello et al.] 2024). Brain-tuning larger models may break this saturation and lead to substantially more alignment in the semantic language regions, further increasing their utility as model organisms (Toneva 2021).
Second, the brain-tuned models most substantially improve performance on downstream semantic tasks while also maintaining performance on core speech low-level tasks (Fig 4 & Fig This pro- vides additional evidence that brain-tuning improves semantic understanding in speech models while not negatively impacting their fundamental speech capabilities. We attribute the brain-tuned models' balanced semantic and low-level understanding to the inclusion of voxels from both late language ar- eas and auditory cortex during brain-tuning. We expect that if brain-tuning only considered auditory cortex voxels, there would not be improvements in the model's semantic understanding. Conversely, if brain-tuning only considered late language areas voxels, the model's 's speech performance would substantially degrade. Future work can investigate these hypotheses.
Lastly, the brain-tuned models also improve their semantic preference in the late layers (Appendix Fig/8.. We further emphasize that the gain in performance and semantic preference is due to ad- ditional training data of less than 0.7% of the original training size, which is comparable to, if not less than, a typical fine-tuning dataset for a specific task. Yet, the brain-tuning case, there is an evident gain in performance and wider generalization across multiple downstream semantic tasks. There is also good evidence that expanding brain data and model families further will increase the gains from brain-tuning (Appendix F.2 & F.1 Overall, the extent of our knowledge, this is the first work to show substantial downstream performance gains when augmenting an MI. model with brain-relevant data, across not only the speech domain but also language and vision.
Both lines of evidence show that brain-tuning not only increases the raw performance metrics but also changes the representations of the model to rely less on low-level features when performing a task that relies on semantics. These findings also strengthen the utility of brain alignment in determining how semantically oriented a given model representation is. Lastly, all results signal that there is still room for improvement in the speech models' semantic capabilities, and we hope that our work inspires future work that improves on our brain-tuning method.
CONCLUSION
Our systematic analyses of the utility of brain-tunini on semantic brain alignment and downstream performance reveal a parallel among the gain in brain alignment, its reduced impact of low-level speech features, and increased downstream performance on several tasks with varying semantic difficulty. We further observe an increase in the semantic preference of late layers of the brain-tuned models. These exciting results provide evidence for the first time that incorporating brain signals into the training of language models improves their semantic understanding. Future work can investigate further refinement of the brain brain-tuning loss and the incorporation of additional participants and brain datasets in the train-tuning process. We hope that the brain- train-tuned models we provide will serve as better model organisms for auditory language processing in the brain, and will inspire more work on improving the alignment between language in machines and language in the brain.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
In this paper, we fully describe our proposed brain-tuning approach and how to evaluate it. (1) Sec- tions 3.1 3.2 and B.3 detail the exact model families, data processing, and fine-tuning settings and hyper-parameee parameters needed to carry out Brain-tuning for any given model. (2) Section Appendix Appendix C and Appendix [.2 detail all evaluation pipelines and metrics, alongside any addi- tional datasets training settings and
hyper-
needed for evaluation. (3) We will make code for brain-tuning and its evaluation publicly available once the paper is accepted.
ACKNOWLEDGMENTS
This work was partially funded by the German Research Foundation (DFG) - DFG Research Unit FOR 5368 and by the CS @ max planck graduate center.
REFERENCES
Mostafa Abdou, Ana Valeria González, Mariya Toneva, Daniel Hershcovich, and Anders Søgaard. Does injecting linguistic structure into language models lead to better alignment with brain record- ings? arXiv preprint arXiv:2101.12600 2021.
Richard Antonello, Javier S Turek, Vy Vo, and Alexander Huth. Low-dimensional structure in the space of language representations reflected brain responses. Advances in Neural Information Processing Systems, 34:8332 8344, 2021.
Richard Antonello, Aditya Vaidya, and Alexander Huth. Scaling laws for language encoding models in fmri. Advances in Neural Information Processing Systems, 36, 2024.
Khai Loong Aw and Mariya Toneva. Training language models to summarize narratives improves brain alignment. In The Eleventh International Conference on Learning Representations, 2023.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli, wav2vec 2.0: frame- work for self-supervised learning of speech representations. Advances in neural information processing systems, 33: 33:1244-112460, 2020.
Emanuele Bastianelli, Andrea Vanzo, Pawel Swietojanski, and Verena Rieser. SLURP: A Spoken Language Understanding Resource Package. In Proceedings the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020.
Houwei Cao, David G Cooper, Michael K Keutmann, Ruben C Gur, Ani Nenkova, and Ragini Verma. CREMA-D: Crowd-sourced emotional multimodal actors dataset. IEEE Trans. Affect.
Comput., 5(4)3377-390, October 2014.
Charlotte Caucheteux and Jean-Rémi King. Brains and algorithms partially converge in natural language processing, Communications Biology. 5(1):134, 2022.
Catherine Chen, Tom Dupré la Tour, Jack L Gallant, Daniel Klein, and Fatma Deniz. The cortical representation of language timescales is shared between reading and listening. Communications Biology, 7(1):884, 2024.
Kwanghee Choi, Ankita Pasad, Tomohiko Nakamura, Satoru Fukayama, Karen Liveseu, and Shinji Watanabe, Self-supervised speech representations are more phonetic than semantic, 2024. URL
htttps://arxiv/ wrw. abs 2406. 08619
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, al. Scaling instruction-finetueed lan- guage models. Journal of Machine Learning Research, 25(70):1-53, 2024.
Fatma Deniz, Anwar O Nunez-Elizalae Alexander G Huth, and Jack L Gallant. The representation of semantic information across human cerebral cortex during listening versus reading is invariant to stimulus modality. Journal of Neuroscience, 2019.

---

Published as a conference paper at ICLR 2025
Rutvik H Desai, Usha Tadimeti, and Nicholas Riccardi. Proper and common names in the semantic system. Brain Structure and Function, 228(1): 2549, 2023.
Jacob Devlin, Ming-We, Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. Proceedings of NAACL-HLT, pp. 4171- 4186, 2019.
John Garofolo. Timit acoustic phonetic continuous speech corpus. Linguistic Data Consortium, 1993, 1993.
Matthew F Glasser, Timothy Š Coalson, Emma Robinson, Carl Hacker, John Harwell, Essa Yacoub, Kamil Ugurbil, Jesper Andersson, Christian F Beckmann, Mark Jenkinson, et al. A multi-modal parcellation of humaan cerebral cortex. Nature, 536(7615):111-178, 2016.
Ariel Goldstein, Zaid Zada, Eliav Buchnik, Mariano Schain, Amy Price, Bobbi Aubrey, Samuel A Nastase, Amir Feder, Dotan Emanuel, Alon Cohen, et al. Shared computational principles for language processing in humans and deep language models. Nature Neuroscience, 25(3):66--380, 2022.
Xue L Gong, Alexander G Huth, Fatma Deniz, Keith Johnson, Jack L Gallant, and Frédéric E Theunissen. Phonemic segmentation narrative speech human cerebral cortex. Nature Com- munications, 14(1):4309, 2023.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451-3460, 2021.
Shailee Jain and Alexander Huth. Incorporating context into language encoding models for fmri. Advances in Neural Information Processing Systems, 31, 2018.
Sharmistha Jat, Hao Tang, Partha Talukdar, and Tom Mitchell. Relating simple sentence represen- tations in deep neural networks and the brain. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5137-5154, 2019.
Antonia Karamolegkou, Mostafa Abdou, and Anders Søgaard. Mapping brains with language mod- els: A survey. In Findings of the Association for Computational Linguistics: ACL 2023, pp- 9748- 9762, 2023.
Mathis Lamarre, Catherine Chen, and Fatma Deniz. Attention weights accurately predict lan- guage representations in the brain. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 4513-4529, 2022.
Amanda LeBel, Lauren Wagner, Shailee Jain, Aneesh Adhikari-Desai, Bhavin Gupta, Allyson Mor- genthal, Jerry Tang, Lixiang Xu, and Alexander G Huth. A natural language fmri dataset for voxelwise encoding models. Scientific Data, 10(1):555, 2023.
Amanda LeBel, Lauren Wagner, Shailee Jain, Aneesh Adhikari-Dessi Bhavin Gupta, Allyson Mor- genthal, Jerry Tang, Lixiang Xu, and Alexander G. Huth. "an fmri dataset during a passive natural language listening task". 2024. doi: doi: 181 1
Gabriele Merlin and Mariya Toneva. Language models and brain alignment: beyond word-level semantics and prediction. arXiv preprint arXiv:2212.0059% 2022.
Juliette Millet, Charlotte Caucheteux, Yves Boubenec, Alexandre Gramfort, Ewan Dunbar, Christophe Pallier, Jean-Remi King, et al. Toward a realistic model of speech processing in the brain with self-supervised learning. Advances in Neural Information Processing Systems, 35: 33428-33443, 2022.
Mohammad Momenian, Zhengwu Ma, Shuyi Wu, Chengcheng Wang, Jonathan Brennan, John Hale, Lars Meyer, and Jixing Li. Le petit prince hong kong (1pphk): Naturalistic finr and eeg data from older cantonese speakers. Scientific data, 11(1 J:992, 2024.
12

---

Published as a conference paper at ICLR 2025
Samuel A Nastase, Yun-Fei Liu, Hanna Hillman, Asich Zadbood, Liat Hasenfratz, Neggin Ke- shavarzian, Janice Chen, Christopher J Honey, Yaara Yeshurun, Mor Regev, et al. The "narra- tives" fmni dataset for evaluating models of naturalistic language comprehension. Scientific data, (1), 2021.
Subba Reddy Oota, Jashn Arora, Veeral Agarwal, Mounika Marreddy, Manish Gupta, and Bapi Surampudi. Neural language taskonomy: Which nlp tasks are the most predictive of fmri brain activity? In Proceedings the 2022 Conference the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2220-3237, 2022.
Subba Reddy Oota, Khushbu Pahwa, Mounika Marreddy, Manish Gupta, and Bapi S Raju. Neu- ral architecture of speech. In ICASSP 2023-2023 IEEE International Conference Acousties, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Subba Reddy Oota, Emin Çelik, Fatma Deniz, and Mariya Toneva. Speech language models lack important brain-relevan semantics. ACL, 2024a.
Subba Reddy Oota, Manish Gupta, and Mariya Toneva. Joint processing of linguistic properties in brains and language models. Advances in Neural Information Processing Systems, 36, 2024b.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.
Language models are unsupervised multitask leamner. OpenAI, 2019. URL https:
www.bettt.betet..t. / Language meeeeuaageemeeeeeeee 1sleess// sss/aanguageee models are
unsupervised wwwpsuulllittttssm laaaaassk_eaaaeeeeeeeeee learners. -df Accessed: 2024-11-15.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International Conference on Ma- chine Learning, pp. 18442-285111 PMLR, 2023.
Martin Schrimpf, Idan Asher Blank, Greta Tuckute, Carina Kauf, Eghbal A Hosseini, Nancy Kan- wisher, Joshua B Tenenbaum, and Evelina Fedorenko. The neural architecture of language: In- tegrative modeling converges on predictive processing. Proceedings of the National Academy of Sciences, 2021.
Dan Schwartz, Mariya Toneva, and Leila Wehbe. Inducing brain-relevant bias in natural language processing models. Advances in neural information processing systems, 32, 2019.
Anant Singh and Akshat Gupta. Decoding emotions: A comprehensive multilingual study of speech models for speech emotion recognition. arXiv preprint arXiv:2308.087. 2023.
Mariya Toneva. Bridging language in machines with language the brain. PhD thesis, Carnegie Mellon University. 2021.
Mariya Toneva and Leila Wehbe. Interpreting and improving natural-language processing (in ma- chines) with natural panguage-rroessingg (in the brain). Advances in Neural Information Process- ing Systems, 32, 2019.
Mariya Toneva, Tom M Mitchell, and Leila Wehbe. Combining computational controls with natural text reveals aspects of meaning composition. Nature Computational Science, 2(11):745-757, 2022a.
Mariya Toneva, Jennifer Williams, Anand Bollu, Christoph Dann, and Leila Wehbe. Same cause; different effects in the brain. Causal Learning and Reasoning, 2022b.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307. 009888 2023.
Greta Tuckute, Jenelle Feather, Dana Boebinger, and Josh H McDermott. Many but not all deep neural network audio models capture brain responses and exhibit correspondence between model stages and brain regions. Plos Biology, 21 21(2)xe3002366, 2023.
13

---

Published as a conference paper at ICLR 2025
Aditya R Vaidya, Shailee Jain, and Alexander Huth. Self- Self-supervised models of audio effectively explain human cortical responses to speech. In International Conference on Machine Learning, pp. 21927-2194 PMLR, 2022.
Pete Warden, Speech commands: A dataset for limited-vocabulary speech recognition, 2018.
Leila Wehbe, Brian Murphy, Partha Talukdar, Alona Fyshe, Aaditya Ramdas, and Tom Mitchell. Simultaneously uncovering the patterns of brain regions involved in different story reading sub- processes. PloS One, (11), 2014.
14

---

Published as a conference paper at ICLR 2025
A BRAIN ALIGNMENT
A.1 ROIS DETAILS
The human cerebral cortex multi-modal parcellation (Glasser Atlas) has 180 labeled ROIs per hemi- sphere(Glasser al. 2016). It has language regions that include the following labels: Angular gyrus (AG: PFm, PGs, PGi. TPOJ2, TPOJ3), lateral temporal cortex (LTC: STSda, STSva, STGa, TEIa, TE2a, TGv, TGd, A5, STSdp. STSvp. PSL, STV, TPOJ1), inferior frontal gyrus (IFG: 44, 45, IFJa, IFSp) and middle frontal gyrus (MFG: 55b) (QOota et al. 2024a). Desai et al. (2023)). It also has the primary auditory (A1) and the early auditory (A1, PBelt, MBelt, LBelt, RI, A4) regions.
Comparison with HuBERT Large in Late Language Regions
522
Pretrained Base
Pretrained Large
Brain-tuned Base
Normalised Brain Alignment
0.4
40
Low-level Impact
0.3
30
20
0.1
10
0.0
HUBERT
HuBERT
Figure 5: Brain alignment and low-level impact comparison with pretrained HuBERT large ar- chitecture on subjects. The HuBERT base architecture performs closely the large pretrained architecture and is less affected by removal of low-level features
A.2 BRAIN-TUNING ALIGNMENT COMPARISON WITH BIGGER MODELS
To get clearer idea about the improvement in brain alignment we get from Brain-tuning on late language regions, compare the HuBERT model's base architectures 90M parameters both pretrained and brain-tumed to the pretrained HuBERT large architecture (320M parameters) in Fig The pretrained large architecture has much larger aligmment in late language regions than the pretrained base model, which is in line with the trend of increase in alignment when scaling the model size shown (Chung et al. 2024); however, the Brain-tuned base model is very close to the pretrained large model's alignment. Moreover, the low-level impact of the Brain-tuned model is noticeably less than the pretrained large model. This an indicator that we can explain much more in the brain late language semantic regions with smaller models and brain-tuning bigger models might break the plateau we see in (Antonello et al.) (0224 when reaching huge model sizes. Breaking this plateau might allow for better and more accurate computational models for speech semantic processing in the brain.
A.3 LOW-LEVEL SPEECH FEATURES DETAILS
Here, we detail the definition and acquisition method of the low-level speech features used in our computation of low-level impact (namely Power Spectrum, Di-Phones Tri-Phones, and Articula- tion ). We obtain these features for the fMRI stories used for brain alignment computation (Section 3.4.Ip.
Power Spectrum. We follow the method described in (Gong et al. 2023). For each TR time (a 2s segment), we quantify the time-varying power spectrum across 448 frequency bands. This power spectrum is obtained by estimating the power of the sound signal between 25 Hz and 15 kHz, in 33.5 Hz bands (giving a total of 448 bands). For our experiments, we use the power spectrum features from (Deniz al. 2019)
15

---

Published as a conference paper at ICLR 2025
Di-Phones & Tri-Phones. For given speech utterance, DiPhones represent the adjacent pair of For
phones (e.g., [da], [al]). For each audio segment of length 2 seconds (TR length), we have a one-hot encoding vector representing the presence absence of all possible 858 diphones. The same is done for Tri-Phones but for triplets of phones instead of pairs. The annotations of this data were done using the Praat software and were obtained from the shared data by the authors of (Oota al., 2024ah
Articulation. Phoneme articulations are the specific ways speech organs (e.g., tongue or vocal cord) move to produce different phonemes. They associate each phoneme with set of properties that specify things like whether this phoneme makes vocal cords vibrate. We use phoneme artic- ulations as mid-level speech features, mapping hand-labeled phonemes to a set of 22 articulatory characteristics. For our experiments, we use the articulation annotations from (Deniz al. 2019)
A.4 BRAIN ALIGNMENT WITH RANDOM BRAIN-TUNED
We extend oiy showing the Random Brain-tuned baseline alongside the brain-tuned, pretrained, and BigSLM-tunee models. Figlobjclearly shows that Random Brain-tunee tuned models have the lowest brain alignment values in late language regions compared to even BigSL.Mtuuned baselines, and in the primary auditory regions, it's also much lower than the pretrained version. This strongly indi- cates that randomly permuting the fMRI targets while brain-tuning the models with the same stimuli substantially harms the model's alignment. Hence, having the correct fMRI targets essential for the alignment results we get from the brain-tuned models.
000 Pretrained
Random Brain-tuned
BigSLM-tunee
Brain-tuned
* Sig different from pratrained
Normalised Brain Alignment
Late Language Regions
formalisedd Brain Alignment
0.40
Primary Auditory Regions
0.35
0.35
0.30
0.30
0.25
0.25
0.20
0.20
0.15
0.05
0.05
0.00
Wav2Vec2.0
Whisper
0.00
Wav2Vec2.0
HuBERT
Whisper
(a) Normalised alignment for Late Language Regions
(b) Normalised alignment for Primary Auditory
Figure 6: (a), (b) Normalized brain alignment across participants and models for different ROIs, where means brain-tuned has alignment values that are statistically significantly different from pretrained. Random Brain-tuned baselines have much lower brain alignment than pretrained models in the late language and primary auditory areas.
A.5 LOW-LEVEL IMPACT WITH RANDOM BRAIN-TUNED
We extend Fig.3yy showing the Random Brain-tuned baseline alongside the brain-tuned, pretrained, and BigSLM-ttuned models. 7 clearly shows that Random Brain-tuned models have the highest low-level impact (highest drop) compared even BigSLM-uuned baselines in both late language regions and the primary auditory region. Even though the original normalized alignment values for the Random Brain-tuned models are very low to begin with, they still undergo a substantial drop after we remove the low-level features. Similar to the results from Fig these results also indicate that randomly permuting the fMRI targets while brain-tuning the models with the same stimuli substantially harms the model's 's semantics. Hence, having the correct fMRI targets is essential for reduced low-level impact we get in the brain-tuned models.
hhtps::/wwww.ffer fon hum. www..wwwwwwpp.aaaaa//////a/aaaat
16

---

Published as a conference paper at ICLR 2025
000
Brain-tuned
BigSLMrtuned
* Sig pretral
Late Language Regions
Primary Auditory Regions
50
50
Low-level Impact
Low-level Impact
40
40
30
30
20
20
10
10
Wav2Vec2.0
Whisper
Wav2Vec2.0
HuBERT
Whisper
(a) Low-level Impact in Late Language Regions
(b) Low-level Impact the Primary Auditory Region
Figure 7: (a), (b) Low-level Speech Features Impact (Percentage Drop in Brain Alignment) for different ROIs, where means brain-tuned has alignment values that are statistically significantly different from pretrained. RandomBrain-tume -utuee baselines have much higher low-level impact than pretrained models in that late language and primary auditory areas.
B DOWNSTREAM TASKS
We detail here the description and dataset used for each downstream task mentioned in Section
3.4.3
Automatic Speech Recognition (ASR). We run an ASR fine-tuning pipeline for the self- supervised models (Wav2vec2.0 and HuBERT). For ASR fine-tune the whole model to get the best possible performance. The rationale to essentially see if the brain-tunin tuinnt process will make harder require longer training to fine-tune the speech model for tasks like ASR. We of TIMIT (Garofolo) 1993) for ASR to carry out this experiment. We use the CTC loss (Baevski al. 2020) for both self-supervised models and calculate Word-Error-Rate (WER) the evaluation metric (on the padded test-set). We then report the ASR accuracy as WER.
version
Phonetic Sentence Type Prediction. The TIMIT dataset Garofolo (1993) has 3 different sentence types: SA (Dialect), SX (Compact), and SI (Diverse). The SA sentences supposed expose the dialectal variations of the speakers and are designed to span all the phonemes American English); the SX sentences should provide good coverage of phones (phonetically balanced and cover wide range of phonetic contexts with small number of words). The SI sentences are anything else (phonetically diverse and more naturalistic). We add classification head and use the Fl-score for evaluation.
Sequence Understanding. This is very similar to Intent Classification tasks; it tests the ability to understand sequence. We the SLURP (Bastianelli al. 2020) dataset that has audio paired with actions. For example, if the input is "Wake me up at eight o' o'clo"k", the action should be "set_alarm". We have 46 possible actions and add linear head predict the action and evaluate using the Fl-score.
Phonemes Prediction. Phoneme prediction is formulated as multi-labe classification problem, where the classifier predicts which of the 39 Phonemes present in the input audio clip. We use the TIMIT dataset Garofolo (993) for its phonetically rich sentences; we evaluate the test set using the F1-score.
Word Identity Prediction. We want to test the model's ability decode words from input audio. To simplify this task to befit a classification head, we convert it to classification task on the Speech Commands dataset (Warden 2018) which has audio clips, each of which has only one word belong- ing to a set of 35 commands. The classifier predicts which of the commands were said, and the evaluation is done using the F1-score.
17

---

Published as a conference paper at ICLR 2025
Emotion Recognition, We add classification head for emotion recognition the CREMA- dataset Cao et al. (2014); CREMA-D has 7.4K clips from 91 actors, and six different emotions (Anger, Disgust, Fear, Happy, Neutral, and Sad). The classifier predicts which of the 6 emotions is present and is evaluated using the F1-score.
C MEASURING SAMANTIC-PHONETIE PREFERENCE
To test there are semantic changes to the models' representations at a more fundamental level, try quantify how model prefers phonetic over semantic features and compare brain- tuned models to the pretrained ones. To be able to quantify this difference, we take inspiration from the work by Choi et al. (2024), which found that speech models have huge bias towards phonetic features. This was done by constructing phonetically similar pairs and semantically similar pairs and then computing their similarity/ distance in the embedding space. Doing this clearly shows that phonetically similar words are closer than semantically similar ones in the embedding space (i.e.. the model has strong phonetic preference across models. One aspect of desirable change in that behavior is to have the differences between phonetically and semantically similar pairs lower in the more semantic layers (both should still be better than random), or at least to have a clear hierarchy of that change. To do similar but more curated analysis, we build a dataset of 2K words, each of word paired with several semantically similar (e.g., Synonyms) and phonetically similar words (e.g.. Homophones). Then, we compute the representational distance between them, namely semantic distance for the distance of the word to the synonym and phonetic distance for the distance of the word to the homophone. After that, able to compute a semantic-poonetic preference dl for any given layer or model. We define as the average difference between semantic and phonetic distances. Essentially, since know phonetic distances are smaller from the work Cyoi et al. (2024). then when d decreases it means that the gap between semantic and phonetic decreases (semantic closer to phonetic) and vice versa. If d < O, means that the given layer is more semantic than it's phonetic. Thus, comparing the values of d for the brain- tunee models and the pretrained ones will tell us there is a difference in the phonetic preference across layers between them.
This behavior persists across layers and
Brain-tuned
Pretrained
BigSLM
Brain-tuned
Pretrained
BigSLM
Semantic- Semantic-Phonetie Phonetic Preference
0.30
Sommantic-Phonetiie Preference
0.30
0.25
0.25
0.20
0.20
0.15
0.15
0.10
0.10
0.05
0.05
2
6
10
12
2
8
10
12
Layer
Layer
(a) comantic-phoneee preference for Wec20ec2.0
(b) Semantic-phoneee preference for HuBERT
Figure 8: Semantic-phonetis preference (d) Comparison between brain-tuned and Pretrained mod- els. d decreases in late layers for brain-tuned models, suggesting that the phonetic preference for these layers is decreased.
Finally, when we investigate the semantic-phonetii preference (the delta of the distances between semantic and phonetic features) in the brain-tuned vs. pretrained models (Fi we find that d decreases in the late layers for the brain-tuned models, but either increases or stays the same for the pretrained ones. Our findings about the pretrained model replicate those of (Choi et al.) 2024). The findings about the brain-tuned models indicate that, unlike the pretrained models, semantic pairs in the late layers of these models closer the phonetic ones, and hence the layers phonetically dominated (i.e., the semantic preference increases and the phonetic vs semantic pref- erence decreases these layers). This, together with the downstream results from Fig indicates
less
18

---

Published as a conference paper at ICLR 2025
that the brain-tunee models are not phonetically biased (in late layers) and that they are capable of performing better on both phonetic and semantic tasks.
D ADDITIONAL TASKS AND BASELINES
In this section, we extend the evaluation of brain-tuned models to include two low-level tasks, namely MFCC and FBANK prediction. We also add language model baseline (comparison model) and report its downstream and low-level performance relative to to the other models (i.e., brain- tuned, Pretrained and BigSLM-tuned),
D.1 BRAIN-TUNED LOW-LEVEL PERFORMANCE
For MFCC prediction, we train linear probes to predict the MFCC coefficients, while for FBANK prediction, we train linear probes to predict the filter banks. For both tasks, the TIMIT dataset Garofolo 1993) and we evaluate using the R2 coefficient, averaged across layers.
Fig 9 reports pretrained, BigSLM-tuned, and brain-tuned models' performance two low-level tasks (MFCC prediction and FBANK prediction); shows that brain-tuned and BigSLMM-unned models don't 't perform substantially lower than their pretrained counterparts. This indicates that fine-tuned models are still capable doing these core low-level tasks just as well as their pretrained version. We think this evidence reduces the possibility that catastrophic forgetting happened after fine-tuning.
5.40
2.40
0.75
0.35
R-5quared Value
R-Squared Value
0.30
2.15
0.35
3.30
0:10
5.05
5.08
5.00
8.00
Wav2Vec2.0
HuBERT
Whisper
Wav2Vec2.0
HUBERT
Whisper
(a) MFCC prediction mean R² values
(b) FBANK prediction mean R² values
0.40
0.40
Broin ounes Wiwizeecz
0.35
Prstrsined Wov2Wec2.0
0.35
S-Squared Value
Pretriived HLRE"T
Squared Value
0.30
0.25
0.25
0.20
0.10
Brih ained War2rec2o
Introinee Wor2wcc20
Brainitured HUEERTT
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
Normalized Layer Depth
Normalized Layer Depth
(c) R2 values of MFCC prediction layer-wise
(d) R² values FBANK prediction layer-wise
Figure 9: Low-level task performance comparison for different models. It signifies that brain-tuned models didn't lose low-level capabilities at the expense of gaining more semantics, as the averaged R² values very close and the layer-wise values are the same in early layers and slightly decreased in late layers.
19

---

Published as a conference paper at ICLR 2025
D.2 DETAILS FOR COMPARISON BASELINES
We elaborate here the comparison models introduced in Section 3.3. and provide a complete con- trast for all comparison models on all downstream tasks for the Wav2Vec2.0 model family.
Random brain-tuned. In order to test how the addition of any fMRI data impacts model per- formance, this baseline uses the same fine-tuning process for brain-tuning (Section (33) but with incorrect randomly chosen IMRI targets. Instead of using the correct 1MRI responses for the input stimulus, we use plock-permuted IMRI responses that correspond to different stimuli. We carry out the fine-tuning process for each participant, and the training setup is identical to the brain-tuning ones (i.e., the same learning rates, learning rate schedule, optimizer, and number of epochs). The same set of voxels used in brain-tuning is also used for this baseline.
Big spoken language model-tuned (BigSI. fMRI responses as the fine-tuning targets with this baseline. To this end, we replace the fMRI target vectors for the input stimuli with representations for the same stimuli obtained from the BigSLM layers. This creates a proxy for the IMRI targets that is rich in meaningful spoken- pooken-langagge related information. For this baseline, we utilize Whisper Medium (800M parameters) as the BigSLM and use concatenation of all its decoder layers' representations. The fine-tuning objective is then to construct the concatenated representations vectors from the BigSLM layers (instead of reconstruct- ing the fMRT responses). We use the same training setup and the same stimulus data that were used for brain-tuning (Section
We aim to test the importance of having
Stimulus-tuned. This baseline tests whether tuning with the fMRI signal results in additional gains over simply further tuning using the stimulus audio only. This highlights any improvements in the model performance that would be only due to training with more data. To isolate the effect of the audio stimulus alone, we develop a model whose pre-training includes the audio stimulus used in the fMRI data (LeBel et al., 2024). To this end, for additional pre-training on the stimulus audio data, we use the same losses with the same weights of Wav2Vec2.0 self-supervised pre-training (the diversity loss and the contrastive loss) (Baevski et al. 2020). This dimulus-tuned model trained for 200 epochs and uses a base learning rate of 2 x updates and then a linear decay schedule.
with a warm-up for the first 10% of the
Text language model-tuned (LM-tuned). We develop two text language model-tuned baselines to test the potential of using a text language model (LM) in a similar fashion to the MigSLM-tuned baseline. The first one uses representations from the pretrained GPT2-Medium model (Radford) let al., 2019) which is similar in size to the BigSLM model, and the second one uses representa- tions from Llama2-7B Parameter Model (Touvron al. 2023) which is a much bigger and more recent large L.M. We apply a similar fine-tuning pipeline of brain-tuning with the concatenated rep- resentations of the corresponding LM as targets. For the GPT2-Medium model, we concatenate the representations of all layers, while for LLama2-7B one in every 3 layers is taken. We found that LM-tuned models need more training than the BigSLM-tuned one; we train them for 200 epochs. We report the downstream performance of these baselines (the GPT2-tuned and the Llama2-tuned -uuned) for the Wev2Vec2.0 model family.
The siimulus-tuned model performs very similarly to the pretrained model, indicating that further self-supervised training on these additional few hours of stimulus data doesn't impact downstream model performance (Fig [T). As for the LM-tuned baselines, GPT2-tuned and I. L.ama22 models improve over the pretrained and BigSLMM-uned ones on two tasks (namely Phonemes and Phonetic Sentence Type Prediction), with the Llama2-tuned being slightly better than the brain-tuned one on the same two tasks. The jump from pretrained to GPT2-tuned to Llama2-tuned indicates that more semanties lead to better performance tasks. These tasks are also the ones for which brain-tuning leads to the largest improvement over the pretrained models, further supporting tic understanding in speech models. The results for the Random Brain-tuned and BigSLM-tunm baselines are identical to Fig they mainly emphasize that Random Brain- tuniig hurts the model performance and BigSLM-tuning doesn't 't help strongly with the more semantic tasks.
Phonemes and Phonetic Sentence Type Prediction
conclusions that brain-tuning can improve seman-
20

---

Published as a conference paper at ICLR 2025
Random Brain-tuned
305 Pretrained
RigSLN-tuned
Xemulua-tuned
GPT2-tuned
Liamad-tunad
Brain-tuned
0.8
0.8
0.8
0.7
0.7
0.7
0.6
0.6
0.6
ASR Accurac
0.5
0.5
0.5
0.4
0.4
0.4
0.3
0.3
0.3
0.2
0.2
0.2
0.1
0.1
0.1
0.0
0.0
0.0
Speech Recognition
Phonetic Sentence-type Prediction
Seq. Understanding SLURP
0.8
0.8
0.8
0.7
0.7
0.7
0.6
0.6
0.6
11 Sccoe
0.5
E Score
0.5
0.5
0.4
0.4
0.4
0.3
0.3
0.3
0.2
0.2
0.2
0.1
0.1
0.1
0.0
0.0
0.0
Command Prediction
Figure 10: Downstream task performance of brain-tuned models vs. all baselines for Wav2 The additional baselines beyond those presented Fig are shown in blue hues. We observe that LM-tuned baselines (GPT2-tumed and Llama2-tuned) -unee) improve over the pretrained model on tasks where we see big improvements with brain brain-tuning, further supporting our conclusions that brain- tuning can improve semantic understanding in speech models. The stimulus-tuned model performs very similarly the pretrained one, indicating that simply further training on the audio stimulus not sufficient for the gains observed from brain-tuning.
E MORE STUDIES ON DOWNSTREAM TASKS
E. DOWNSTREAM PERFORMANCE COMPARISON WITH HUBERT LARGE
To complement our results on the brain alignment gap between HuBERT Large and the Brain-tuned Base version (Appendix A.2)). we report here the gap between them in downstream performance (reporting the same participants for brain-tuned models). IT shows that for HuBERT, the brain- tuned base architecture is closing on the pretrained large architecture. In other words, we gain performance close the large architecture with the same number of parameters of the base archi- tecture (after it's brain-tuned).
E.2 EFFECT OF MODEL AND DATA SIZES ON BRAIN-TUNING
We study here how brain-tuning is affected when we increase the pretrained model size or decrease the size of the data used for brain-tuning. We do this by plotting the performance improvement with brain-tuning (brain-tuning performance pretrained performance) across model sizes data sizes, for a range of downstream tasks. Fig 12 shows that as the model size increases, the gain in downstream tasks drops. When lower the amount of data
performance due to brain-tuning used for brain- brain-tuning tuning, the gain in performance due to brain-tuning also drops. This provides a clear signal that data size plays key role in the positive results
from brain-tuning.
21

---

Published as a conference paper at ICLR 2025
2000
Pretrained
000 Pretrained Large
Brain-tuned Base
0.8
0.8
0.6
0.6
0.6
ASR Accuracy
0.5
0.4
0.4
0.3
0.2
0.2
0.1
0.1
0.1
0.
AAtommttc Speech Recogrition (ASR)
Phonetic Sentence type Prediction
5eg Understanding on SLURP
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.4
17
17
0.2
0.2
0.2
0.0
Phoneme Prediction
Command Word Idt. Prediction
Emotion Prediction
Figure 11: Downstream performance comparison of pretrained HuBERT large architecture vs Brain- tuned HuBERT base architecture. The HuBERT base architecture performs closely to the large pretrained architecture, bridging the gap between the base and large model sizes.
Command
Phonamas
Sent. Type
20
% A. in Performance
10
90M
317M
10
90M
317M Model
10 90M
317M
18
Model
Model
Performance Improvement vs. Model Size HuBERT
Command nd Word Idt.
Phonemes Prediction
Phon. Type Prediction
Prediction
20
% In Performance
15
20
50
70
100
20
50
70
100
20
50
70
100
% Brain-tuning Data
% Brain tuning Data
% Brain-tuning Data
(b) Performance Improvement Data Size for Wav2Vec2.0
Figure 12: Performance Improvement (brain-tuned pretrained performance) across model sizes and data sizes. We clearly that lowering the amount of data limits the gain in performance due to brain- brain-tuning tuning, and the same goes for increasing the model size while keeping the data fixed.
22

---

Published as a conference paper at ICLR 2025
High NC Voxels for Subj 2
High Voxels Subj
Figure 13: Voxels with Noise Ceiling values above the threshold for Participants and 3. These the voxels that will be used for brain-tuning
F ADDITIONAL BRAIN PLOTS FOR DIFFERENT PARTICIPANTS
Here, we show samples of fiiierfileeeed voxels (the ones that will be used for brain tuning). More- over, we extend the whole-brain plots (flat and lateral views) for different participants for both brain alignment and low-level impact analyses.
F.1 HIGH NC FILTERED VOXELS
Fig shows the remaining voxels after applying the noise threshold mentioned in Section 3.2 for two participants. These the voxels that will be used during brain-tuning. Looking their locations, we see that they cover large number of semantic regions, as well as the auditory cortex.
F.2 BRAIN ALIGNMENT VOXEL-WISE DIFFERENCES
We show in Fig. more whole-brain analyses for the voxel-wise differences in brain alignment be- tween brain-tuned and pretrained models for different participants. The trend we detailed in section i. is also consistent with the plots below and is in line with the increase in the values alignment in late language regions (Fig 2a) and the insignificant change in alignment in primary auditory regions (Fig 2b). All shown Brain plots are for the Wav2vec2.0 model family.
F.3 LOW-LEVEL IMPACT VOXEL-WISE DIFFERENCES
In Fig 15 we show more whole-brain analyses for the voxel-wise differences in low-level impact due to brain-tuning (between brain-tuned and pretrained models) for different participants. The trend we detailed in section 4.2 is also generally consistent with the plots below and in line with the increase in the values of alignment in late language regions (Fig and the insignificant change in alignment in primary auditory regions (FigBb) All shown Brain plots are for the Wav2vec2.0 model family.
23

---

Published as a conference paper at ICLR 2025
Alghment tuned Pretrained!
(a) Difference in Brain Alignment for Participant
Offfrencce Brain de Pnotronnoll
(b) Difference in Brain Alignment for Participant 2
Oifference Brain Algg (Braintuned Pedd
(c) Difference in Brain Alignment for Participant 6
24

---

Published as a conference paper at ICLR 2025
Offference Brain (Bral-tuued nedd
(d) Difference in Brain Alignment for Participant
Dfffcrence brain Alggners Protra
(e) Difference in Brain Alignment for Participant 8
Figure 14: Difference in brain alignment performance (measured by Pearson correlation) between brain brain-tuned and pretrained Wav2Vec2.0 models for different participants. It shows better align- ment of the brain-tuned model semantic language areas.
25

---

Published as a conference paper at ICLR 2025
Impact tunai
Difference in low-level impact Participant
impact tunad
(b) Difference low-level impact for Participant impact tundi
(c) Difference in low-level impact for Participant 6
26

---

Published as a conference paper at ICLR 2025
impact 1Brain tunco
(d) Difference in low-level impact for Participant 7
Loaelezel impart Diffeence (ftraind Peetralned)
(c) Difference in low-level impact for Participant 8
Figure 15: Difference in low-level impact brain brain-tuned and pretrained Wav2Vec2.0 models for different participants. It shows a lower low-level impact (lower drop due to low-level feature removal) for the brain-tuned model in semantic language areas.
27


Reviews:
[{"review_id": "i9b3hlueJx", "reviewer": "Reviewer_x8dp", "summary": "- In short the main idea is to fine-tune existing speech models so they are better aligned with fMRI recordings and then show that this results in better, more semantic alignment with the brain, and better downstream performance on speech tasks.\n- The data comes from fMRI recordings that have been aligned to the podcast audio stimulus. For baselines, the authors run the off-the-shelf models and also fine-tune using fMRI signal unrelated to the task as well as audio representations from another model\n- Finally, the authors formalize a notion of low-level feature impact, i.e., how much of the alignment can be attributed to low level (non-semantic) audio features, and show that their brain-tuning results in an alignment that can be explained less by low-level features.", "questions": "- How is performance on the original speech task preserved even during brain-tuning? Do the authors have an explanation for why no catastrophic forgetting occurs, given that there is no incentive for the base model to retain it's original speech representations during brain-tuning? Is it just because the learning rate is very small? The brain-tuning set is very small?\n- The ASR performance of wav2vec2 seems low? The original wav2vec2 paper (https://arxiv.org/pdf/2006.11477) reports accuracies of >0.8 on Librispeech. What explains the difference?\n- I'm having a hard time following the reasoning behing the low-level feature impact results. If I understand correctly, models were brain-tuned. And then the low-level impact on the brain-tuned models was found to be smaller, compared to the random fMRI baseline. Why should this be the case? Shouldn't we expect the opposite based on the evidence cited on lines 237-238, which suggests that brain-tuning should increase alignment between the brain and the model, and therefore the reliance on low level features?\n\n## small stuff\n- Figure 1c: Are the bar charts supposed to have an x-axis label?\n- Line 257: `` gives open quotes in LaTeX", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The fact that brain-tuning yields improvements on automatic speech recognition is very noteworthy. I have some points that need clarification (see below) before I can endorse this conclusion, but if they are addressed, then this will increase my confidence in the significance of this work.\n- The motivation for this project is good. This work closes the loop on the large amount of existing work that has shown the emergent alignment between pretrained speech models and the brain. It appears novel in this respect.\n- The random fMRI and BigSLM baselines increase my confidence in the fact that the recorded brain activity is a useful target for alignment.", "weaknesses": "- The conclusion that brain-tuning yields improvements over the off-the-shelf models is incredible, to the degree that I feel I must be misinterpreting the conclusion. There are two things I want to be sure of before this conclusion can be accepted:\n  - First, is there any architectural enhancement that the brain-tuned version of wav2vec2 has that the original version does not? Extra projection layers?\n  - Second, are the performances of wav2vec2 and HuBERT being fairly represented? It seems puzzling to me that the reported off-the-shelf ASR performances are so low (see questions below). One thing that may be happening: there is a distribution difference between the Librispeech data that wav2vec2 is trained on and the podcast audio used in the experiment. During brain-tuning, the model gets a chance to fine-tune on the podcast audio, or a proxy for that audio (via the brain), and this explains the improved performance. The gold standard experiment for this claim would be to do brain-tuning over one dataset of fMRI-audio pairs, and then show improvement on a completely new set of audio. To this end, it would be good to report the model performance on their original tasks. \n- The fact that brain-tuning does not result in catastrophic forgetting of the original speech model representations is so incredible that I feel I must be missing something. The brain is a very noisy proxy for the audio signal, and using the brain as a target seems to only give very weak incentive to preserve the original speech representations (see questions). As mentioned above, it would be good to report model performance on their original eval datasets (ex. Librispeech) to make sure that any forgetting did not occurr.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZOiWVVFkxK", "reviewer": "Reviewer_BDuH", "summary": "The paper incorporates a new finetuning approach for speech encoder models using neural recordings from subjects listening to stories. The approach uses the fMRI responses as a target during finetuning and modifies transformer layers of the speech. The paper demonstrates three areas of general improvement: (1) better brain alignment, (2) reduced impact of low level features for brain alignment, and (3) improved downstream performance across certain tasks.", "questions": "* What areas of the brain were voxels covering for your fMRI target? Were semantic areas covered or other areas covered?\n* My familiarity with fMRI is low by why would you downsampling using a 3-lobed Lanczos filter? Just a bit confused on the need to do so, not anything wrong with the choice.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Novelty: The paper isn’t the first to propose brain-tuning (which has been proposed in the vision and language domains) but is the first to apply this approach to speech models which often lack semantic understanding.\n* Improvements in downstream performance: Most impressively is the finding that tuning with fMRI leads to strong improvements in ASR. While prior work has shown improvements from incorporating neural signals, I haven’t seen consistent improvements on downstream performance. \n* Reduced impact of low level features in brain alignment: Low level features become less impactful indicating fewer simple audio features are driving alignment to the brain. This is a beneficial and impactful finding. \n* Phonetic-Semantic Split: Following the prior strength, I liked Appendix C which quantified the phonetic-semantic preferences between finetuning and pretrained models.", "weaknesses": "* Baselines: In general, I think the current baselines are noteworthy but I think there are a few baselines which are missing. \n    * If the goal of the paper is to improve semantic understanding in speech models, the SLM baseline is a bit odd to me. Wouldn’t it be better to use a pretrained language model embedding as a target instead? I would recommend adding this baseline as another way to incorporate semantic information into the model. I also found a model embedding target to be a bit unfair in comparison to fitting voxels since the target is one-dimensional but any target from an embedding model is higher-dimensional according to my understanding, reducing the complexity of the fit.\n    * I also think it would be great to include a baseline that controls for particular voxels of fMRI recordings. For example, I was wondering whether the area of the brain had an effect on the outcome of the model.\n* Forgetting: One problematic aspect of finetuning is catastrophic forgetting where models can forget currently learned capabilities. I have this concern after seeing the performance decrease in your emotion prediction task. It would be very useful to quantify this point more formally to give a sense of whether semantic information is better incorporated at the sacrifice of basic phonetic understanding. This doesn’t seem to be reflected in appendix C as well. I would appreciate seeing 1-2 low level audio tasks benchmarked as well.\n* Neural Signal Details: Some details were missing. See questions.\n* (Nit) Consider adding [1] to your related work as another paper that tunes models based on neural information. \n\nOverall, I enjoyed this paper. I think adding a few additional baselines will make this paper much stronger. After seeing these, I will raise my score. \n\n[1] Dapello et. al. 2023 Aligning Model and Macaque Inferior Temporal Cortex Representations Improves Model-to-Human Behavioral Alignment and Adversarial Robustness. ICLR 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5bNykm57Bx", "reviewer": "Reviewer_SHAj", "summary": "This paper explores an innovative approach to enhance the semantic alignment of speech language models with the human brain by incorporating brain data. By fine-tuning models using fMRI data of participants listening to natural stories, The goal is to improve the models' semantic understanding by aligning them better with brain regions involved in language comprehension. The authors evaluate the method across three model families (Wav2vec2.0, HuBERT, Whisper) and show improvements in brain alignment, reduced reliance on low-level features, and better performance on downstream tasks that require semantic understanding. This idea is excitingly novel due to its cross-disciplinary implications between neuroscience and AI.", "questions": "1. Please provide more detailed explanation why so few comparison models are provided or add more comparison models in the paper. This will dramatically affect my rating, if no reasonable explanation were provided, I would highly question the effectiveness of the method and definitely downgrade my rating.\n2. If possible, answer the second weaknesses, this is relatively subtle.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. **Novelty**: The concept of \"brain-tuning\" is a novel way to integrate neuroscience data into AI training, bridging the gap between machine learning and brain sciences.\n2. **Comprehensive Evaluation**: The evaluation strategy is thorough, with alignment tests, low-level feature analysis, and performance on multiple downstream tasks.\n3. **Cross-disciplinary impact**: The paper offers insights valuable to both cognitive neuroscience and AI communities, particularly for understanding semantic processing in the brain and improving language models.\n4. **Significant Results**: The results are robust, showing improved alignment with brain regions and downstream task performance, indicating the model's improved semantic capabilities, **but only in their restricted comparison models.**", "weaknesses": "1. **Comparison models are too weak and restrictive for the experiment**: There are **only two** comparison models: BigSLM Fine-tuned and Random-fMRI, the latter is not very valid since using random-fMRI results will definitely deteriorate performance, so there is in fact **only one** valid compasion model BigSLM, which is not convincing at all! BTW, BigSLM sometimes degrade performance as Fig 4, it's not valid as well.  **To prove effectiveness of brain-tuning, the paper should better compare brain-tuning with other valid finetuning models, ont only pretrained models.**\n2. fMRI data, in some degree, is too hard to acquire. As the paper mentioned, the largest public dataset of fMRI recordings only cover 8 participants. This will limit the usability of the proposed method, but this won't downgrade my rating for this paper since I think the idea side is novel and useful enough.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4y2EwRMWH9", "reviewer": "Reviewer_61Sy", "summary": "The authors propose the concept of brain-tuning in this paper, which aims to augment pretrained speech model training directly with brain recordings. In this way, the authors claim to compensate for the shortcomings of previous methods that only use low-level speech features for brain and speech model alignment, and successfully incorporate speech models with improved brain-relevant semantics. Experiments on three popular pretrained transformer-based speech language model families in three distinct ways (alignment with new brain recordings in semantic regions of the brain, effect of low-level features, such as Tri-Phones and Articulation, on the alignment with these semantic regions, downstream performance on linguistic tasks that require semantic understanding) also highlight the significance of inducing brain-relevant semantics in brain and speech model alignment.", "questions": "1. Why is brain-tuning not effective for whisper? More analysis needs to be conducted. Is it caused by the encoder-decoder architecture?\n2. How is the cortical parcellation conducted? The LeBel dataset doesn't provide any standardized space for all the subjects if I remember correctly. Are the fMRI images from different subjects first projected to the same space?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper first proposes fine-tuning pretrained speech models with brain responses to help improve the alignment between computational models and human brain. The experimental results are also impressive and solid.\n2. The authors design several interesting experimental settings and baseline methods to help investigate the influence of brain-tuning.\n3. The motivation is clear enough. The paper is also very well written with many experimental details supported.", "weaknesses": "1. Despite the authors effort in illustrating the effect of brain-tuning, the experiments are still insufficient in the following two aspects. (1)  The authors choose LeBel's dataset which only contains 8 subjects for practice. I don't think the conclusion drawn from this dataset is convincing enough given the very few subjects. Instead I suggest trying the Narratives dataset [1] which contains hundreds of subjects in a passive storying hearing task for experiments (by the way this dataset is considered as the largest fMRI dataset in naturalistic language comprehension, not the LeBel's dataset as the authors claimed in line 54), as many previous work [2][3][4] did. (2) The authors fail to investigate the influence of model parameters in the brain-tuning process. I think such experiments are easy but important, and should be included in the paper.\n\n2. The methods of obtaining four low-level speech features need to be detailed in this paper (at least in Appendix), instead of just citing previous work.\n\n[1] The “Narratives” fMRI dataset for evaluating models of naturalistic language comprehension. Nature Scientific Data\n\n[2] Evidence of a predictive coding hierarchy in the human brain listening to speech. Nature Human Behavior\n\n[3] Disentangling syntax and semantics in the brain with deep networks. ICML\n\n[4] Deep language algorithms predict semantic comprehension from brain activity. Nature Scientific Reports", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)