Improving Semantic Understanding in Speech Language Models via Brain-tuning

## Abstract

Abstract Speech language models align with human brain responses to natural language to an impressive degree. However, current models rely heavily on low-level speech features, indicating they lack brain-relevant semantics which limits their utility as model organisms of semantic processing in the brain. In this work, we address this limitation by inducing brain-relevant bias directly into the models via fine-tuning with fMRI recordings of people listening to natural stories–a process we name brain-tuning . After testing it on 3 different pretrained model families, we show that brain-tuning not only improves overall alignment with new brain recordings in semantic language regions, but also reduces the reliance on low-level speech features for this alignment. Excitingly, we further show that brain-tuning leads to 1) consistent improvements in performance on a range of downstream tasks and 2) a representational space with increased semantic preference. Our results provide converging evidence, for the first time, that incorporating brain signals into the training of language models improves the models’ semantic understanding.

## 1 Introduction

It is an exciting time for the cognitive neuroscience of language with the rise of language models which have been shown to align with (e.g. predict) brain activity evoked by natural language to impressive and unprecedented degrees (Wehbe et al., ; Jain & Huth, ; Toneva & Wehbe, ; Schrimpf et al., ; Caucheteux & King, ; Goldstein et al., ; Karamolegkou et al., ). Researchers aim to use language models as model organisms (Toneva, ) of reading and listening in the brain to learn more about the underlying information processing that leads to brain-like representations of language.

However, recent work has questioned whether current popular speech language models can serve this role fully, as their alignment with semantic brain regions was shown to be mostly due to low-level speech features, indicating that speech language models lack brain-relevant semantics (Oota et al., ). Given that most large public brain recordings datasets are of speech-evoked language (LeBel et al., ; Nastase et al., ; Deniz et al., ; Momenian et al., ), having access to speech models with improved brain-relevant semantics is important and will provide better model organisms for auditory language processing. The lack of brain-relevant semantics in speech models (Oota et al., ) may also be related to their incomplete semantic understanding for downstream language tasks (Choi et al., ).

To bridge the gap between language understanding in speech models and the human brain, we propose to augment pretrained speech model training directly with brain recordings in a process we call brain-tuning (see Fig. 1(a) for illustration of the training approach). We then evaluate the resulting brain-tuned speech models in three distinct ways (see Fig. 1(c) for an illustration of the evaluation approach):
1) alignment with new brain recordings in semantic regions of the brain, which we expect to significantly increase if brain-tuning successfully induces brain-relevant semantics,
2) effect of low-level features, such as Tri-Phones and Articulation, on the alignment with these semantic regions, which we expect to significantly decrease if brain-tuning successfully induces brain-relevant semantics
3) downstream performance on linguistic tasks that require semantic understanding, which we expect to significantly improve if the brain-relevant semantic understanding induced by the brain-tuning is also useful for downstream tasks.

We brain-tune three popular speech language models using the largest available fMRI dataset, recorded when participants listened to natural stories. Across all models, we find that brain-tuning 1) significantly improves alignment with new fMRI recordings in semantic brain regions, 2) significantly reduces the impact of low-level features on this alignment, and 3) significantly improves downstream performance on 5 tasks that are thought to require semantic understanding. We show that these results hold when comparing the brain-tuned models to their pretrained counterparts, and to two additional strong baselines (i.e. brain-tuning with block-permuted fMRI data, and fine-tuning using representations from a larger speech model).

Our results provide converging evidence that augmenting speech models with brain signals from listening to natural language improves semantic understanding in speech models. Excitingly, our findings indicate for the first time that improving alignment with semantic understanding in the brain also translates to downstream gains for the models. We will make all models and code publicly available, and hope that the improved speech models our work provides will contribute to a better understanding of listening in the brain.

Our main contributions can be summarized as follows:

- 1.
We provide an approach to fine-tune pretrained speech models using fMRI recordings of people listening to natural stories, and validate it across three popular model families.
- 2.
We conduct extensive analyses to understand the impact of this fine-tuning on the speech model representations and behavior.
- 3.
For the first time, we show that improving alignment with the brain has a substantial and significant downstream benefit for an AI model.

## 2 Related Work

Our work is most closely related to that of Schwartz et al. (), who fine-tune one pretrained text-based language model (BERT (Devlin et al., )) using fMRI and MEG recordings of participants reading a chapter of a book. We instead focus on speech models, validate our method across three model families, and conduct comprehensive analyses to reveal that brain-tuning improves semantic understanding in speech language models for the first time. Separately, a growing literature investigates the alignment between human brains and pretrained language models. A number of studies have shown a degree of alignment between language-evoked brain activity with text-based language models  (Wehbe et al., ; Jain & Huth, ; Toneva & Wehbe, ; Caucheteux & King, ; Jat et al., ; Abdou et al., ; Schrimpf et al., ; Toneva et al., ; ; Antonello et al., ; Oota et al., ; Merlin & Toneva, ; Aw & Toneva, ; Oota et al., ; Lamarre et al., ; Antonello et al., ), and with speech-based language models (Millet et al., ; Vaidya et al., ; Tuckute et al., ; Oota et al., ; ; Chen et al., ). Our approach of brain-tuning pretrained language models is complementary and can be used in addition to previously proposed techniques for analysis of the alignment between language models and brain activity.

## 3 Methods

Figure: (a) Proposed brain-tuning approach
Refer to caption: x1.png

### 3.1 Speech Language Models

We build on three popular pretrained transformer-based speech language model families: Wav2vec2.0 (Baevski et al., ), HuBERT (Hsu et al., ), and Whisper (Radford et al., ). We chose versions of these models that have comparable sizes ($\sim$90M parameters), the same number of encoder layers (12), and the same embedding size (768). Wav2vec2.0 and HuBERT are self-supervised models that are trained to predict representations of masked portions of the input. They both divide the input into tokens of 20ms and then use a CNN feature extractor.
We use the base architectures which are trained on $\sim$960 hours of audio. Whisper, unlike Wav2Vec2.0 and HuBERT, is trained in a weakly supervised manner, using 680K hours of paired audio-text data and has an encoder-decoder architecture. Contrary to HuBERT and Wav2Vec2.0, Whisper takes a fixed 30s input and then converts it to log-mel spectrograms. We fine-tune only the Whisper encoder for two reasons: 1) to keep the model of comparable size to the other two models, and 2) since the encoder is expected to represent lower-level information than the decoder, it is a good testbed for whether brain-tuning can induce semantic understanding.

### 3.2 Naturalistic Brain Dataset and Data Preprocessing

We use the largest public dataset of fMRI recordings (LeBel et al., ) for brain-tuning. The dataset contains fMRI recordings for 8 participants listening to 27 short stories from the Moth Radio Hour podcast for a total of 6.4 hours of audio per participant ($11,543$ fMRI images (TRs) with $\text{TR}=2.0045$s).
To fine-tune a model using fMRI recordings, we need to build a paired dataset of fMRI recordings and the corresponding audio snippets that were presented to the participants. We follow previously proposed approaches for this (Oota et al., ; Vaidya et al., ; Antonello et al., ; Schwartz et al., ). Specifically, we first partition the audio input by utilizing a sliding window of length $T$ seconds with a stride $W$ seconds. This way, at each time $t$ in the audio, a window of length $[t-T,t]$ seconds is provided as input to the speech model. We use $T=16\text{s}$ and $W=0.1\text{s}$. We next align the stimulus presentation rate with the slower fMRI acquisition rate by downsampling using a 3-lobed Lanczos filter.
Lastly, we account for the slowness of the fMRI hemodynamic response by modeling it as a finite response filter with 10 seconds (5 TRs). These steps result in a audio-fMRI paired dataset that can be used for brain-tuning or evaluation.

Estimated noise ceiling. Noise in the fMRI data can impair the brain-tuning and evaluation, so it is important to estimate the ”noise ceiling” of each voxel and participant in the fMRI recordings. We estimate the voxel-wise noise ceiling for all participants’ fMRI data based on the preferred method by the original dataset paper (LeBel et al., ), which leverages within-participant repetitions of the same story. This noise ceiling value estimates the amount of explainable variance (the maximum variance that can be explained by a given model) and its values range from 0 0 to $1$. We use this estimated noise ceiling to filter noisy voxels and to normalize the brain alignment during evaluation. We chose the threshold for filtering as $0.4$, in line with the findings of Antonello et al. (). After filtering voxels with low noise ceiling, there remain $30,000$ to $50,000$ voxels per participant.

### 3.3 Brain-tuning Speech Models

Brain-tuning approach. Given an input audio and its corresponding fMRI response, obtained via the method in Section 3.2, we aim to fine-tune a pretrained speech model with the fMRI responses (i.e., brain-tune the model). Specifically, we fine-tune the model to reconstruct the fMRI responses corresponding to the voxels with high noise ceiling ($>0.4$). The approach is illustrated in Figure 1(a). To this end, we add a pooling layer and a projection head on top of the output tokens. The projection head predicts the fMRI response from the pooled model tokens. More formally, given the $o_{1}....o_{N}$ output tokens, we have a function $\mathbf{H}$, that predicts fMRI targets such that $\mathbf{H}(o_{1}:o_{N})=\mathit{FC}(P(o_{1}:o_{N}))$, where $P$ is an average pooling function and $\mathit{FC}$ is a linear function. The training objective is a reconstruction loss ($L_{2}$ loss) between the outputs of $\mathbf{H}$ and the fMRI voxels. We freeze the feature extractor and backpropagate the loss to fine-tune the projection head as well as the transformer layers.

Training details. We used a base learning rates of $5\times 10^{-5}$ and $10^{-4}$ respectively for the transformer layers and the linear projection head. Both had a linear decay scheduler for the learning rate with a warmup period for $10\%$ of the epochs. The 27 fMRI stories are split into a training set (24 stories), a validation set (2 stories), and a held-out test set (1 story). The training is stopped when the validation loss saturates or begins to diverge. Since the number of voxels differs for each participant, this fine-tuning process is done separately for each fMRI participant. We apply this approach to the 3 pretrained models: Wav2vec2.0, HuBERT, and the Whisper encoder.

Additional comparison models. In addition to comparing the brain-tuned models to the pretrained ones, we further train $2$ baselines for comparison: 1) a Random-fMRI baseline to test how addition of any fMRI data impacts model performance. This baseline uses the same fine-tuning process in Fig. 1(a) but, instead of using the matched fMRI responses for the input stimulus, it uses block-permuted fMRI responses. 2) a Big Spoken-Language Model (BigSLM) baseline to test the importance of having fMRI responses as the reconstruction targets. This baseline replaces the fMRI targets for the input stimuli with representations for the same stimuli obtained from a BigSLM. We use Whisper Medium (800M parameters) as the BigSLM and use a concatenation of its internal representations from all layers of decoder. Both baselines create a proxy for the true fMRI targets, one with random fMRI signals that don’t correspond to the input and the other with rich representations from a large pretrained state-of-the-art speech model. Comparing brain-tuned models with these baselines helps understand changes in performance due to simple regularization and additional data. Any improvements over these baselines will thus be due the correct fMRI targets.

### 3.4 Evaluation

We evaluate multiple aspects of the brain-tuned models and illustrate our evaluation strategy in Fig. 1(c). If brain-tuning successfully improves semantic understanding in speech models, we expect that brain-tuned models will align better with semantic language regions in new brain recordings, have impact of lower low-level features on the alignment with these regions, and have improved downstream performance on semantic tasks.

#### 3.4.1 Brain Alignment

To compare brain alignment for a model before (i.e., the pretrained version) and after brain-tuning, we compute the normalized brain alignment using standard voxel-wise encoding models and report it for language- and speech-related brain regions. For each region, we statistically test whether brain-tuning leads to significantly better alignment.

Normalized brain alignment.
We estimate standard voxel-wise encoding models to evaluate the brain alignment of a model representation (Antonello et al., ; Vaidya et al., ; Oota et al., ). We carry out this voxel-wise encoding as shown in the original alignment branch in Figure 1(b). The audio data is processed as detailed in Section 3.2, then a voxel-wise encoding function $\mathbf{h}$ is learned using ridge regression on the training portion of the dataset. The prediction performance of this encoding function is computed over the held-out testing portion of the dataset via Pearson correlation. For a voxel $v$, we define $\rho_{v}$ (the alignment for voxel $v$) as the Pearson correlation between the predictions of $\mathbf{h}$ and the corresponding brain responses for this voxel across all held-out data samples. Lastly, we define the normalized brain alignment $B$ for a brain region of $V$ voxels as:

$$ $B=\frac{1}{|V|}\sum_{v\in V}\frac{1}{\textit{NC}_{v}}\rho_{v}$ (1) $$

where $\textit{NC}_{v}$ is the noise ceiling for voxel $v$.
This serves as a standardized measure for alignment between a model and different brain regions since it is computed relative to the estimated explainable variance in the brain region.

Parsing language and primary auditory regions. To make the normalized brain alignment comparison focused on language and primary auditory regions, we use FreeSurfer v7 to project the participants’ data, and then we use the human
cerebral cortex parcellation atlas from (Glasser et al., ) to parse the regions of interest (ROIs). We focus mainly on the late language regions (e.g., inferior frontal gyrus, angular gyrus, anterior and posterior temporal lobes, and middle frontal gyrus) and the primary auditory regions. The full ROI list and their functions is provided in Appendix A.1.

Significance testing. To test whether the brain-tuned models have significantly different alignment than the pretrained ones, we use the Wilcoxon signed-rank test. We indicate significant differences (corresponding to p-value $<0.05$) with an asterisk *.

#### 3.4.2 Impact of Low-level Features on Brain Alignment

Previous work showed that the alignment of pretrained speech models with late language regions is mostly due to low-level features (Oota et al., ), which is undesirable. We further set out to test the impact of low-level features on the brain-tuned models’ alignment with the brain. To enable comparisons with previous work, we estimate the low-level feature impact on brain alignment using the same approach as in Oota et al. (). Intuitively, the impact of a specific low-level feature is estimated by comparing the brain alignment of a model before and after this low-level feature is computationally removed from the model. If, after removal of the low-level feature, the alignment is significantly lower than the original one, the low-level feature is said to have high impact on the brain alignment. We illustrate this process in Fig. 1(b) and provide details about this method below.

Low-level features. We focus on four low-level speech features: Power Spectrum (the time-varying power spectrum across frequency bands), Di-Phones & Tri-Phones (adjacent pairs and triples of phonemes), and Articulation (articulatory characteristics of the phonemes). These features cover different stages of speech and are considered to be non-semantic features. The specifics of obtaining these features from the audio are detailed in (Oota et al., )

Low-level feature impact.
First, for a given low-level feature of the input audio, a linear function $\mathbf{F}$ is learned using this feature as input to predict the representations of the model. Then, the predicted model representations are subtracted from the true model representations, and the brain alignment of this residual is estimated via a standard encoding model, as described in Sec. 3.4.1. We define the low-level impact $R$ as:

$$ $R=100\cdot\frac{B_{o}-B_{r}}{B_{o}}$ (2) $$

where $B_{o}$ and $B_{r}$ correspond to the original and residual brain alignments. $R$ represents the percentage drop in alignment due to the removed low-level feature. Large $R$ means that much of the original alignment was due to the low-level feature. To test for significant differences between models, we perform the same statistical tests as described in Sec. 3.4.1.

#### 3.4.3 Downstream Tasks

To test whether improving brain-relevant semantics via brain-tuning also improves semantic understanding in models, we evaluate our models on a range of downstream tasks at different semantic levels. We also test the semantic vs. phonetic preference of the models’ representations.

Downstream tasks. We choose tasks with several semantic difficulties, namely: automatic speech recognition (ASR), phonetic sentence type prediction, sequence understanding, phonemes prediction, word identity prediction, and emotion recognition. We use standard datasets for these tasks: TIMIT (Garofolo, ), Crema-D (Cao et al., ), Speech Commands (Warden, ), and SLURP (Bastianelli et al., ). We consider emotion recognition to be the least semantic as tone and prosodic information are highly predictive of emotions in speech (Singh & Gupta, ). Phonemes and word prediction are moderate in semantic difficulty, and the rest are high in semantic difficulty as they require understanding beyond single-word/phone decoding. All datasets were not seen by the model during brain-tuning. Appendix B details further information about the datasets and the formulations for each task.

Downstream evaluation. To perform downstream analysis, we add a linear projection head $\bm{\mathit{f}}$, where the input is the layer representation and the output is task-specific (e.g., which phonemes were present, which word was present, etc).
Each task performance is evaluated on held-out data, provided by each dataset, and an aggregate performance metric is reported. Except for ASR, all tasks use linear probes across layers and are evaluated using the F1-score on a held-out test set. Since they are classification tasks, we also report an additional Naive classifier as a comparison baseline that predicts the majority class for the given task. For ASR, we fine-tune the whole transformer model, calculate the Word Error Rate (WER) on the held-out test set, and report $(1-\textit{WER})$ as the performance accuracy metric.

Semantic-phonetic preference. Previous work has shown that speech models’ representations are consistently more phonetic than semantic across all layers (Choi et al., ). They also show that even in a seemingly semantic task such as Intent Classification (IC), the models rely on phonetic not semantic features to do the task. We further test the semantic-phonetic preference of our models, using the same method as Choi et al. (), which we detail in Appendix C. Briefly, the method tests the representation distance between a set of words, their phonetic neighbors (e.g. ”divine” and ”divide”), and their semantic neighbors (e.g. ”divine” and ”god”). A model for which phonetic neighbors are closer than semantic ones is said to have a phonetic preference.

## 4 Results

Figure: (a) Normalized alignment for late language regions
Refer to caption: x4.png

### 4.1 Brain Alignment with Heldout Data

We estimate the normalized brain alignment described in Section 3.4.1 separately for two important language-related areas of the brain: the late language regions and the primary auditory regions. The late language regions are thought to support semantic language processing, while the primary auditory regions support mostly lower-level processing related to the speech signal (Deniz et al., ). For each of the three model families, we evaluate the normalized brain alignment for the pretrained and brain-tuned versions, along with the alignments of the two comparison baselines–BigSLM fine-tuned and Random-fMRI (see Section 3.3).

In Fig. 2a and b, we show the normalized brain alignment averaged across voxels, layers, and participants for all models. We observe that brain-tuning significantly improves alignment with late language regions for the self-supervised models (Wav2vec2.0 and HuBERT), with an increase of 30% over the corresponding pretrained models. This gain in alignment with late language regions can also be seen on the level of individual voxels (Fig. 2c for Wav2vec2.0 and one representative participant; the brain maps for the remaining participants are shown in Appendix D.1). In contrast, the two comparison models–BigSLM Fine-tuned and Random-fMRI (see Appendix Fig. 6 for Random-fMRI results)–lead to lower brain alignment than the pretrained models. This suggests that the gain from the brain-tuned models is due to incorporating the correct fMRI signal that corresponds to the audio input. We do not observe significant gains for Whisper in the late language regions or for any of the model families in the primary auditory regions.

The result that brain-tuning improves the alignment of two of the pretrained models with semantic late language regions, and not with less semantic regions, such as the primary auditory cortices, suggests that brain-tuning may improve the brain-relevant semantics in at least some speech language models. We test this further in the next sections.

Figure: (a) Low-level impact in late language regions
Refer to caption: x4.png

### 4.2 Effect of Low-level Features on Brain Alignment

We further test the dependence on low-level features of the observed gain in brain alignment due to brain-tuning. Fig. 3a and b present the impact of low-level features on the brain alignment across model families (averaged over voxels, layers, low-level features, and participants).

We observe that brain-tuning reduces the impact of low-level features on alignment with late language regions across all model families, including Whisper, which did not originally show improvements in the total brain alignment in these regions (Fig. 2a). The comparison models once again do not account for the improvements due to brain-tuning. Similarly to before, for all models, brain-tuning leads to no significant changes related to the primary auditory regions. These observations also hold on the voxel-level (Fig. 3(c) for Wav2Vec2.0 and one representative participant; see Appendix D.2 for the remaining participants)
). Moreover, brain-tuning can also make up for scale: we observe that a brain-tuned smaller model (HuBERT small) is similarly aligned with late language regions as a substantially larger pretrained model of the same family (HuBERT large), while having lower impact of low-level features on this alignment (see Appendix A.2 for more details).

Overall, these results indicate that brain-tuning’s gain in alignment with the late language areas is not merely in the magnitude but also in its nature, as the reliance on low-level features for alignment with semantic regions is reduced. Next, we investigate if the induced brain-relevant semantics via brain-tuning also lead to improvement in semantic understanding of the models.

Figure: (a) Automatic speech recognition (ASR)
Refer to caption: x11.png

### 4.3 Downstream Performance

We next investigate the models’ semantic understanding via their performance on downstream tasks (detailed in Section 3.4.3), and their semantic-phonetic preference (detailed in Appendix C).

##### Performance on downstream tasks.

Fig. 4 presents the models’ performances on popular downstream tasks that require different levels of semantic understanding. The brain-tuned performance is averaged across layers and participants, with the exception of ASR where the performance is only averaged across participants since evaluations are done only once per model. We observe that brain-tuning boosts the performance of all model families across all downstream tasks with varying semantic difficulty, with the exception of emotion prediction, in which the performance improves slightly for one model family and decreases slightly for the other two (Fig. 4(f)). Specifically, brain-tuning leads to the biggest gains for tasks that are most semantically challenging and require recognition and understanding beyond the local context. Brain-tuning increases the ASR accuracy on English ASR by up to 12% (Fig. 4(a)), enhances the phonetic sentence type understanding by 20-23% (Fig. 4(b)), and nearly doubles the performance on sequence understanding for the self-supervised model families (Fig. 4(c)). Tasks that relate to local word or phoneme understanding see moderate gains (6% gain in phonemes prediction across model families (Fig. 4(d)), and 3% gain in word identity prediction for the self-supervised models and 17% for Whisper (Fig. 4(e))).

General trends in downstream performance. We observe a few trends in all downstream tasks: (1) the BigSLM fine-tuned models are the same or worse than the pretrained ones, except for ASR, for which BigSLM offers a noticeable improvement over pretraining. (2) Generally, the pretrained, brain-tuned, and BigSLM fine-tuned models perform better than the Random-fMRI baselines, which indicates that permuting fMRI targets does not lead to improvement on downstream tasks. (3) The models perform better than naive classifiers, so substantial gains in performance cannot be attributed to random behavior. (4) More interestingly, the brain-tuned Whisper encoder consistently and substantially outperforms its pretrained and BigSLM-finetuned versions, which often have close to random or naive performances. This shows that the substantial gain in performance is due to fine-tuning with the matched fMRI data.

Change in semantic-phonetic preference.
Finally, when we compare the semantic-phonetic preference for the brain-tuned vs. pretrained models (Appendix Fig. 8), we find that the preference becomes more semantic in the late layers for the brain-tuned models. In contrast, the pretrained models show either no change or a decrease in semantic preferences. This indicates that the brain-tuned models have reduced phonetic preference in the later layers. We elaborate on these findings in Appendix C.

Excitingly, the consistent improvement in downstream tasks and reduced phonetic preference in late layers is in tandem with the improvement in brain alignment with late language regions and the reduced impact of low-level features on this brain alignment. These results provide converging evidence for improved semantics in the brain-tuned models.

## 5 Discussion

In this work, we present a method to augment speech model training directly with fMRI recordings of people listening to natural stories and show two converging lines of evidence that this leads to improved semantic understanding in the models.

First, two of the three tested model families have improved alignment with new brain recordings in the semantic language regions after brain-tuning (Fig. 2). For all model families, brain-tuning also significantly reduces the impact of low-level speech features on alignment with late language regions (Fig. 3). This is a marked improvement on pretrained models, which were shown to almost entirely rely on low-level speech features to align with semantic brain regions Oota et al. (). Furthermore, brain-tuned models align with semantic language regions as well as larger pretrained models while also having a lower low-level feature impact (Appendix Fig. 5). This suggests that brain-tuning has the potential to break the brain alignment plateau for speech models, which has been shown to reach saturation around 700M parameters (Antonello et al., ). Brain-tuning larger models may break this saturation and lead to substantially more alignment in the semantic language regions, further increasing their utility as model organisms (Toneva, ).

Second, the brain-tuned models generally improve in performance on a range of downstream tasks with different semantic difficulties (Fig. 4). This shows that brain-tuned models have improved general capabilities and that the fine-tuning process did not degrade the performance or make the models gain evident capabilities in one task but not the others. Moreover, specifically Whisper gains substantially from brain-tuning, improving from close to naive/random F1 scores to significant enhancement across the board. Lastly, the brain-tuned models also improve their semantic preference in the late layers (Appendix Fig. 8). We further emphasize that the gain in performance and semantic preference is due to additional training data of less than $0.7\%$ of the original training size, which is comparable to, if not less than, a typical fine-tuning dataset for a specific task. Yet, in the brain-tuning case, there is an evident gain in performance and a wider generalization across multiple downstream semantic tasks. Overall, to the extent of our knowledge, this is the first work to show substantial downstream performance gains when augmenting an ML model with brain-relevant data, across not only the speech domain but also language and vision.

Both lines of evidence show that brain-tuning not only increases the raw performance metrics but also changes the representations of the model to rely less on low-level features when performing a task that relies on semantics. These findings also strengthen the utility of brain alignment in determining how semantically oriented a given model representation is. Lastly, all results signal that there is still room for improvement in the speech models’ semantic capabilities, and we hope that our work inspires future work that improves on our brain-tuning method.

## 6 Conclusion

Our systematic analyses of the utility of brain-tuning on semantic brain alignment and downstream performance reveal a parallel among the gain in brain alignment, its reduced impact of low-level speech features, and increased downstream performance on several tasks with varying semantic difficulty. We further observe an increase in the semantic preference of late layers of the brain-tuned models. These exciting results provide evidence for the first time that incorporating brain signals into the training of language models improves their semantic understanding. Future work can investigate further refinement of the brain-tuning loss and the incorporation of additional participants and brain datasets in the brain-tuning process. We hope that the brain-tuned models we provide will serve as better model organisms for auditory language processing in the brain, and will inspire more work on improving the alignment between language in machines and language in the brain.

Reproducibility Statement. In this paper, we fully describe our proposed Brain-tuning approach and how to evaluate it. (1) Sections 3.1, 3.2, and 3.3 detail the exact model families, data processing, and fine-tuning settings and hyper-parameters needed to carry out Brain-tuning for any given model. (2) Section 3.4, Appendix B, and Appendix C detail all evaluation pipelines and metrics, alongside any additional datasets or training settings and hyper-parameters needed for evaluation. (3) We will make the code for brain-tuning and its evaluation publically available once the paper is accepted.

## Appendix A Brain Alignment

### A.1 ROIs Details

The human cerebral cortex multi-modal parcellation (Glasser Atlas) has 180 labeled ROIs per hemisphere(Glasser et al., ). It has language regions that include the following labels: Angular gyrus (AG: PFm, PGs,
PGi, TPOJ2, TPOJ3), lateral temporal cortex (LTC:
STSda, STSva, STGa, TE1a, TE2a, TGv, TGd, A5,
STSdp, STSvp, PSL, STV, TPOJ1), inferior frontal
gyrus (IFG: 44, 45, IFJa, IFSp) and middle frontal
gyrus (MFG: 55b) ((Oota et al., ), Desai et al. ()). It also has the primary auditory (A1) and the early auditory (A1, PBelt, MBelt, LBelt, RI, A4) regions.

Figure: Figure 5: Brain alignment and low-level impact comparison with pretrained HuBERT large architecture on 3 subjects. The HuBERT base architecture performs closely to the large pretrained architecture and is less affected by removal of low-level features
Refer to caption: x18.png

### A.2 Brain-tuning Alignment comparison with bigger models

To get a clearer idea about the improvement in brain alignment we get from Brain-tuning on late language regions, we compare the HuBERT model’s base architectures ( 90M parameters ) both pretrained and brain-tuned to the pretrained HuBERT large architecture (320M parameters) in Fig. 5. The pretrained large architecture has a much larger alignment in late language regions than the pretrained base model, which is in line with the trend of increase in alignment when scaling the model size shown in (Chung et al., ); however, the Brain-tuned base model is very close to the pretrained large model’s alignment. Moreover, the low-level impact of the Brain-tuned model is noticeably less than the pretrained large model. This is an indicator that we can explain much more in the brain late language semantic regions with smaller models and brain-tuning bigger models might break the plateau we see in (Antonello et al., ) when reaching huge model sizes. Breaking this plateau might allow for better and more accurate computational models for speech semantic processing in the brain.

### A.3 Brain Alignment With Random-fMRI

We extend Fig. 6 by showing the Random-fMRI baseline alongside the brain-tuned, pretrained, and BigSLM fine-tuned models. Fig. 6(b) clearly shows that Random-fMRI models have the lowest brain alignment values in late language regions compared to even BigSLM fine-tuned baselines, and in the primary auditory regions, it’s also much lower than the pretrained version. This strongly indicates that randomly permuting the fMRI targets while brain-tuning the models with the same stimuli substantially harms the model’s alignment. Hence, having the correct fMRI targets is essential for the alignment results we get from the brain-tuned models.

Figure: (a) Normalised alignment for Late Language Regions
Refer to caption: x19.png

### A.4 Low-level Impact With Random-fMRI

We extend Fig. 3 by showing the Random-fMRI baseline alongside the brain-tuned, pretrained, and BigSLM fine-tuned models. Fig. 7 clearly shows that Random-fMRI models have the highest low-level impact (highest drop) compared to even BigSLM fine-tuned baselines in both late language regions and the primary auditory region. Even though the original normalized alignment values for the Random-fMRI models are very low to begin with, they still undergo a substantial drop after we remove the low-level features. Similar to the results from Fig. 6, these results also indicate that randomly permuting the fMRI targets while brain-tuning the models with the same stimuli substantially harms the model’s semantics. Hence, having the correct fMRI targets is essential for reduced low-level impact we get in the brain-tuned models.

Figure: (a) Low-level Impact in Late Language Regions
Refer to caption: x19.png

## Appendix B Downstream Tasks

We detail here the description and dataset used for each downstream task mentioned in Section

3.4.3.

##### Automatic Speech Recognition (ASR).

We run an ASR fine-tuning pipeline for the self-supervised models (Wav2vec2.0 and HuBERT). For ASR we fine-tune the whole model to get the best possible performance. The rationale is to essentially see if the brain-tuning process will make it harder or require longer training to fine-tune the speech model for tasks like ASR. We use a version of TIMIT (Garofolo, ) for ASR to carry out this experiment. We use the CTC loss (Baevski et al., ) for both self-supervised models and calculate Word-Error-Rate (WER) as the evaluation metric (on the padded test-set). We then report the ASR accuracy as $1-\textit{WER}$.

##### Phonetic Sentence Type Prediction.

The TIMIT dataset Garofolo () has 3 different sentence types: SA (Dialect), SX (Compact), and SI (Diverse). The SA sentences are supposed to expose the dialectal variations of the speakers ( and are designed to span all the phonemes of American English); the SX sentences should provide good coverage of phones (phonetically balanced and cover a wide range of phonetic contexts with a small number of words). The SI sentences are anything else (phonetically diverse and more naturalistic). We add a classification head and we use the F1-score for evaluation.

##### Sequence Understanding.

This is very similar to Intent Classification tasks; it tests the ability to understand a sequence. We use the SLURP (Bastianelli et al., ) dataset that has audio paired with actions. For example, if the input is ”Wake me up at eight o’clock”, the action should be ”set_alarm”. We have 46 possible actions and we add a linear head to predict the action and evaluate using the F1-score.

##### Phonemes Prediction.

Phoneme prediction is formulated as a multi-label classification problem, where the classifier predicts which of the 39 Phonemes were present in the input audio clip. We use the TIMIT dataset Garofolo () for its phonetically rich sentences; we evaluate the test set using the F1-score.

##### Word Identity Prediction.

We want to test the model’s ability to decode words from input audio. To simplify this task to befit a classification head, we convert it to a classification task on the Speech Commands dataset (Warden, ) which has audio clips, each of which has only one word belonging to a set of 35 commands. The classifier predicts which of the 35 commands were said, and the evaluation is done using the F1-score.

##### Emotion Recognition.

We add a classification head for emotion recognition on the CREMA-D dataset Cao et al. (); CREMA-D has 7.4K clips from 91 actors, and six different emotions (Anger, Disgust, Fear, Happy, Neutral, and Sad). The classifier predicts which of the 6 emotions is present and is evaluated using the F1-score.

## Appendix C Measuring Semantic-Phonetic preference

To test if there are semantic changes to the models’ representations at a more fundamental level, we try to quantify how a model prefers phonetic over semantic features and compare our brain-tuned models to the pretrained ones. To be able to quantify this difference, we take inspiration from the work by Choi et al. (), which found that speech models have a huge bias towards phonetic features. This was done by constructing phonetically similar pairs and semantically similar pairs and then computing their similarity/ distance in the embedding space. Doing this clearly shows that phonetically similar words are closer than semantically similar ones in the embedding space (i.e., the model has strong phonetic preference ). This behavior persists across layers and across models. One aspect of desirable change in that behavior is to have the differences between phonetically and semantically similar pairs lower in the more semantic layers (both should still be better than random), or at least to have a clear hierarchy of that change. To do a similar but more curated analysis, we build a dataset of 2K words, each of word is paired with several semantically similar (e.g., Synonyms) and phonetically similar words (e.g., Homophones). Then, we compute the representational distance between them, namely semantic distance for the distance of the word to the synonym and phonetic distance for the distance of the word to the homophone. After that, we are able to compute a semantic-phonetic preference $\bm{d}$ for any given layer or model. We define $\bm{d}$ as the average difference between semantic and phonetic distances. Essentially, since we know phonetic distances are smaller from the work by Choi et al. (), then when $d$ decreases it means that the gap between semantic and phonetic decreases (semantic is closer to phonetic) and vice versa. If $d<0$, it means that the given layer is more semantic than it’s phonetic. Thus, comparing the values of $d$ for the brain-tuned models and the pretrained ones will tell us if there is a difference in the phonetic preference across layers between them.

Figure: (a) Semantic-phonetic preference for Wav2Vec2.0
Refer to caption: x24.png

Finally, when we investigate the semantic-phonetic preference $d$ (the delta of the distances between semantic and phonetic features) in the brain-tuned vs. pretrained models (Fig. 8), we find that $d$ decreases in the late layers for the brain-tuned models, but it either increases or stays the same for the pretrained ones. Our findings about the pretrained model replicate those of Choi et al. (). The findings about the brain-tuned models indicate that, unlike the pretrained models, semantic pairs in the late layers of these models are closer to the phonetic ones, and hence the layers are less phonetically dominated (i.e., the semantic preference increases and the phonetic vs semantic preference decreases in these layers). This together with the downstream results from Fig. 4 indicate that the brain-tuned models are not phonetically biased (in late layers) and that they are capable of performing better on both phonetic and semantic tasks.

## Appendix D Brain Plots for Different Participants

Here, we extend the whole-brain plots (flat and lateral views) for different participants for both brain alignment and low-level impact analyses.

### D.1 Brain Alignment Voxel-wise differences

We show in Fig. 9 more whole-brain analyses for the voxel-wise differences in brain alignment between brain-tuned and pretrained models for different participants. The trend we detailed in section 4.1 is also consistent with the plots below and is in line with the increase in the values of alignment in late language regions (Fig. 2(a)) and the insignificant change in alignment in primary auditory regions (Fig. 2(b)). All shown Brain plots are for the Wav2vec2.0 model family.

Figure: (a) Difference in Brain Alignment for Participant 1
Refer to caption: x26.png

Figure: (a) Difference in Brain Alignment for Participant 6
Refer to caption: x28.png

### D.2 Low-level Impact Voxel-wise differences

In Fig. 10, we show more whole-brain analyses for the voxel-wise differences in low-level impact due to brain-tuning (between brain-tuned and pretrained models) for different participants. The trend we detailed in section 4.2 is also generally consistent with the plots below and is in line with the increase in the values of alignment in late language regions (Fig. 3(a)) and the insignificant change in alignment in primary auditory regions (Fig. 3(b)). All shown Brain plots are for the Wav2vec2.0 model family.

Figure: (a) Difference in low-level impact for Participant 1
Refer to caption: x31.png

Figure: (a) Difference in low-level impact for Participant 7
Refer to caption: x34.png

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: - How is performance on the original speech task preserved even during brain-tuning? Do the authors have an explanation for why no catastrophic forgetting occurs, given that there is no incentive for the base model to retain it's original speech representations during brain-tuning? Is it just because the learning rate is very small? The brain-tuning set is very small?
- The ASR performance of wav2vec2 seems low? The original wav2vec2 paper (https://arxiv.org/pdf/2006.11477) reports accuracies of >0.8 on Librispeech. What explains the difference?
- I'm having a hard time following the reasoning behing the low-level feature impact results. If I understand correctly, models were brain-tuned. And then the low-level impact on the brain-tuned models was found to be smaller, compared to the random fMRI baseline. Why should this be the case? Shouldn't we expect the opposite based on the evidence cited on lines 237-238, which suggests that brain-tuning should increase alignment between the brain and the model, and therefore the reliance on low level features?

## small stuff
- Figure 1c: Are the bar charts supposed to have an x-axis label?
- Line 257: `` gives open quotes in LaTeX
soundness: 3
strengths: - The fact that brain-tuning yields improvements on automatic speech recognition is very noteworthy. I have some points that need clarification (see below) before I can endorse this conclusion, but if they are addressed, then this will increase my confidence in the significance of this work.
- The motivation for this project is good. This work closes the loop on the large amount of existing work that has shown the emergent alignment between pretrained speech models and the brain. It appears novel in this respect.
- The random fMRI and BigSLM baselines increase my confidence in the fact that the recorded brain activity is a useful target for alignment.
summary: - In short the main idea is to fine-tune existing speech models so they are better aligned with fMRI recordings and then show that this results in better, more semantic alignment with the brain, and better downstream performance on speech tasks.
- The data comes from fMRI recordings that have been aligned to the podcast audio stimulus. For baselines, the authors run the off-the-shelf models and also fine-tune using fMRI signal unrelated to the task as well as audio representations from another model
- Finally, the authors formalize a notion of low-level feature impact, i.e., how much of the alignment can be attributed to low level (non-semantic) audio features, and show that their brain-tuning results in an alignment that can be explained less by low-level features.
weaknesses: - The conclusion that brain-tuning yields improvements over the off-the-shelf models is incredible, to the degree that I feel I must be misinterpreting the conclusion. There are two things I want to be sure of before this conclusion can be accepted:
  - First, is there any architectural enhancement that the brain-tuned version of wav2vec2 has that the original version does not? Extra projection layers?
  - Second, are the performances of wav2vec2 and HuBERT being fairly represented? It seems puzzling to me that the reported off-the-shelf ASR performances are so low (see questions below). One thing that may be happening: there is a distribution difference between the Librispeech data that wav2vec2 is trained on and the podcast audio used in the experiment. During brain-tuning, the model gets a chance to fine-tune on the podcast audio, or a proxy for that audio (via the brain), and this explains the improved performance. The gold standard experiment for this claim would be to do brain-tuning over one dataset of fMRI-audio pairs, and then show improvement on a completely new set of audio. To this end, it would be good to report the model performance on their original tasks. 
- The fact that brain-tuning does not result in catastrophic forgetting of the original speech model representations is so incredible that I feel I must be missing something. The brain is a very noisy proxy for the audio signal, and using the brain as a target seems to only give very weak incentive to preserve the original speech representations (see questions). As mentioned above, it would be good to report model performance on their original eval datasets (ex. Librispeech) to make sure that any forgetting did not occurr.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: * What areas of the brain were voxels covering for your fMRI target? Were semantic areas covered or other areas covered?
* My familiarity with fMRI is low by why would you downsampling using a 3-lobed Lanczos filter? Just a bit confused on the need to do so, not anything wrong with the choice.
soundness: 3
strengths: * Novelty: The paper isn’t the first to propose brain-tuning (which has been proposed in the vision and language domains) but is the first to apply this approach to speech models which often lack semantic understanding.
* Improvements in downstream performance: Most impressively is the finding that tuning with fMRI leads to strong improvements in ASR. While prior work has shown improvements from incorporating neural signals, I haven’t seen consistent improvements on downstream performance. 
* Reduced impact of low level features in brain alignment: Low level features become less impactful indicating fewer simple audio features are driving alignment to the brain. This is a beneficial and impactful finding. 
* Phonetic-Semantic Split: Following the prior strength, I liked Appendix C which quantified the phonetic-semantic preferences between finetuning and pretrained models.
summary: The paper incorporates a new finetuning approach for speech encoder models using neural recordings from subjects listening to stories. The approach uses the fMRI responses as a target during finetuning and modifies transformer layers of the speech. The paper demonstrates three areas of general improvement: (1) better brain alignment, (2) reduced impact of low level features for brain alignment, and (3) improved downstream performance across certain tasks.
weaknesses: * Baselines: In general, I think the current baselines are noteworthy but I think there are a few baselines which are missing. 
    * If the goal of the paper is to improve semantic understanding in speech models, the SLM baseline is a bit odd to me. Wouldn’t it be better to use a pretrained language model embedding as a target instead? I would recommend adding this baseline as another way to incorporate semantic information into the model. I also found a model embedding target to be a bit unfair in comparison to fitting voxels since the target is one-dimensional but any target from an embedding model is higher-dimensional according to my understanding, reducing the complexity of the fit.
    * I also think it would be great to include a baseline that controls for particular voxels of fMRI recordings. For example, I was wondering whether the area of the brain had an effect on the outcome of the model.
* Forgetting: One problematic aspect of finetuning is catastrophic forgetting where models can forget currently learned capabilities. I have this concern after seeing the performance decrease in your emotion prediction task. It would be very useful to quantify this point more formally to give a sense of whether semantic information is better incorporated at the sacrifice of basic phonetic understanding. This doesn’t seem to be reflected in appendix C as well. I would appreciate seeing 1-2 low level audio tasks benchmarked as well.
* Neural Signal Details: Some details were missing. See questions.
* (Nit) Consider adding [1] to your related work as another paper that tunes models based on neural information. 

Overall, I enjoyed this paper. I think adding a few additional baselines will make this paper much stronger. After seeing these, I will raise my score. 

[1] Dapello et. al. 2023 Aligning Model and Macaque Inferior Temporal Cortex Representations Improves Model-to-Human Behavioral Alignment and Adversarial Robustness. ICLR 2023

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 3
questions: 1. Please provide more detailed explanation why so few comparison models are provided or add more comparison models in the paper. This will dramatically affect my rating, if no reasonable explanation were provided, I would highly question the effectiveness of the method and definitely downgrade my rating.
2. If possible, answer the second weaknesses, this is relatively subtle.
soundness: 4
strengths: 1. **Novelty**: The concept of "brain-tuning" is a novel way to integrate neuroscience data into AI training, bridging the gap between machine learning and brain sciences.
2. **Comprehensive Evaluation**: The evaluation strategy is thorough, with alignment tests, low-level feature analysis, and performance on multiple downstream tasks.
3. **Cross-disciplinary impact**: The paper offers insights valuable to both cognitive neuroscience and AI communities, particularly for understanding semantic processing in the brain and improving language models.
4. **Significant Results**: The results are robust, showing improved alignment with brain regions and downstream task performance, indicating the model's improved semantic capabilities, **but only in their restricted comparison models.**
summary: This paper explores an innovative approach to enhance the semantic alignment of speech language models with the human brain by incorporating brain data. By fine-tuning models using fMRI data of participants listening to natural stories, The goal is to improve the models' semantic understanding by aligning them better with brain regions involved in language comprehension. The authors evaluate the method across three model families (Wav2vec2.0, HuBERT, Whisper) and show improvements in brain alignment, reduced reliance on low-level features, and better performance on downstream tasks that require semantic understanding. This idea is excitingly novel due to its cross-disciplinary implications between neuroscience and AI.
weaknesses: 1. **Comparison models are too weak and restrictive for the experiment**: There are **only two** comparison models: BigSLM Fine-tuned and Random-fMRI, the latter is not very valid since using random-fMRI results will definitely deteriorate performance, so there is in fact **only one** valid compasion model BigSLM, which is not convincing at all! BTW, BigSLM sometimes degrade performance as Fig 4, it's not valid as well.  **To prove effectiveness of brain-tuning, the paper should better compare brain-tuning with other valid finetuning models, ont only pretrained models.**
2. fMRI data, in some degree, is too hard to acquire. As the paper mentioned, the largest public dataset of fMRI recordings only cover 8 participants. This will limit the usability of the proposed method, but this won't downgrade my rating for this paper since I think the idea side is novel and useful enough.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: 1. Why is brain-tuning not effective for whisper? More analysis needs to be conducted. Is it caused by the encoder-decoder architecture?
2. How is the cortical parcellation conducted? The LeBel dataset doesn't provide any standardized space for all the subjects if I remember correctly. Are the fMRI images from different subjects first projected to the same space?
soundness: 3
strengths: 1. This paper first proposes fine-tuning pretrained speech models with brain responses to help improve the alignment between computational models and human brain. The experimental results are also impressive and solid.
2. The authors design several interesting experimental settings and baseline methods to help investigate the influence of brain-tuning.
3. The motivation is clear enough. The paper is also very well written with many experimental details supported.
summary: The authors propose the concept of brain-tuning in this paper, which aims to augment pretrained speech model training directly with brain recordings. In this way, the authors claim to compensate for the shortcomings of previous methods that only use low-level speech features for brain and speech model alignment, and successfully incorporate speech models with improved brain-relevant semantics. Experiments on three popular pretrained transformer-based speech language model families in three distinct ways (alignment with new brain recordings in semantic regions of the brain, effect of low-level features, such as Tri-Phones and Articulation, on the alignment with these semantic regions, downstream performance on linguistic tasks that require semantic understanding) also highlight the significance of inducing brain-relevant semantics in brain and speech model alignment.
weaknesses: 1. Despite the authors effort in illustrating the effect of brain-tuning, the experiments are still insufficient in the following two aspects. (1)  The authors choose LeBel's dataset which only contains 8 subjects for practice. I don't think the conclusion drawn from this dataset is convincing enough given the very few subjects. Instead I suggest trying the Narratives dataset [1] which contains hundreds of subjects in a passive storying hearing task for experiments (by the way this dataset is considered as the largest fMRI dataset in naturalistic language comprehension, not the LeBel's dataset as the authors claimed in line 54), as many previous work [2][3][4] did. (2) The authors fail to investigate the influence of model parameters in the brain-tuning process. I think such experiments are easy but important, and should be included in the paper.

2. The methods of obtaining four low-level speech features need to be detailed in this paper (at least in Appendix), instead of just citing previous work.

[1] The “Narratives” fMRI dataset for evaluating models of naturalistic language comprehension. Nature Scientific Data

[2] Evidence of a predictive coding hierarchy in the human brain listening to speech. Nature Human Behavior

[3] Disentangling syntax and semantics in the brain with deep networks. ICML

[4] Deep language algorithms predict semantic comprehension from brain activity. Nature Scientific Reports

Meta-review: The paper introduces "brain-tuning," a method for fine-tuning speech language models (SLMs) using fMRI data from individuals listening to natural stories. The central claim is that incorporating brain signals into the training process enhances the models' semantic understanding, making them better aligned with how the human brain processes language. The work has implications for both cognitive neuroscience (understanding semantic processing in the brain) and AI (developing more human-like language models). The reliance on fMRI data is a practical limitation. While not a weakness of the paper's core idea, the reliance on fMRI data, which is expensive and difficult to acquire, could limit the scalability of the approach. The authors acknowledge this and suggest future work on combining data from multiple participants or experiments. Initial claims of "consistent improvements" on downstream tasks were too broad. Reviewers raised concerns about the ASR results, leading the authors to clarify that the main benefits are observed in more semantic tasks. Reviewer SHAj suggested adding a baseline pretrained on X + the fMRI stimulus audio, which the authors agreed to include in the final version. This will provide a crucial control for the effect of the added audio data.

Decision: Accept (Poster)