Improving Semantic Understanding in Speech Language Models via Brain-tuning

Authors: Omer Moussa, Dietrich Klakow, Mariya Toneva
Primary area: applications to neuroscience & cognitive science
Keywords: fMRI, Speech Models, Speech Recognition, Alignment, Brain Alignment, Cognitive Neuroscience, Encoding Models, Transformers

Abstract:
Speech language models align with human brain responses to natural language to an impressive degree. However, current models rely heavily on low-level speech features, indicating they lack brain-relevant semantics which limits their utility as model organisms of semantic processing in the brain. In this work, we address this limitation by inducing brain-relevant bias directly into the models via fine-tuning with fMRI recordings of people listening to natural stories--a process we name brain-tuning. After testing it on 3 different pretrained model families, we show that brain-tuning not only improves overall alignment with new brain recordings in semantic language regions, but also reduces the reliance on low-level speech features for this alignment. Excitingly, we further show that brain-tuning leads to 1) consistent improvements in performance on semantic downstream tasks and 2) a representational space with increased semantic preference. Our results provide converging evidence, for the first time, that incorporating brain signals into the training of language models improves the models’ semantic understanding. We make the code available at https://github.com/bridge-ai-neuro/brain-tuning.

Decision: accept (poster)
Reviewer scores: 6, 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1729160426661

thread_type:
official_review

<thread_object>
creator: reviewer_61sy

<content>
summary:
The authors propose the concept of brain-tuning in this paper, which aims to augment pretrained speech model training directly with brain recordings. In this way, the authors claim to compensate for the shortcomings of previous methods that only use low-level speech features for brain and speech model alignment, and successfully incorporate speech models with improved brain-relevant semantics. Experiments on three popular pretrained transformer-based speech language model families in three distinct ways (alignment with new brain recordings in semantic regions of the brain, effect of low-level features, such as Tri-Phones and Articulation, on the alignment with these semantic regions, downstream performance on linguistic tasks that require semantic understanding) also highlight the significance of inducing brain-relevant semantics in brain and speech model alignment.

strengths:
1. This paper first proposes fine-tuning pretrained speech models with brain responses to help improve the alignment between computational models and human brain. The experimental results are also impressive and solid.
2. The authors design several interesting experimental settings and baseline methods to help investigate the influence of brain-tuning.
3. The motivation is clear enough. The paper is also very well written with many experimental details supported.

weaknesses:
1. Despite the authors effort in illustrating the effect of brain-tuning, the experiments are still insufficient in the following two aspects. (1)  The authors choose LeBel's dataset which only contains 8 subjects for practice. I don't think the conclusion drawn from this dataset is convincing enough given the very few subjects. Instead I suggest trying the Narratives dataset [1] which contains hundreds of subjects in a passive storying hearing task for experiments (by the way this dataset is considered as the largest fMRI dataset in naturalistic language comprehension, not the LeBel's dataset as the authors claimed in line 54), as many previous work [2][3][4] did. (2) The authors fail to investigate the influence of model parameters in the brain-tuning process. I think such experiments are easy but important, and should be included in the paper.

2. The methods of obtaining four low-level speech features need to be detailed in this paper (at least in Appendix), instead of just citing previous work.

[1] The “Narratives” fMRI dataset for evaluating models of naturalistic language comprehension. Nature Scientific Data

[2] Evidence of a predictive coding hierarchy in the human brain listening to speech. Nature Human Behavior

[3] Disentangling syntax and semantics in the brain with deep networks. ICML

[4] Deep language algorithms predict semantic comprehension from brain activity. Nature Scientific Reports

questions:
1. Why is brain-tuning not effective for whisper? More analysis needs to be conducted. Is it caused by the encoder-decoder architecture?
2. How is the cortical parcellation conducted? The LeBel dataset doesn't provide any standardized space for all the subjects if I remember correctly. Are the fMRI images from different subjects first projected to the same space?
</content>
</thread_object>

</thread>
<thread>
creation date:
1729409663425

thread_type:
official_review

<thread_object>
creator: reviewer_shaj

<content>
summary:
This paper explores an innovative approach to enhance the semantic alignment of speech language models with the human brain by incorporating brain data. By fine-tuning models using fMRI data of participants listening to natural stories, The goal is to improve the models' semantic understanding by aligning them better with brain regions involved in language comprehension. The authors evaluate the method across three model families (Wav2vec2.0, HuBERT, Whisper) and show improvements in brain alignment, reduced reliance on low-level features, and better performance on downstream tasks that require semantic understanding. This idea is excitingly novel due to its cross-disciplinary implications between neuroscience and AI.

strengths:
1. **Novelty**: The concept of "brain-tuning" is a novel way to integrate neuroscience data into AI training, bridging the gap between machine learning and brain sciences.
2. **Comprehensive Evaluation**: The evaluation strategy is thorough, with alignment tests, low-level feature analysis, and performance on multiple downstream tasks.
3. **Cross-disciplinary impact**: The paper offers insights valuable to both cognitive neuroscience and AI communities, particularly for understanding semantic processing in the brain and improving language models.
4. **Significant Results**: The results are robust, showing improved alignment with brain regions and downstream task performance, indicating the model's improved semantic capabilities, **but only in their restricted comparison models.**

weaknesses:
1. **Comparison models are too weak and restrictive for the experiment**: There are **only two** comparison models: BigSLM Fine-tuned and Random-fMRI, the latter is not very valid since using random-fMRI results will definitely deteriorate performance, so there is in fact **only one** valid compasion model BigSLM, which is not convincing at all! BTW, BigSLM sometimes degrade performance as Fig 4, it's not valid as well.  **To prove effectiveness of brain-tuning, the paper should better compare brain-tuning with other valid finetuning models, ont only pretrained models.**
2. fMRI data, in some degree, is too hard to acquire. As the paper mentioned, the largest public dataset of fMRI recordings only cover 8 participants. This will limit the usability of the proposed method, but this won't downgrade my rating for this paper since I think the idea side is novel and useful enough.

questions:
1. Please provide more detailed explanation why so few comparison models are provided or add more comparison models in the paper. This will dramatically affect my rating, if no reasonable explanation were provided, I would highly question the effectiveness of the method and definitely downgrade my rating.
2. If possible, answer the second weaknesses, this is relatively subtle.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730675711151

thread_type:
official_review

<thread_object>
creator: reviewer_bduh

<content>
summary:
The paper incorporates a new finetuning approach for speech encoder models using neural recordings from subjects listening to stories. The approach uses the fMRI responses as a target during finetuning and modifies transformer layers of the speech. The paper demonstrates three areas of general improvement: (1) better brain alignment, (2) reduced impact of low level features for brain alignment, and (3) improved downstream performance across certain tasks.

strengths:
* Novelty: The paper isn’t the first to propose brain-tuning (which has been proposed in the vision and language domains) but is the first to apply this approach to speech models which often lack semantic understanding.
* Improvements in downstream performance: Most impressively is the finding that tuning with fMRI leads to strong improvements in ASR. While prior work has shown improvements from incorporating neural signals, I haven’t seen consistent improvements on downstream performance. 
* Reduced impact of low level features in brain alignment: Low level features become less impactful indicating fewer simple audio features are driving alignment to the brain. This is a beneficial and impactful finding. 
* Phonetic-Semantic Split: Following the prior strength, I liked Appendix C which quantified the phonetic-semantic preferences between finetuning and pretrained models.

weaknesses:
* Baselines: In general, I think the current baselines are noteworthy but I think there are a few baselines which are missing. 
    * If the goal of the paper is to improve semantic understanding in speech models, the SLM baseline is a bit odd to me. Wouldn’t it be better to use a pretrained language model embedding as a target instead? I would recommend adding this baseline as another way to incorporate semantic information into the model. I also found a model embedding target to be a bit unfair in comparison to fitting voxels since the target is one-dimensional but any target from an embedding model is higher-dimensional according to my understanding, reducing the complexity of the fit.
    * I also think it would be great to include a baseline that controls for particular voxels of fMRI recordings. For example, I was wondering whether the area of the brain had an effect on the outcome of the model.
* Forgetting: One problematic aspect of finetuning is catastrophic forgetting where models can forget currently learned capabilities. I have this concern after seeing the performance decrease in your emotion prediction task. It would be very useful to quantify this point more formally to give a sense of whether semantic information is better incorporated at the sacrifice of basic phonetic understanding. This doesn’t seem to be reflected in appendix C as well. I would appreciate seeing 1-2 low level audio tasks benchmarked as well.
* Neural Signal Details: Some details were missing. See questions.
* (Nit) Consider adding [1] to your related work as another paper that tunes models based on neural information. 

Overall, I enjoyed this paper. I think adding a few additional baselines will make this paper much stronger. After seeing these, I will raise my score. 

[1] Dapello et. al. 2023 Aligning Model and Macaque Inferior Temporal Cortex Representations Improves Model-to-Human Behavioral Alignment and Adversarial Robustness. ICLR 2023

questions:
* What areas of the brain were voxels covering for your fMRI target? Were semantic areas covered or other areas covered?
* My familiarity with fMRI is low by why would you downsampling using a 3-lobed Lanczos filter? Just a bit confused on the need to do so, not anything wrong with the choice.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730688196062

thread_type:
official_review

<thread_object>
creator: reviewer_x8dp

<content>
summary:
- In short the main idea is to fine-tune existing speech models so they are better aligned with fMRI recordings and then show that this results in better, more semantic alignment with the brain, and better downstream performance on speech tasks.
- The data comes from fMRI recordings that have been aligned to the podcast audio stimulus. For baselines, the authors run the off-the-shelf models and also fine-tune using fMRI signal unrelated to the task as well as audio representations from another model
- Finally, the authors formalize a notion of low-level feature impact, i.e., how much of the alignment can be attributed to low level (non-semantic) audio features, and show that their brain-tuning results in an alignment that can be explained less by low-level features.

strengths:
- The fact that brain-tuning yields improvements on automatic speech recognition is very noteworthy. I have some points that need clarification (see below) before I can endorse this conclusion, but if they are addressed, then this will increase my confidence in the significance of this work.
- The motivation for this project is good. This work closes the loop on the large amount of existing work that has shown the emergent alignment between pretrained speech models and the brain. It appears novel in this respect.
- The random fMRI and BigSLM baselines increase my confidence in the fact that the recorded brain activity is a useful target for alignment.

weaknesses:
- The conclusion that brain-tuning yields improvements over the off-the-shelf models is incredible, to the degree that I feel I must be misinterpreting the conclusion. There are two things I want to be sure of before this conclusion can be accepted:
  - First, is there any architectural enhancement that the brain-tuned version of wav2vec2 has that the original version does not? Extra projection layers?
  - Second, are the performances of wav2vec2 and HuBERT being fairly represented? It seems puzzling to me that the reported off-the-shelf ASR performances are so low (see questions below). One thing that may be happening: there is a distribution difference between the Librispeech data that wav2vec2 is trained on and the podcast audio used in the experiment. During brain-tuning, the model gets a chance to fine-tune on the podcast audio, or a proxy for that audio (via the brain), and this explains the improved performance. The gold standard experiment for this claim would be to do brain-tuning over one dataset of fMRI-audio pairs, and then show improvement on a completely new set of audio. To this end, it would be good to report the model performance on their original tasks. 
- The fact that brain-tuning does not result in catastrophic forgetting of the original speech model representations is so incredible that I feel I must be missing something. The brain is a very noisy proxy for the audio signal, and using the brain as a target seems to only give very weak incentive to preserve the original speech representations (see questions). As mentioned above, it would be good to report model performance on their original eval datasets (ex. Librispeech) to make sure that any forgetting did not occurr.

questions:
- How is performance on the original speech task preserved even during brain-tuning? Do the authors have an explanation for why no catastrophic forgetting occurs, given that there is no incentive for the base model to retain it's original speech representations during brain-tuning? Is it just because the learning rate is very small? The brain-tuning set is very small?
- The ASR performance of wav2vec2 seems low? The original wav2vec2 paper (https://arxiv.org/pdf/2006.11477) reports accuracies of >0.8 on Librispeech. What explains the difference?
- I'm having a hard time following the reasoning behing the low-level feature impact results. If I understand correctly, models were brain-tuned. And then the low-level impact on the brain-tuned models was found to be smaller, compared to the random fMRI baseline. Why should this be the case? Shouldn't we expect the opposite based on the evidence cited on lines 237-238, which suggests that brain-tuning should increase alignment between the brain and the model, and therefore the reliance on low level features?

## small stuff
- Figure 1c: Are the bar charts supposed to have an x-axis label?
- Line 257: `` gives open quotes in LaTeX
</content>
</thread_object>

</thread>
<thread>
creation date:
1732384312221

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We thank all reviewers for their detailed and constructive feedback. We were very happy to see that the reviewers believe our work is "noteworthy” (x8dp, BDuH), "beneficial and impactful” (BDuH), "novel” and "thorough” (SHAj), “impressive” and “very well written” (61Sy).
 
The reviews highlight two main areas that can help strengthen the confidence in the results: adding a new baseline and adding new results on additional low-level speech tasks. We do both in our response and show that our brain-tuned models behave as we expect: outperform a new language model-finetuned baseline, and do not significantly impact performance on low-level speech tasks. We provide more details below in the common response as well as respond to each individual reviewer concerns in separate threads. We additionally include some clarifications that we believe are helpful:

1. **ASR results in Fig 4a:** The results are correct and were correctly reported in the text, but the plot in the submission showed the word error rate for two of the models, rather than the accuracy (1-word error rate). We have now fixed this issue in the revised manuscript (Fig 4a). We understand how this subplot was confusing and apologize for this error. We hope that the revised figure alleviates the concerns regarding what appeared to be a low pretrained performance on ASR. In fact, the pretrained ASR performance we report on the TIMIT dataset is on par with previous results reported on this dataset. [1, 2]. The brain-tuned model outperforms the pretrained model by 12%, as reported in the text.

2. **Datasets for downstream evaluation:** The datasets used for evaluation on the downstream tasks are popular benchmark datasets and are completely independent of the fMRI data that the brain-tuned model had access to. The details of these datasets can be found in Section 3.4.3 and Appendix B. We believe that using datasets that are new to all models leads to more fair evaluation and increases confidence in the downstream results.

3. **New results on additional low-level speech tasks:** To alleviate Reviewers x8dp and BDuH's concerns about potential catastrophic forgetting of speech capabilities, we now evaluate additional low-level speech tasks: MFCC and FBANK prediction. We find that brain-tuned and BigSLM models perform on par with the pretrained model on these low-level tasks, indicating that brain-tuning does not significantly impact low-level speech capabilities (Appendix D.1). We believe this is due to the fact that the brain regions that are included in the training do not only include semantic regions but also low-level speech regions (the auditory cortex). Moreover, we observe that the model layers that are affected most by brain-tuning are middle to late layers (Appendix D.1), and not the early layers which best encode the core acoustic features [3]. 

4. **New baseline and clarification on existing baselines:**

    1. **Language Model-finetuned Baseline**: Reviewer BDuH suggested a new baseline which uses representations from a Language Model (LM) as a fine-tuning target. We implemented an LM-finetuned baseline using representations from GPT2, and we report its downstream performance in Appendix D.2. Most notably, for all reported tasks, the brain-tuned model outperforms the GPT2-finetuned baseline. As compared to the BigSLM-finetuned models, the GPT2-finetuned baseline falls behind it in 4 of the 6 reported tasks. These results don’t show evidence that an LM-finetuned baseline would necessarily be consistently better than a BigSLM-finetuned baseline. 

    2. **BigSLM baseline**: We believe this baseline, which uses representations from a large speech model as a fine-tuning target, is the best comparison to test the added value of brain data over speech models, because: 1) It shows what semantics the fMRI targets provide that representations from an advanced speech model do not. 2) It allows us to contrast how the speech-related abilities of the models are affected (decreased or increased) by fMRI targets as compared to targets from an advanced speech model. This intuition is confirmed by BigSLM fine-tuned models generally performing better or on par with the pretrained ones on downstream tasks (especially ASR). 

    3. **Random-fMRI baseline**: This baseline is useful for two reasons. First, it ensures that having the correctly aligned fMRI target is essential (i.e., not any listening brain signal can work). Second, noise injection and noisy targets can act as regularization and have positive effects on the model [4, 5]. Hence, grounding results obtained with correct brain targets by comparing them to what random targets can achieve is necessary to ensure there are benefits to the process beyond a simple regularization effect.

We believe the new results we have provided have strengthened our work and hope that they address the concerns of the reviewers. We will appreciate the opportunity to engage in a discussion if there are remaining concerns.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces "brain-tuning," a method for fine-tuning speech language models (SLMs) using fMRI data from individuals listening to natural stories. The central claim is that incorporating brain signals into the training process enhances the models' semantic understanding, making them better aligned with how the human brain processes language. The work has implications for both cognitive neuroscience (understanding semantic processing in the brain) and AI (developing more human-like language models). The reliance on fMRI data is a practical limitation. While not a weakness of the paper's core idea, the reliance on fMRI data, which is expensive and difficult to acquire, could limit the scalability of the approach. The authors acknowledge this and suggest future work on combining data from multiple participants or experiments. Initial claims of "consistent improvements" on downstream tasks were too broad. Reviewers raised concerns about the ASR results, leading the authors to clarify that the main benefits are observed in more semantic tasks. Reviewer SHAj suggested adding a baseline pretrained on X + the fMRI stimulus audio, which the authors agreed to include in the final version. This will provide a crucial control for the effect of the added audio data.