Brain-tuning Improves Generalizability and Efficiency of Brain Alignment in Speech Models
Authors: Omer Moussa
Mariya Toneva
Venue: NeurIPS 2025 poster ()
Keywords: fMRI
Alignment
Brain Alignment
Cognitive Neuroscience
Encoding Models
Speech Models
Primary area: neuroscience_and_cognitive_science
TL;DR: Fine-tuning speech models jointly with brain responses from multiple participants improves brain alignment (even on novel datasets) and decreases the amount of fMRI data needed to predict new participants.

Abstract: Pretrained language models are remarkably effective in aligning with human brain responses elicited by natural language stimuli, positioning them as promising model organisms for studying language processing in the brain. However, existing approaches for both estimating and improving this brain alignment are participant-dependent and highly affected by the amount of data available per participant, hindering both generalization to new participants and population-level analyses. In this work, we address these limitations by introducing a scalable, generalizable brain-tuning method, in which we fine-tune pretrained speech language models to jointly predict fMRI responses from multiple participants. We demonstrate that the resulting brain-tuned models exhibit strong individual brain alignment while generalizing across participants. Specifically, our method leads to 1) a 5-fold decrease in the amount of fMRI data needed to predict brain data from new participants,  2) up to a 50\% increase in the overall brain alignment, and 3) strong generalization to new unseen datasets. Furthermore, this multi-participant brain-tuning additionally improves downstream performance on semantic tasks, suggesting that training using brain data from multiple participants leads to more generalizable semantic representations. Taken together, these findings demonstrate a bidirectional benefit between neuroscience and AI, helping bridge the gap between the two fields. We make our code and models publicly available at https://github.com/bridge-ai-neuro/multi-brain-tuning.

Brain-tuning Improves Generalizability and Efficiency of Brain Alignment in Speech Models
Omer Moussa
Mariya Toneva
Max Planck Institute for Software Systems Saarbrücken, Germany
Max Planck Institute for Software Systems
Saarbrücken, Germany mtonevadmpi-stte -wwrrammmiissss...mm
omoussa$mpi-ssee org
Abstract
Pretrained language models are remarkably effective in aligning with human brain responses elicited by natural language stimuli, positioning them as promising model organisms for studying language processing in the brain. However, existing approaches for both estimating and improving this brain aligmment are participant- dependent and highly affected by the amount of data available per participant, hindering both generalization to new participants and population-level analyses. In this work, we address these limitations by introducing scalable, generalizable brain-tuning method, in which we fine-tune pretrained speech language models to jointly predict 1MRI responses from multiple participants. We demonstrate that the resulting brain-tunee models exhibit strong individual brain alignment while generalizing across participants. Specifically, our method leads to 1) a 5-fold decrease in the amount of fMRI data needed to predict brain data from new participants, 2) up 50% increase in the overall brain alignment, and 3) strong generalization to new unseen datasets. Furthermore, this multi patlliipatiiipantt brain-tuning additionally improves downstream performance on semantic tasks, suggesting that training using brain data from multiple participants leads to more generalizable semantic representations. Taken together, these findings demon- strate a bidirectional benefit between neuroscience and AI, helping bridge the gap between the two fields. We make our code and models publicly available at
1 Introduction
Language models (LMs) substantially align with human brain responses elicited during natural language processing, such as functional magnetic resonance imaging (fMRI) signals recorded while participants listen to read stories Toneva and Wehbe, 2019 Schrimpf et al. 2021 Goldstein et al.) 2022, Millet et al.) 2022 Oota et al., 2024| Moussa et al. [2025]. This intriguing correspondence has positioned pretrained LMs promising model organisms for studying human cognition, particularly language comprehension [Toneva 202T]. Such models can offer new insights into how the brain represents and processes complex linuussiic information.
However, despite the growing success of pretrained models in predicting brain activity, current brain aligmment approaches face significant limitations. Existing methods are data inefficient, requiring extensive fMRI data from a new participant to estimate alignment with a specific LM (Antonello) et al. 2 Additionally, these methods are participant-dependent, which limits generalization to new participants. Even methods that incorporate brain data directly into the model during training remain patticpaant-dependent [Moussa al.) 2025 Vattikonda et al. 2025). As result, leveraging these models to study language processing across populations remains a challenge. Addressing these issues is critical to advancing the use of language models proxies for human language processing.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

To overcome these challenges, we propose a novel approach, designed to improve the efficiency and generalization of brain alignment in speech models. Our method fine-tunes pretrained speech language models to jointly predict brain responses from multiple participants who are exposed to the same naturalistic speech stimuli. By pooling data across individuals, this brain-tuning approach leverages common patterns in language processing, reducing the dependency on large amounts of participant-specifi data. This scalable strategy enables the creation of brain-tuned models that not only align well with individual brain responses but also generalize effectively to new participants.
Our extensive experiments demonstrate the multiple benefits of brain-tuning. First, our method reduces the amount of fMRI data required to achieve reliable brain alignment for new participants by a factor of five, significantly lowering the required fMRI data for robust estimate of brain alignment. Second, brain-tuning yields up to 50% improvement in overall brain alignment, indicating that jointly training on data from multiple participants enhances model robustness and prediction quality. Third, our approach improves brain alignment on novel stimuli and participants, indicating a strong generalization ability. Notably, brain-tuned models also improve downstream performance on semantics-related tasks relative to their pretrained counterparts, opening up the possibility of practical integration of brain-tuned speech models into speech processing pipelines without compromising linguistic utility. We hope that our extensive experiments on multiple pretrained models, as well as comprehensive ablation studies, will help establish best practices for brain- taiingtuning.
The proposed brain- tuiing method provides a stepping stone towards scalable, participant aggnosicc brain alignment, facilitating more inclusive and generalizable models of human language processing. By enabling the use of pretrained language models in population- neuroscience studies, our work bridges the gap between advanced computational techniques and the study of human cognition, providing a robust foundation for future interdisciplinary research. We make all code and trained models publicly available to facilitate reproducibility and further research in brain-tuning.
2 Related Work
A growing number of studies investigate the degree of alignment between language-evoked brain
activity and text-based language models [Wehbe al., 2014 Jain and Huth 2018 Toneva and
Wehbe 2019 Abdou et al. 1 (Schrimpl ct al. 202 Toneva et al. 2022a b Antonello et al.
1021OOla al. [2022] Merlin and Toneva 2022 Antonello et al. 2024] and speech based language
models Millet et al. 2022 Vaidya et al. 2002 luckute et al., 2023 Oota et al., 2023 2024 Chen ctt al. 2024]. These works rely on prrticipant-speiife brain encoding models, as brain representations exhibit substantial inter- variability due to anatomical and functional differences. Moreover, accurately estimating brain alignment has been shown to require a large amount of per-participant data [Antonello et al., 2024]. This participant-speciff nature severely limits model generalization across individuals, particularly with modalities like fMRI, EEG, and MEG.
Previous work has proposed methods to unify multiple participants into the same space [Chen et al. 2015 Haxby et al. 2020 Nastase et al.) 2020 Beliy et al.] 2025]. However, these approaches need huge datasets and are typically not focused on the language and speech domains. Moreover, they do not leverage the existing powerful pretrained models and instead train brain encoding models from scratch. Additional work that focuses on brain decoding (i.e., predicting what stimulus a participant observes from their brain response) has also made strides towards unifying brain responses from multiple participants. This is typically achieved by training participant-speciiee projection networks [Tang and Huth| 2025 Jayalath et al. 2024] Défossez et al. 2023]. Such methods usually focus on learning low-dimense dimenniniioall features, scale poorly with many participants, and require participants to share the stimuli. In contrast, our method leverages pretrained language models for improved efficiency and performance for encoding models, with focus language and speech in the brain.
promising recent direction for improving brain encoding involves fine-tuning pretrained language models using brain fMRI responses for naturalistic stimuli (i.e., brain-tuning [Mousa et al. 2025) and BrainWavL.M /Vattikonda et al.) (0255). ). Brain-tuning demonstrated that fine-tuning with brain data enhances a speech model's 's brain alignment to semantic cortical regions and also improves the model's semantic performance on downstream tasks [Moussa et al. 2025). Similarly, Brain WavLM uses low-rank adaptation (LoRA) to fine-tune a pretrained speech transformer on naturalistic fMRI data, showing improvement in brain alignment for a training participant and a held-out participant Vattikonda et al. 2025). While these methods illustrate that limited neural data can significantly
2

---

enhance the alignment of pretrained model representations, they are built per participant and do not learn from brain data from multiple sources, limiting their scalability and generalization. Our method overcomes these limitations, enabling the model to be jointly trained using responses from multiple participants. By jointly brain-tuning models across multiple participants, we increase efficiency and enhance generalization, reducing the need for huge per- per-participant data and facilitating robust and scalable models for brain alignment and cognitive neuroscience research. Additionally, we explicitly investigate the impact of our brain-tuning on the amount of IMRI data needed for brain encoding models, as well as the impact of the tuning data size on generalization to novel participants and data.
3 Methods
3.1 Pretrained Speech Models
To evaluate different starting points for brain-tuning, we used two popular pretrained transformer- based speech model families: Wav2Vec2.0 [Baevski et al. 2020] and HuBERT [Hsu et al., [02T]] We selected comparable versions of the models, each with 90M parameters, 12 transformer layers. an embedding dimension of 768, and a 20ms input token length. Both models are self-supervised and were pretrained to predict masked segments fMRI datasets that we use for developing and testing our brain-tuned models.
960-hour audio dataset that is independent of the
3.2
Naturalistic Brain Datasets
3.2.1 Datasets Details
For brain- traingtuning and evaluation, we use the Moth Radio Hour dataset [LeBel al., 2024], which is the largest per- per-participan fMRI dataset that is publicly available. This dataset consists of fMRI recordings of 8 participants who listened autobiographical stories from the Moth Radio Hour podcast. Three participants listened to 84 stories (x16..1 Ih of audio for each participant), while the rest listened to 27 stories (256.4h of audio). fMRI images were acquired every 2s (TR 2.0s).
To test cross-dataset generalization, we use a subset of the Narratives fMRI dataset [Nastase et al. 2021]. in which 16 participants listened to 56-minute fictional short story (with TR 1.5s). This subset provides a suitable setting to test generalization to a new dataset, as it has many participants and less per-participant data than the Moth Radio Hour dataset.
3.2.2 Spatial Alignment of fMRI Data Across Participants
A key challenge in multi-participan brain-tuning is the anatomical differences across individuals, leading to variations in brain size (and hence the number of fMRI voxels), surface geometry, and region boundaries. This makes it challenging to train jointly or make predictions for new participant. Moreover, using the whole cortex for fine-tuning, as done in [Moussa et al.) [0255] Vattikonda et al.. 2025) limits our ability to control what areas are included during fine-tuning. We solve this by spatially aligning participants.
To spatially align participants and be able to parse specific regions of interest (ROIs), we project each participant's is data to a common cortical surface with FreeSurfer v7. We then use the cerebral parcellation atlas from Glasser al. 2016 to parse auditory regions (A1 through A4) and late language ROIS (e.g., bilateral inferior frontal gyrus, angular gyrus, anterior and posterior temporal lobes, and middle frontal gyrus). The full list of ROIs and their functions is provided in Supp A.I
3.3 Brain-tuning Approach
In this section, we elaborate on our approach and training details for fine-tuning speech models with (MRI data from multiple participants (i.e., seati-brai-tttninn as well as the comparison baselines, namely brain-tuning with a single participant, LLM-tuning, and Stimulus-tuning (Sec) 333.4)
3.3.1 Data Preparation
We follow Moussa et al. (2025), Antonello et al. [204]] to preprocess the stimulus and brain data for training. We divide the audio stimulus into snippets of 2s. We then concatenate each snippet
3

---

TR-aligned, Audio
Transformer Layers
Feature Extractor
Pooling
FC Layer
Froeze
Fine-tune
Spatial Alignment
Spotta AN Migned Resspoor
O
Loss
Figure I: Multi-brain-tuning Approach. Given participant responses, we project them to common space (see 3...2. The aligned responses of multiple participants are then used to fine-tune the pretrained model with low-rank updates, binding by the shared stimulus as detailed in Sec 3.3.2
with the preceding 8s (4TRs) account for the fMRI hemodynamic response delay. This results in paired audio-fMRO dataset, with each sample having a 10s audio clip and its one corresponding fMRI TR. fMRI responses then spatially aligned and parcellated as described in Scc Parsing the desired auditory and language ROIs results in 30K voxels across both hemispheres.
3.3.2 Brain-tuning with Multiple Participants
After preparing the training data, we perform brain-tuning across multiple participants (Multi- brain-tuning shown in Fig[]. To achieve this, we add an average pooling layer followed by unified projection head on top of the speech model. Given stimulus batch S that is sampled contiguously from the audio, and the corresponding fMRI responses from participants PI.... we fine-tune the pretrained model to predict the participants' fMRI responses. This involves computing and backpropagating the training loss on each (S, P:) pair independently and sequentially.
This training strategy encourages the model to learn more generalizable representations and increases robustness noise inherent in individual responses. We observed that it outperforms methods that predict the averaged IMRI response across participants or those that average the loss across participants, possibly because these alternatives risk discarding unique, informative signals from individual responses. Furthermore, we found using a unified projection head on top of FreeSurfer ROIS be better than other alternatives like shared response modeling [Chen al. 2015) and petiiipaant-spccinn projection heads (as in [Déossez et al. 2023 Detailed comparisons are provided in Su.3
Note that our method does not require all participants to have listened to the same stimulus set. anchor during training, presenting the model with all available Instead, each stimulus consecutively. This design makes the method robust to datasets fMRI responses for that stimulus where participants have varying levels stimulus overlap. Importantly, we find that the critical factor for successful Multi-brain-tumm teaiingtunngg is having sufficient tuning data, as discussed in Sec 4.2
We test multiple training objectives (see Sec 4.4), and conduct the best-performing one-Lz reconstruction objective- as we observed that it scales better than the alternatives. Lastly, for easier scalability with more data, reduce the number of trainable parameters using a low-rank adaptation (LoRA) over the pretrained speech models [Hu et al. 2022).
final set of experiments with
Our approach removes the limitation of per-participant fine-tuning, in previous work MMoussa al. 2025] Vattikonda et al., 2025) and allows the brain-tuned model to integrate information across participants due to the shared projection head. The method is also adaptable multiple datasets.

---

3.3.3 Training details
We train the
model using fMRI responses from the three participants with the most data from the Full Moth Radio dataset, and the rest are held out for evaluation. During fine-tuning, we update the LoRA parameters and the projection head while keeping the feature extractor frozen.
We LoRA rank 8, which corresponds to 0.625% of the total model parameters. Increasing the rank beyond 8 did not help the model (Sec 4.4). We used a learning rate of 1 x 10 with a 10% warmup period and a linear decay. We split the fMRI stories into 2 validation stories, 1 held-out test story (exclusively used for evaluation and never during training), and the remaining 81 stories for training. At tuning time, we use a batch size of 128 samples of (audio, fMRI response) pairs (see Sec and train the model for 30 epochs. The training is stopped when the validation loss saturates or begins to diverge. Training takes approx. 6h on two NVIDIA A40 48GB GPUs.
3.3.4 Comparison Baselines
Single-brain-tuned. The most important baseline is the brain-tunce model with data from single participant. We do this by limiting the data to a single source and carrying out the same method and training settings (Fig BegsTaam and See B.3.3) We train Single-brain brriin--ttteedd models (one for each training participant), and report their average performance when evaluating on held-out participants.
LLM-tuned. An alternative way to improve a speech model is to fine-tune using representations from an LLM that encode rich semantics [Moussa et al. 2025]. Specifically, we replace the brain responses with representations obtained from LLama2-7B (see Supp B.3 for more details).
Htimulus-tunee This baseline aims to measure the benefits of brain-tuning against simply fine-tuning with the stimulus audio. We use the same pretraining self-supervised objective ([aevski et al. 2020 Hsu al. 202T]) to fine-tune the model with the stimulus set (see Supp B.3 for more details).
3.3.5 Ablations
The training objective and the number of trainable parameters can largely affect how model can learn from noisy fMRI data. Moreover, some objectives may be effective when the training data size is small but scale poorly and vice versa. We investigate these effects by performing ablations in our train-tuning approach. Specifically, we vary the training objective and LoRA Ranks. We test 3 objectives: the L2 loss, done in [Moussa et al., 2025 ]. Spatial Correlation Loss adapted from (Vattikonda et al.) 2025), and a combined Cosine Similarity + L2 loss. The exact formulation of the losses can be found in Au.p To test the effect of the number of trainable parameters, we compare different LoRA ranks, measuring how the ranks affect performance and scaling.
3.4 Evaluation
We evaluate multiple aspects of Multi-brain-tuning: efficiency, generalization, and downstream performance. Specifically, a successful brain-tuning approach should: (1) require less data to achieve reliable brain alignment for unseen participants (improved data efficiency), (2) yield higher brain alignment on new participants and datasets (improved generalization) when tuned with more data, and (3) result in no substantial degradation of the downstream utility of the model.
3.4.1 Estimating Brain Alignment
We compute brain alignment using standard voxel-wise encoding models. We follow Vaidya et al. 2022). Oota et al. (222), Moussa et al. (2025] in preparing the dataset needed for evaluation (estimating the brain alignment). First, extract speech features via 16.0s sliding window (stride 0.1s) over the audio stimulus S. We feed these segments into the speech model and retain the representations of the final token. Then, we interpolate these features with Lanczos filter to match the fMRI acquisition rate. Finally, we concatenate the features for the preceding 10s to account for the hemodynamic delay. The concatenated features used train the voxel-wise encoding model.
We carry out this voxel-wise encoding by learning a linear function per-voxel on the concatenated features. We use ridge regression to fit the model on the training portion of the encoding dataset, and the ridge parameter is chosen via cross validation. The encoding performance is evaluated on the held-out test set via Pearson correlation.
5

---

To estimate the alignment for the language areas, we first normalize the obtained voxel-wise correla- tion on the test set by the estimated voxel-wise noise ceiling (the maximum explainable variance. Supp(A.2). After this normalization, we compute the mean of the voxels belonging to the language ROIs (detailed in Supp/A.] A.T). This serves as a standardized measure for brain alignment since is computed relative to the estimated explainable variance in the brain region.
In all experiments, normalized brain alignment is reported an average across the upper-middle layers of the corresponding model. These layers were selected because they have been shown to best align with language regions (Antonello et al., 2024 Oota et al. 2024 Vaidya et al. 2022]. We report the mean of this alignment across participants and the standard error of the mean in all figures.
3.4.2 Efficiency of Brain Alignment
The brain alignment of the existing pretrained language models is poor when the amount data used to train the voxel-wise encoding is small [Antonello et al. [202). Hence, these models require large per-participant data to achieve a good brain alignment. A model that's more efficient would need much less data to obtain the same good brain alignment.
To investigate this efficiency aspect of our Multi-brain-tue brain-tuned model, we quantify the amount of encoding data needed to match the pretrained performance. We do this by gradually reducing the encoding data size, then computing the brain alignment of the voxel-wise encoding model trained on the reduced data. At each fraction of data, we compare alignment to the pretrained counterpart fitted on the full encoding dataset. We report this for training participants (the ones used for brain-tuning) and for held-out participants (unseen during brain-tuning).
In addition the Multi-brain-ture the pretrained models. Due to shared information among participants, it is expected that the Single- brain-tuned model will be more efficient than its pretrained counterpart on the held-out participants. However, we still expect the Multi-brain- tuned one to outperform both because it could better leverage shared and general information learned from multiple participants.
model, we also test the efficiency of the Single-brain-tuned and
3.4.3 Generalizahility of Brain Alignment
A generalizable brain-tuned model should improve brain alignment with held-out participants and -ito-f-distribution datasets by leveraging the shared general information it learned during brain- tuning using multiple participants. To investigate this generalization aspect, examine a) how brain alignment changes when we vary the size of the data used for brain-tuning and b) the alignment improvement on completely different dataset.
We expect the model to generalize when it's trained on a diverse enough dataset that also has a sufficient amount of data per participant. This is due to the noise in fMRI data, which necessitates both scale per participant and diversity to learn meaningful structure. For comparison, we similarly test the single-brain-tung
models.
To test cross-datased generalization, we use the subset of 16 participants from the Narratives dataset detailed in See 3.2.1 and estimate brain alignment (as in Sec 3.4.1) on a held-out 20% of this data. As a strong baseline, we also estimate the brain alignment of a model on the same 16 participants used for testing.
brain-tuned model, a
3.4.4 Downstream Performance
Previous work has shown that Brain-tuning could benefit downstream performance [Moussa et al., 2025), but it didn't it show whether it scales when the amount of brain-tuning data is beyond a single participant. Ideally, more data should benefit downstream performance, but the risk of catastrophic forgetting is also greater due to training the model with a larger dataset on specific task.
To test how our Multi cillti-brriin-ae ttaiin affects downstream performance, we test how downstream per- formance scales with the data size used for brain-tuning We do this for 2 tasks (namely Phonemes Prediction and Phonetic Sentence Type prediction), following Moussa et al. (2025). Detailed for- mulations of these tasks can be found in Au4p We use linear probes to perform the tasks and report the mean over model layers. In addition to the Multi-brain-tunm model, we also report the Single-brad benee-brin-tuuue tuned and the LLM-tuned models for comparison.

---

Multi-brain-tuned
Single-brain-tuned
Pretrained
A. Wav2Vec2.0
Trkliing Partiipants
Held-out Partiijante
S. M greeent From Man Pig-avel
5.5 Me (Mgreenn From Nan PHE-
-20
Buii Encesing Data Siee (Houss
Brain Encesing Data Siez (Hors)
B. HuBERT
Trakiin Particpants
Held out Participants
San S. Atgenan fen Max Proraieed
S.5 it Agunaa frem Max Protaceed
Bois Encoding Data Size (Hooes)
Boin Excoding Data Size (Hours)
Figure 2: Brain encoding efficiency for both training and held-out participants and two model families (Wav2Vec2.0 and HuBERT), Brain-tuned models consistently outperform pretrained counterparts on both training and held-out participants. Multi-brain-tunet models consistently outperform Single- brain-tuned ones, reaching the max. pretrained performance with approximately a fifth of the brain encoding data and improving brain alignment up to 50% from the max. pretrained brain alignment.
4 Results
4.1 Brain Alignment Efficiency
We evaluate Brain Alignment Efficiency by varying the size of brain encoding data detailed in S.4.. This enables us to measure the fraction of encoding data required by a tuned model to achieve or surpass the best brain alignment achieved by the pretrained model, which is obtained using the full encoding data. We report this Fig 2 across two model families (Wav2Vec2.0 and HuBERT) for the training participants (seen during Brain-tuning) and the held-out participants (unseen by all models). We note that all brain alignment is evaluated on fMRI data held-out from training.
For the training participants (left plots of Fig 2). both Multi-brain-tune (Sec 3.3.2 and the Single- brain-tuned (..4.3.3.4) models match the maximum pretrained performance using approximately one-fifth of the data. With more encoding data, both models continue to improve, reaching up to 50% more alignment when using the full encoding data. Notably, the brain-tuned model consistently outperforms the Single-brain-tuned model across all evaluated data fractions, alleviating the need for the standard practice of building Single- tanlic-patticipat models to attain the best performance.
The Multi-brain-tumm tuned model maintains its efficiency advantage for held-out participants (right plots of Fig reaching the pretrained performance with roughly one-fifth of the data, whereas the Single- brain-tuned model requires approximately twice this amount. Both models continue to improve with more data, with clear advantage for the Multi-brain-tuned model. These results are consistent across both the Wav2Vec2. O and HuBERT model families. The comparable efficiency of the Multi-brain- tuned model on training and held-out participants strongly suggests that Multi-brain culti-brain-tuning leads to more efficient and general representations that are beneficial for unseen participants.
7

---

A. Performance scaling with tuning data size
Mult-brain--uned
Single-rain-tuned
Traiiing Partiipants
Participants
Sain Algmne Frn Max Predixedd
Aigggmmnnt From Man Pretsche
Briivtusing Stamados See (foww)
friiittoing Soe (Hoves)
B. Flat map visualization of improvement over pretrained
C. Improvement Narratives
Peorson Correation
Pecratad Bite
Brannsadd Betar
4.2 Mayeen from Man Teatraad
Figure 3: Brain-tuning generalization. (A) Scaling with tuning data size, showing that Multi-brain- tuning scales better, especially on held-out participants. (B) Voxel-wise change in brain alignment after celti-brain-tuuingg for held-out participant from the Moth Radio hour dataset (the remaining participants are reported in Supp B.2). showing improved alignment across frontal and parietal regions. (C) Aligmment improvement on novel stimuli and participants (from the Narratives dataset), indicating the capability of Multi-brain ciii-bbriintteeee tuned models for cross cross-dataset generalization.
4.2 Brain-tuning Generalization
In Sec we demonstrated that Multi-brain tuiinn enhances alignment magnitude and efficiency for training and held-out participants. Here, we investigate how the brain-tuned models generalize when provided with increasing amounts of data during brain- main-tuning and how these models perform diverse nut-of-distribution dataset.
Fig reports the change in brain alignment (relative the pretrained Wav2Vec2.00 0) as a function of increasing stimulus set size. The results for HuBERT can be found in Supp C.1 Generally, Multi-brain-tuning strongly benefits from increasing the tuning data size. For training participants, both Multi- and Single-brain-tun -unndd models demonstrate increased alignment as the tuning data grows, although the Multi-brain-tun utilized. As for the held-out participants, both models perform similarly when the tuning data size is less than hours. Nonetheless, Multi-brain-tunie models show a strong upward trend when more tuning data is used, while the Single-brain cere-bbrin-ttnee tuned ones saturate. Overall, Multi-brain--e ciii-bbrr--ttnnggggg shows greater ability to improve and generalize with more brain-tuning training data.
model consistently performs better when the entire stimulus set is
When we visualize the improvement in brain alignment of the Multi multi-brain--uuee over the pretrained Wav2Vec2.0 O for held-out participants from the Moth Radio Hour dataset (Fig33, Supp B.2). we observe widespread improvement across the brain, especially in the frontal and parietal regions. The auditory cortex shows a slight decrease alignment. This is because we report alignment over the upper-min midele and later layers of the models, which encode more semantics. An additional reason

---

Multi-brain-tuned
Single-brain-uned
LLM-tuned
A. Phoneme Prediction
B. Phonetic Sentence Type Prediction
Score from Presoieed
12
56.5 3m F1 Score from Peetraired
(30
Braieetninng Stinulas (Hours)
Brain-tuning Scimulas Stet (Hours)
Figure 4: Scaling of downstream performance with tuning data size. Brain-tuned models' performance increases with more data, with the Multi-brain-tumm taking less data to match the LLMM-tuned model.
may be that the auditory cortex be dominated by the larger semantic language areas during brain-tuning. This explanation also supported by the results of [Vatikonda et al. 2025).
Finally, Fi shows the mean brain alignment improvement in late language regions for Multi- and Single-brain- tuned models, which are tuned on the Moth Radio Hour dataset, when evaluated on an entirely new dataset 16 participants of the Narratives dataset. Both Moth-Brain-tume models show improvement over their pretrained ccounterparts, with the Moth-Multe brain-tuned being considerably better. In fact, (i.e.. a model tuned using the same 16 participants from the Narratives dataset), which indicates a strong generalization ability to novel stimuli and participants.
Ma---MMbbraiiiii -tuned does not lag much behind the Narratives-Mal Maslti-uuubbttt brain-rra---
Our results confirm the scalability ticipants and stimuli when trained on sufficient amounts of data. The strong upward trend also indicates room for further improvement more data is integrated during brain-tuning. We further corroborate these findings by showing that this scaling trend cannot be achieved with stimulus-tuning or LLM-uunin in Supp B.3 Next, we investigate the downstream performance brain-tuning.
calti-brain-tuning
improving alignment with unseen par-
4.3 Downstream Performance
While the gains of Multi celtti-brain-tte brain-tuming in brain alignment are clear (Sec 4.2 and 4.1 it's important to verify that Multi-brain brain-tuning doesn't 't lead to catastrophic forgetting or degrade downstream performance. Here, we evaluate the downstream performance of the tasks described in Sec(3.4.4 as function of tuning data size. We vary the size of the data similarly to Sec by increasing the size of the brain-tuning training stimulus set). In addition reporting the performance of Single- brain-tuned models, we also include the LLM-tuned baseline as it was shown to substantially improve performance similar tasks by Moussa al. (2225) and Vattikonda et al. [2025]
Fig shows the percent improvement over the pretrained Wav2Vec2.0 model. Similar results for HuBERT can be found in Supp C.2 Across all data fractions, brain-tuned models never underperform pretrained model, ruling out catastrophic forgetting. Impressively, the Multi-brain-tuned model matches the LLM-tuned performance when the size of the data increases. Both Multi- and Single- brain-tuned models benefit strongly from more tuning data, with the Multi-brain- bii--brrriinteee performing better at smaller tuning data sizes. Overall, muuti-brain-tuning improves downstream performance, which also scales up with the amount of data, eventually matching the L.MM-tuned baseline.
4.4 Effect of Tuning Objective and Model Size
In the previous sections, reported the results using the L2 objective and LoRA rank-8 updates. Here, we explain the reasons for these choices by comparing the Multi-brain brainb-tatt-t behavior for different LoRA ranks and training objectives Wav2Vec2.0 5F.
Fig 5A reports gains brain alignment several brain-tuning training data sizes for different LoRA ranks. We can observe that increasing the LoRA rank beyond 8 doesn't 'ts improve performance,

---

A. Effect of LoRA Rank
B. Effect of Training Objective
Rank Rank-
Cornelation
in Atgimon from Max Pretsined
S6 Alggmmett from Max Pretaired
10
12
16
Rrain-tuning Stimohis (Houm)
Brainstuning Stimalus Size (Moorrs)
Figure 5: Effect of varying the loss function and LoRA rank Multi-brain-tuning with (A) showing that Rank-8 updates perform best, and (B) indicating that the Lz loss scales better.
especially when the data size increases. We find that even larger LoRA ranks well brain-tuning the full model do not improve performance over using rank-8 updates (see Supp [..).
5 contrasts different training losses for increasing tuning data sizes. Generally, T- performs better when the size of the brain-tuning training data increases, while the other two losses saturate. At low brain-tuning data sizes (S. hours), the Correlation loss outperforms other losses. While these results motivate using the L2 loss because it scales better, they also highlight that the brain-tuning loss can have a large effect brain-tuning. Future work may find improved brain-tuning losses.
Discussion and Conclusion
In this work, we introduced a novel brain-tuning approach that significantly improves the general- izability and efficiency of brain alignment in pretrained speech models. By fine-tuning language models on brain responses from multiple participants exposed to speech stimuli, we demonstrated that our method effectively addresses the limitations of existing participant-depender Our extensive experiments showed that brain-tuning not only reduces the fMRI data requirements by a factor of five but also increases brain alignment by up to 50% and generalizes to new brain datasets.
approaches.
These results highlight the potential of brain-tuning to create robust, participant-agnostie models that generalize well across individuals, paving the way for more scalable and inclusive approaches to studying language processing in the brain. Moreover, our comprehensive ablation studies on training loss and model parameters establish best practices for implementing brain-tuning in speech models.
Notably, brain-tuning also improves speech models' downstream performance on semantic tasks. This finding suggests that incorporating brain data during fine-tuning not only aligns models with the human brain in efficient and generalizable ways but also leads to representations that better capture generalizable semantic information. This direct evidence for bidirectional benefit between neuroscience and Al contributes towards bridging the gap between the two fields.
Limitations and Future Work. We focused here on language-related brain regions, as they are most directly involved in processing the naturalistic speech stimuli used in our experiments. Future work can leverage our brain-tuning method for non-language regions specific brain areas to gain new insights into their functional roles. Notably, our method is flexible and can be adapted to target different brain regions. Second, our experiments were conducted exclusively in English, reflecting the availability of large public fMRI datasets. Investigating brain-tuning with multilingual learn innuuggg-indeeeddent, generalizable
data in the future assess whether brain-tuning semantic representations. Lastly, while we explored several training losses, there remains potential for developing new loss functions that lead to even better data efficiency and generalization.
By making our code and trained models publicly available, we aim to foster reproducibility and encourage further research on brain-tuning. We hope that our work will not only advance the integration of language models into cognitive neuroscience but also inspire new approaches to modeling human language comprehension.
10

---

Acknowledgments
This work was partially funded by the German Research Foundation (DFG) DFG Research Unit FOR 5368 and by the Max Planck Institute for Software Systems.
References
Mostafa Abdou, Ana Valeria González, Mariya Toneva, Daniel Hershcovich, and Anders Søgaard. Does injecting linguistic structure into language models lead to better alignment with brain recordings? arXiv preprint arXiv:2101. 2021.
Richard Antonello, Javier S Turek, Vy Vo, and Alexander Huth. Low-dimensional structure in the space of language representations reflected in brain responses. Advances in Neural Information Processing Systems, 34:8332-8344, 2021.
Richard Antonello, Aditya Vaidya, and Alexander Huth. Scaling laws for language encoding models in fmri. Advances in Neural Information Processing Systems, 36, 2024.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33:12449-12444
2020.
Roman Beliy Navve Wasserman, Amit Zalcher, and Michal Irani. The wisdom of crowd of brains: A universal brain encoder, 2025. URL https: org abs/ 12179
Catherine Chen, Tom Dupré la Tour, Jack L Gallant, Daniel Klein, and Fatma Deniz. The cortical representation of language timescales shared between reading and listening. Communications Biology, 7(1):284, 2024.
Po-Hsuan (Cameron) Chen, Janice Chen, Yaara Yeshurun, Uri Hasson, James Haxby, and Peter J Ramadge. A reduced-dimmnssion fmri shared response model. In C. Cortes, Lawrence, Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28, Curran Associates, Inc., 2015. URL /ttp:
ceurips.cc/paper pdf
Alexandre Défossez, Charlotte Caucheteux, Jérémy Rapin, Ori Kabeli, and Jean-Rémi King. Decod- ing speech perception from non-invasive brain recordings. Nature Machine Intelligence, 5(10): 1097-1107, 2023.
Rutvik H Desai, Usha Tadimeti, and Nicholas Riccardi. Proper and common names in the semantic system. Brain Structure and Function, 228(1):239-2) 254, 2023.
John S Garofolo. Timit acoustic phonetic continuous speech corpus. Linguistic Data Consortium, 1993, 1993.
Matthew F Glasser, Timothy S Coalson, Emma C Robinson, Carl D Hacker, John Harwell, Essa Yacoub, Kamil Ugurbil, Jesper Andersson, Christian F Beckmann, Mark Jenkinson, et al. A multi-modal parcellation of human cerebral cortex. Nature, 536(7615):171-178 2016.
Ariel Goldstein, Zaid Zada, Eliav Buchnik, Mariano Schain, Amy Price, Bobbi Aubrey, Samuel A Nastase, Amir Feder, Dotan Emanuel, Alon Cohen, et al. Shared computational principles for language processing humans and deep language models. Nature Neuroscience, 25(3):369-380 2022.
James V Haxby. J Swaroop Guntupalli, Samuel A Nastase, and Ma Feilong. Hyperalignment: Modeling shared information encoded in idiosyneratic cortical topographies. eLife, 9:e56601, jun 2020. ISSN 2050-084X. doi: 10.7554/el.ife.566011 URL httph:ttp///t. :/dp::://do.... org/10. 7554/ eLif 56601
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEFF/ACM Transactions on Audio, Speech, and Language Processing, 29:3451-3460, 2021.

---

Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. LoRA: Low-rank adaptation of large language models. International
Conference on Learning Representations, 2022. URL htttp:://openreeieee.eettt I2eVKeeFYf9
Shailee Jain and Alexander Huth. Incorporating context into language encoding models for fini. Advances in Neural Information Processing Systems, 31, 2018.
Dulhan Jayalath, Gilad Landau, Brendan Shillingford, Mark Woolrich, and Oiwi Parker Jones.
The brain's 's bitter lesson: Scaling speech decoding with self-supervised learning. arXiv preprint arXiv:2406.00 04328, 2024.
Amanda LeBel, Lauren Wagner, Shailee Jain, Aneesh Adhikari-Desai, Bhavin Gupta, Allyson Morgenthal, Jerry Tang, Lixiang Xu, and Alexander G. Huth. "an fmri dataset during passive natural language listening task". 2024. doi: doi:10.18/ 222oopenncuro.ds000002222 v2.2.0.
Gabriele Merlin and Mariya Toneva. Language models and brain alignment: beyond word-level semantics and prediction. arXiv preprint arXiv:2212.00596, 2022.
Juliette Millet, Charlotte Caucheteux, Yves Boubenec, Alexandre Gramfort, Ewan Dunbar, Christophe Pallier, Jean-Remi King, et al. Toward realistic model of speech processing in the brain with self-supervised learning. Advances in Neural Information Processing Systems, 35:33428-3344 -3344,, 2022.
Omer Moussa, Dietrich Klakow, and Mariya Toneva. Improving semantic understanding in speech language models via brain-tuning. In The Thirteenth International Conference Learning Representations, 2025. httptttpp::/pppre////// www.:ewwwet//oommmmee -aaaamnddddKLLLAaeeeeee
Samuel A. Nastase, Ariel Goldstein, and Uri Hasson. Keep it real: rethinking the primacy of experimental control in cognitive neuroscience. Neurolmage, 222: 1 17254, 2020. ISSN 1053-81 1553-8119 19. doi: https://doi.org// http:://doo.or/11...0fff/..emmee h0w.:////...eumomma...0......... 117254. UtLphtps::/ http://wwwwccinnniiir.....
Samuel A Nastase, Yun-Fei Liu, Hanna Hillman, Asieh Zadbood, Liat Hasenfratz, Neggin Ke- shavarzian, Janice Chen, Christopher J Honey, Yaara Yeshurun, Mor Regev, et al. The "narratives" " fmri dataset for evaluating models of naturalistic language comprehension. Scientific data, (1), 2021.
Subba Reddy Oota, Jashn Arora, Veeral Agarwal, Mounika Marreddy, Manish Gupta, and Bapi Surampudi. Neural language taskonomy: Which nlp tasks are the most predictive of fmni brain activity? In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3220-3237, 2022.
Subba Reddy Oota, Khushbu Pahwa, Mounika Marreddy, Manish Gupta, and Bapi S Raju. Neural architecture of speech. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023.
Subba Reddy Oota, Emin Çelik, Fatma Deniz, and Mariya Toneva. Speech language models lack important brain-relevant semantics. ACL, 2024.
Martin Schrimpf, Idan Asher Blank, Greta Tuckute, Carina Kauf, Eghbal A Hosseini, Nancy Kan- wisher, Joshua B Tenenbaum, and Evelina Fedorenko. The neural architecture of language: Integrative modeling converges on predictive processing. Proceedings of the National Academy of Sciences, 2021.
Jerry Tang and Alexander G. Huth. Semantic language decoding across participants and stimulus modalities. Current Biology, 35(5):1023-10322ee6 2025. ISSN 0960-9822. doi: https://doi.or//10 1016/j.cub.2025. $09609882225000445
www.wsecccccccctcccoomm URL [rg.:atppee//pmrscccccaeeeeeeeee atw.iiencc//rtiie/pp....
Mariya Toneva. Bridging language in machines with language in the brain. PhD thesis, Carnegie Mellon University, 1021.
12

---

Mariya Toneva and Leila Wehbe. Interpreting and improving natural- processing (in machines) with natural seggage-processing (in the brain). Advances in Neural Information Processing Systems, 32, 2019.
Mariya Toneva, Tom M Mitchell, and Leila Wehbe. Combining computational controls with natural text reveals aspects of meaning composition. Nature Computational Science, 2(11):745-757, 2022a.
Mariya Toneva, Jennifer Williams, Anand Bollu, Christoph Dann, and Leila Wehbe. Same cause; different effects in the brain. Causal Learning and Reasoning, 2022b.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09200 2023.
Greta Tuckute, Jenelle Feather, Dana Boebinger, and Josh H McDermott. Many but not all deep neural network audio models capture brain responses and exhibit correspondence between model stages and brain regions. Plos Biology, 21(12):3302336, 2023.
Aditya R Vaidya, Shailee Jain, and Alexander Huth. Self-supervised models of audio effectively explain human cortical responses to speech. International Conference on Machine Learning, pages 21927-21944. PMLR, 2022.
Nishitha Vattikonda, Aditya R. Vaidya, Richard Antonello, and Alexander G. Huth. Brainwavim: Fine-tuning speech representations with brain responses to language, 2025. URL htthhttss///axii brggabs/2502..00 08866
Leila Wehbe, Brian Murphy, Partha Talukdar, Alona Fyshe, Aaditya Ramdas, and Tom Mitchell. Simultaneously uncovering the patterns of brain regions involved in different story reading subpro- cesses. PloS One, (11), 2014.
13

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
In the abstract and at the end of the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
In section 5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
Our paper does not involve any proofs or theoretical assumptions.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: As described in the Methods section and throughout the Appendix, we have provided detailed descriptions and analyses of the experimental setups for all our investiga- tion.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
15

---

reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We plan to release all code and models publicly to replicate our analyses upon publication. The data is already publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
public/guides/ wttalees//ooeeSSmmimiiionoffeee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
Cooee//ddeeeebmimcia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: As described in the Methods section and throughout the Appendix, have provided detailed descriptions of the training and evaluation setups.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Error bars are described in Section 3.4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
16

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: In Methods.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We are convinced that we comply with NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have addressed the broader impacts of our work in Section 5. Additionally, as our research is primarily a fundamental scientific exploration and poses no additional social risks, we have included a discussion on potential harmfulness.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
17

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We use pretrained speech models and they are properly credited the Methods, and publicly available IMRI dataset and downstream task datasets, which we have properly cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, batasets.papersstt paperswi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We plan release code and models publicly and we will document them properly before release upon publication.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: use previously collected publicly available data.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We use previously collected publicly available data.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
19

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification:
We use pretrained speech models and describe the usage throughout the paper.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: http::://neurps.... http://erriiscccmommeeereneeee/e cc//oonferencee//222//LL. for what should or should be described.
20

---

A Additional Methodology Details
A.1 Brain ROIs Details
The Glasser Atlas for human cerebral cortex parcellation has 180 labeled ROIs per hemisphere Glasser et al., 2016]. From these labels, we extract the following regions to be used during brain- tuning: Angular gyrus, lateral temporal cortex, inferior frontal gyrus, and middle frontal gyrus (Oota et al., 2024 Desai et al. [023). It also has the primary auditory and the early auditory regions. Fig highlights the ROIs used for brain- traiintuuning on the right hemisphere. Table details each region and
ROI labels that cover it from the parcellation atlas.
Table I: Brain regions and corresponding ROI labels.
Region
Labels
Angular gyrus (AG)
PFm, PGs, PGi, TPOJ2, TPOJ3
Lateral temporal
cortex
STSda, STSva, STGa, TEla, TE2a, TGv, TGd, A5, STSdp. STSvp, PSL. STV, TPOJI
(LTC)
Inferior frontal gyrus (IFG)
44, 45, IFJa, IFSp
Middle frontal gyrus (MFG)
55b
Primary auditory cortex (A1)
A1
Early auditory regions
A1, PBelt, MBelt, LBelt, RI, A4
A.2 Noise Ceiling Calculation
Noise in IMRI data is very common and can impair brain-tuning and brain alignment estimation, so it is important estimate the noise ceiling of each voxel in the fMRI responses. The voxel-wise noise ceiling is estimated for all participants based on the method by the fMRI dataset paper LeBel al. 2022]. This method leverages repetitions of the same story for the participant (e.g., story is repeated 10 times), then uses these repetitions to compute the maximum explainable variance for each voxel. This noise ceiling value estimates the amount of explainable variance in the brain signal, ranging from 0 to 1. We use this estimated noise ceiling to normalize the brain alignment during brain alignment estimation, as mentioned in 3.c43 of the main paper.
Figure 6: Brain tainntuningg ROIs. higliw-highlighted regions are used for brain-tuning
A.3 Loss Functions and Training Details
Here, we detail the formulations of the different loss functions compared in Sections 3.3 and 4.4 of the main paper. We then compare different training techniques and alternatives for Multi-brain-uning showing that the one we used in 3.3.2 performs best.
21

---

A.3.1 Loss Functions
We define the loss functions over batch B of audio-fMRI pairs for Participant i, where the ground- truth fMRI responses are It and the predicted fMRI responses are it.
Le Loss. We compute the L2 loss between R² and Rt as follows:
C2
R
R
(I)
B
b=0
Correlation Loss. We compute the correlation loss between Rs and fe as follows (where corr correlation over voxels):
B
cor
Σ
(1
corr( Rij, Rt))
B
b=0
Cosine + Le Loss. We compute the Cosine + L2 loss between Ht' and it' as follows (where cos is the cosine similarity over voxels):
|BI-1
Loss
Σ (1
cos(Rit Rb))
(3)
[B]
6=0
Then we use alongside the L2 Loss (with X 0.5):
Leos-12 Loss + AL
(4)
A.3.2 Comparing Multi-brain-tuning Techniques
The training strategy we in the paper is predicting each participant's FreeSurfer ROIs responses independently (but using the same projection head), then computing the loss, and updating the model parameters. We detail here other techniques and alternatives and compare them to this setting, highlighting that our approach works best.
Loss Average. In the Loss average method, compute the loss for each participant, then average across participants before updating the parameters.
Response Average. For the Response average method, we average responses over participants, then compute the loss and update the parameters.
Separate Heads. Another alternative is to use a separate projection head for each participant. While this might work well with limited number of participants, it will increase the training parameters considerably when fine-tuning with many participants.
SRM-tuned. Instead of FreeSurfer, we could potentially other multi-participg alignment methods (c.g., SRM: Shared Response Modeling [Chen et al., (015]) and apply the same tuning method. One limitation of SRM is the reduced projection dimension and difficulty in controlling the included ROIS
Non-linear Heads. Lastly, rather than one FC layer to predict brain responses from the output tokens, we could use non-linear network (of 2 or more fully connected layers).
Fig reports the brain alignment improvement over the pretrained Wav2Vec2.0 for all aforementioned alternatives. It shows that our method works better than these alternatives as it allows the models to leam more robustly from information across participants.
A.4 Downstream Tasks
We elaborate here the datasets and the formulation of the downstream tasks mentioned in Sections 3.4.4 and 4.3 of the main paper.
Phonemes Prediction. Phoneme recognition is done multi-label classification problem, following the work of Moussa al. [2025). A línear classifier projects the layer representation to set of 39 possible phonemes that occurred in the original input audio segment. We use the TIMIT dataset
22

---

% A. Alignment from Max Pretrained
10
30
Loss Aug
Resporse Aug
Sepurate
Non-linear Heads
Independent
Heads
Tuned
(Ours)
Figure 7: Comparing Multi-brain-tumm taraiinnttninggggg using FreeSurfer projection and independent loss compu- tations(i.e., the loss is computed and backpropagated for each participant separately) versus other alternative techniques (refer to Sec 3.3.2 of the main paper).
Garofolo 1993]
because of its phonetically rich audio snippets. The final performance measure is the We report the Fl-score averaged over the upper-middle
classifier's FI-score the held-out layers as done for brain alignment (refer to Sec.3.4.1 in the main paper).
Phonetic Sentence Type Prediction. Predicting the phonetic sentence type be used to evaluate model's ' phonetic understanding beyond single phonemes (or words). The TIMIT dataset [Garofolo 1993 has one of three phonetic types for each utterance: SA (for utterances that cover all English phonemes), SX (for phonetically balanced utterances that cover many phones with few words), and SI (for natural and phonetically diverse utterances). Each of the three types (SA, SX, SI) highlights specific speech dialectal or phonetic aspects. To evaluate performance on this task, we follow (Moussa ct al. 20255 to predict the phonetic sentence type. We add projection elassification head predict the sentence type from the given layer's representation. The performance is measured by the Fl-score on the held-out test set, averaged across the upper-middle layers.
B Additional Results
B.1 Extended LoRA Rank Ablations
Fig 8 extends Fig 5 by adding Rank-32 updates and all model updates (fine-tuning all transformer parameters). It supports our finding that we don't need more than rank-8 updates for our method work well. Moreover, it shows that updating the entire model scales more slowly than LoRA, indicating that needs more data to reach the same performance.
B.2 Additional Brain Alignment Plots
Here, we visualize the impact of brain-tuning on brain alignment for the remaining held-out par- ticipants. Fig exttends F by showing the remaining 4 participants. Similarly to Fig observe a widespread improvement across the brain for these participants, especially the frontal and parietal regions, while the auditory cortex shows slight decrease in alignment. We attribute this decrease in auditory cortex alignment to the fact that we report alignment over the upper-middle and later layers of the models, which known be more semantic (refer Sec 4.2 in the main paper for more details).
B.3 Brain Alignment Generalization of LLM-tuning and Stimulus-tuning
In this section, we elaborate on the training details LLM-tuning and Stimulus-tuning then report on how they scale with more tuning data.
L.LM-tuning. For tuning, we use representations from layers 18 to of the I.lama--BB Model Touvron et al. 2023 instead of brain signals. These layers used because they show the best
23

---

Ramb-8
Ramb-16
Ramb-32
Rank-4
All Model
% ( in Alignment from Max Pretrained
50
60
30
10
2
8
10
12
14
16
Brain-tuning Stimulus Size (Hours)
Figure 8: Effect of the number of trainable parameters for Wav2Vec2.0 (Extending Sec 4.4 of the main paper). Increasing the number of trainable parameters with higher rank (e.g.. 32) or by fine-tuning the entire model doesn't 't lead to better scaling than rank-8 updates.
Muli-bnin-tuned
Singl-biin-uuee
M-Muneed
Stimulunttrre
%.A. in Alignment from Max Pestraine
-30
Brain-tuning Stimulus Size (Hours)
Figure Brain Alignment scaling with LLM-tuning and Stimulus-tuning of Wav2Vec2.0. This figure extends Sec of the main paper by reporting the improvement in alignment with more data for L.L.-tuning and Stimulus-tuning It shows that LL-M-tuning can help improve brain alignment with more data, but it tends to saturate and is always worse than brain-tuning. Stimulus-tunin doesn't 't seem to improve alignment and always performs comparably to the pretrained model.
alignment with late language regions. We then apply a similar fine-tuning pipeline to that of brain- tuning (detailed in Sec 3. the main paper). We use LoRA rank-8 updates, learning rate of 10 with linear decay, and a batch size of 128 samples of (audio, fMRI response) pairs. For LLM-tuning, found that it takes longer converge than brain-tuning;
train them for 250 epochs, which
takes around 10h on two NVIDIA A40 48GB GPUs.
Shimulus-tuning This baseline aims to test the benefits of brain-tuning against simply fine-tuning using stimulus audio. This highlights any improvements the model that would be solely due to seeing more data. We follow the training setting in Moussa et al.) 2025) for stimulus-tuning The same pretraining losses (the diversity loss and the contrastive loss) with the same hyperparameters of [acvski al. 2020) are used. The model is then fine-tuned for 300 epochs using base learning rate of x 10 with a warm-up for the first 10% of the updates, followed by a linear decay schedule. takes around the same amount of time to train LLM-tuning.
Next, we test whether scaling the data for LLM-tuning and Himullus-tning leads improved alignment, as we observe with brain-tuning (refer to S.c Fig shows the change in brain- alignment for brain-tuning as well as LLM-tuning and Stimulus-tuning on the training participants. LLMM-tnning improves alignment with more data, but it shows saturation trend and is always lower than brain-tuned models. Stimulus-tuned models don't show improvement over the pretrained counterpart, indicating that seeing more audio data is not the cause for the improved alignment.

---

Multi- Multi-brain-tume brain-uednt
Single-brain brain-tunndd
Training Participants
Held Porticipants
5.3 Algmon bom Pretraree
Algmeen foom Max Preraine
Six
Figure 10: Impact scaling the tuning data of brain-tuning on brain alignment for HuBERT. Similar to brain-tuned models of Wav2Vec2.1 (Sec (Sec when the tuning data scales up, models perform better than Single- Single-brain- brain-tuned ones, on both training and held-out participants.
Multi--bainn-ee traii-tuundd
Single- tuned
LLM-tuned
A. Phoneme Prediction
B. Phonetic Sentence Type Prediction
N.3. in F1 Score from Pettiind
(1.0 from
-3
Brainituning Strmulus Size (fours)
Stmules
Figure 11: Scaling of downstream performance with tuning data size for HuBERT. Brain-tuned
models' performance increases with more data, eventually matching that of the LLM-tuned model.
C HuBERT Results
C.1 Generalization Results
We repeat here the same analysis in Sec 4.2 but for the HuBERT model family. We test how the brain-tuned models generalize against increasing amounts of data during brain-tuning. This is done by measuring brain alignment improvement (relative to pretrained HuBERT) when we scale the data used for Multi- and Single-brain mangle-brain-tuinng Fig[TO shows similar trend Fi both training and held-out participants. When we increase the data used for brain-tuning, Multi- braibbbaiinnnnneee tuned models tend to perform better than Single-brain-tuned ones. for lower data fractions, the improvement of both was comparable. These results (along with their Wav2Vec2.0 parallels in Sec of the main paper) further confirm the scalability Multi-brand wer--brriiitnnngg improving alignment with new unseen participants. The upward trend also indicates the potential for further improvement if more data is integrated for brain-tuning.
C.2
Downstream Results
We report here the downstream performance of HuBERT on the same tasks detailed in Sec 4.3 and Supp A.4 Fig [T shows similar findings to Fig the main paper. When the amount of tuning data increases, brain-tuned models eventually reach the same level of performance as the LLLM-tuned one (which was shown to substantially improve performance on similar tasks by [Moussa al. 2025 Vattikonda et al.) 2025)). Moreover, for all data sizes, brain-tuned models never perform worse than their pretrained counterparts. These results (alongside their Wav2Vec2.0 equivalents in See 4.3 of the main paper) further confirm that our brain-tuning approach doesn't 'tttttn lead to catastrophic forgetting: the contrary, it leads to strong improvement in downstream performance.
25

---

Pearson Correlation
Poetrained
-..5
Participant
Participant
Participant
Participant 8
Impact of Multi-brain-tunin brain alignment for held-out participants. The figure shows the change in brain alignment (measured by Pearson Correlation) after Multi-brain-tuun compared to the pretrained Wav2Vec2.0 model. It shows a widespread improvement across the brain for these participants, especially the frontal and parietal regions.
Figure 12:
26


Reviews:
[{"review_id": "FD8QpzzJy6", "reviewer": "Reviewer_9aJn", "summary": "The authors present a robust model employing a novel method for aligning neural activity with natural language stimuli. They use both brain-tuning methods at both the individual and group levels to sucessfully generalize semantic task representations across participants. Furthermore, this model does not require all participants to be exposed to the same stimuli, and performs well in higher-order regions in the brain.", "questions": "The current model demonstrated improvement in frontal and parietal regions (when tuning with Wav2Vec 2.0) but a decrease in auditory regions. Is this an artifact of the task itself, which is cognitively demanding and would need to capture more high-level semantics?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Pros:\n\n* The model takes into account individual variability across the task and physiology of brain responses by applying brain-tuning to individual participants separately.\n\n* In order to address group-level activity, the model projects fMRI responses of multiple participants into a common space, in addition to individual tuning.\n\n* There is quite a lot of flexibility in stimuli in that participants do not need to been exposed to the same stimuli.\n\n### Cons:\n\n* (Minor comment) The generalizability may be more tuned towards brain responses to complex tasks (narrative comprehension) rather than across low-level information processing, as demonstrated by the improvements in higher-order areas in conjunction to decreased performance in auditory areas. A simulation demonstrating the robustness and generalizability of this model across different semantically complicated tasks could address this minor concern.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive evaluation and insightful feedback. We’re pleased that the reviewer recognized the strengths of our approach, including its flexibility and robustness across individuals and stimuli. Below, we address the concern regarding generalization to auditory regions in detail and provide **additional supporting analyses**.\n\n## **Clarification on Performance in Auditory Regions**\n\nThe observed slight alignment drop in auditory regions alongside improvements in frontal and parietal regions reflects the specific tuning approach we adopted:\n\n1. **Reported Layers**: The reported results in our main analysis were averaged over the upper-middle to late layers of Wav2Vec 2.0, known to encode more semantic rather than acoustic information. Thus, these layers are expected to increase in semantic rather than acoustic performance. This is also consistent with single-brain-tuning layer analyses in [1, 2].\n2. **Included Brain Regions**: Our multi-brain-tuned model predominantly includes higher-level (late language) brain areas (see Supp A.1 for details on the regions). Consequently, improvements naturally concentrate in regions capturing semantic processing, which are much larger than early auditory regions.\n\n## **Maintaining Acoustic Performance**\n\nTo further clarify the impact of Multi-brain-tuning on auditory and acoustic performance, we report below the performance of our Multi-Brain-tuned model on 1) alignment with only the auditory regions, and 2) a low-level acoustic task (namely MFCC prediction detailed in [1]). As a baseline, we compare it to a multi-brain-tuned model that was tuned using *only* the brain auditory regions (rather than both auditory and late language areas).\n\nThe tables below show percent improvement relative to the pretrained model for our Multi-brain-tuned vs Auditory Multi-brain-tuned for (i) the brain alignment of early auditory regions and (ii) performance on MFCC prediction; reported results are the mean over early to mid layers of the model (which perform best in acoustic tasks [1]).\n\n- **Table i:**\n|  | **Auditory Multi-Brain-tuned** | **Our Multi-Brain-tuned** |\n| --- | --- | --- |\n| Train participants (mean ± STE) | 4.95 ± 0.13 | 3.465 ± 0.43 |\n| Held-out participants (mean ± STE) | 2.7 ± 0.53 | -1.25 ± 1.4 |\n\n- **Table ii:**\n\n|  | **Auditory Multi-Brain-tuned** | **Our Multi-Brain-tuned** |\n| --- | --- | --- |\n| Pct Improvement | 4.48 | -0.5 |\n\nOur Multi-brain-tuned shows small positive to zero improvement, indicating **no strong negative effects on the acoustic performance** of the model despite being tuned on semantics-dominated data. Auditory Multi-Brain-tuning shows improved performance on both alignment and acoustic performance, consistent with the findings of the single-participant case in [2]. The reason for not having as huge an improvement as in late language areas is both the small size of the auditory cortex and the already high alignment of pretrained models with auditory regions (pretrained speech models are able to predict **~80%** of the estimated explainable variance in auditory regions [3]).\n\nTo sum up, substantial performance gains on high-level rather than low-level tasks is expected, but the above analysis demonstrates that the low-level utility of our Multi-brain-tuned model is either unaffected or slightly improved, alleviating the worry of a compromised low-level performance.\n\nWe appreciate the reviewer’s constructive comments and evaluation, which we believe will strengthen the manuscript. We will include these additional analyses, alongside detailed ablations exploring alignment improvements across different brain regions, in the final version of the paper.\n\n## **References**\n[1] Moussa & Toneva (2025). Brain-tuned speech models better reflect speech processing stages in the brain. *Interspeech 2025.*\n\n[2] Vattikonda et al. (2025). BrainWavLM: Fine-tuning speech representations with brain responses to language. arXiv preprint arXiv:2502.08866.\n\n[3] Oota, et al (2024). Speech language models lack important brain-relevant semantics. ACL, 2024."}, {"review_id": "O5QkmU1hKl", "reviewer": "Reviewer_cQaF", "summary": "The paper introduces a simple yet effective way to fine-tune pretrained speech models on fMRI recordings collected from several listeners who heard the same naturalistic stories. By sharing a projection head across subjects and injecting lightweight LoRA adapters, the method (i) boosts brain–model alignment relative to single-subject tuning, (ii) cuts the amount of fMRI needed per participant, and (iii) yields small but consistent improvements on two downstream speech–semantic tasks. Extensive ablations on loss functions and adapter rank support the design choices, and the authors commit to releasing code and models.", "questions": "No", "limitations": "No", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- The experimental design follows best practice, including rigorous preprocessing and strong baselines. Evaluation is thorough: the authors plot data-efficiency curves, test generalisation on held-out subjects, and probe design choices through ablations. Reproducibility is well supported by (to-be) open-sourced code. The paper itself is clearly written and well illustrated.\n\nWeaknesses\n- I found no substantive weaknesses that would motivate rejection.", "was_revised": true, "final_justification": "As other reviewers have noted, the paper applies the well-established idea of cross-subject generalization from decoding to the encoding context. Still, employing multi-brain tuning for encoding is novel, and the experiments are technically solid. I therefore maintain my Accept rating.", "rebuttal": "We sincerely thank the reviewer for the very positive evaluation and thorough understanding of our work. We’re very pleased that the reviewer recognized the clarity, strength, rigor, and effectiveness of our approach - and \"found no substantive weaknesses that would motivate rejection.\" We promise to release the full code publicly alongside the final version of the paper."}, {"review_id": "Bsp4G4IEk7", "reviewer": "Reviewer_KUd9", "summary": "The authors propose improving the data-efficiency and generalisation of brain-aligning LLMs with new subjects. They use speech perception fMRI data, transform this data into a unified representation-space across participants and propose training a projection layer and fine-tuning the pre-trained LLM with multiple new participants exposed to the same stimulus. The results provide support for these claims.", "questions": "Please see weaknesses. In general, this seems like interesting and valuable work, and I am glad to see the authors addressing this problem.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- Authors tackle a highly prevalent problem in brain decoding/encoding---how to generalise to novel subjects.\n- Well-designed experiments and evaluation methodology.\n- The method is very simple, and seems to be effective.\n- Inclusion of downstream generalisation experiments that verify alignment doesn’t degrade decoding.\n\nWeaknesses:\n- Subjects listen to the same natural stimuli, weakening the impact of generalisation from novel stimuli + novel subject generalisation to just novel subject generalisation. It would be valuable to see how the method performs on novel stimuli + novel subjects. Is this an experiment the authors could perform?\n- Lines 118-120: Are varied projection dimensions really a problem? For example, [C] use conditional projections to put variable-dimension inputs into a shared representation space.\n- The method of adding a projection head and LoRA fine-tuning the LLM is not particularly novel or insightful and has been done in similar subfields, e.g. [E]. Though the application to encoding models might be new.\n- Minor: the related work could use more contextualisation. E.g. noting that brain decoding models (even if not encoding models) have previously achieved good subject generalisation (see [A, B, C, D, E, F]). Especially [D] with fMRI. Similarly, line 69-71 suggests prior methods unifying subjects haven’t focused on speech when [A, B, F] use subject-specific linear layers, [C] use FiLM-based subject conditioning, and [G] uses source space transformations to project into a shared space. All of these focus on speech decoding. \n\n[A] Défossez, Alexandre, et al. \"Decoding speech perception from non-invasive brain recordings.\" Nature Machine Intelligence 5.10 (2023): 1097-1107.\n\n[B] d'Ascoli, Stéphane, et al. \"Decoding individual words from non-invasive brain recordings across 723 participants.\" arXiv preprint arXiv:2412.17829 (2024).\n\n[C] Jayalath, Dulhan, et al. \"The Brain's Bitter Lesson: Scaling Speech Decoding With Self-Supervised Learning.\" ICML (2025).\n\n[D] Tang, Jerry, and Alexander G. Huth. \"Semantic language decoding across participants and stimulus modalities.\" Current Biology (2025).\n\n[E] Yang, Yiqian, et al. \"Neuspeech: Decode neural signal as speech.\" arXiv preprint arXiv:2403.01748 (2024).\n\n[F] Jayalath, Dulhan, et al. \"Unlocking non-invasive brain-to-text.\" arXiv preprint arXiv:2505.13446 (2025).\n\n[G] Gideoni, Yonatan, et al. \"Non-invasive Neural Decoding in Source Reconstructed Brain Space.\" arXiv preprint arXiv:2410.19838 (2024).", "was_revised": true, "final_justification": "The authors have satisfied my main concern by conducting an experiment on an entirely new dataset with novel stimuli. They have also addressed my more minor concerns and questions about design choices for subject modelling in encoding models.", "rebuttal": "We thank the reviewer for the thorough evaluation of our work and the valuable literature suggestions. We are glad the reviewer found the problem important and the experiments well-designed. Below, we address the raised concerns in detail and **provide additional results** and clarifications. All changes described will be included in the final paper.\n\n## **Showing Generalization to Novel Subjects and Stimuli from a Different Dataset**\n\n**Experiment on novel stimuli + novel participants:** To evaluate our model’s generalization to both unseen participants and unseen stimuli, we report below the improvement in brain alignment of multi-brain-tuned and single-brain-tuned models on an entirely **new fMRI dataset** that was not used at all in our brain-tuning approach: the Narratives dataset [K]. Results were calculated over **16 participants** using only a **56 mins** stimulus. Furthermore, as a strong baseline, we compare to a Multi-brain-tuned model trained directly on the Narratives participants (i.e., multi-brain-tuning with all of the 16 participants). The table below shows the means and standard error for the percent improvement in brain alignment relative to the pretrained model. The reported P-values are to indicate statistical significance from pretrained (obtained using the paired t-test).\n\n\n|  | **Our Multi-brain-tuned** | **Single-brain-tuned** | **Narratives-multi-brain-tuned** |\n| --- | --- | --- | --- |\n| Improvement Mean ± STE | 10.38 ± 2.69 | 7.53 ± 3.15 | 13.8 ± 3.01 |\n| P-value | 0.0018 | 0.0033 | 0.00011 |\n\nOur brain-tuned models show general improvement on the new fMRI stimuli and participants from the Narratives dataset. Notably, our original Multi-brain-tuned model performs close to the  Narratives-multi-brain-tuned model. This further demonstrates the generalization and efficiency of our method on **novel subjects and stimuli with very limited data**.\n\n## **Design Choices for Multi-brain-tuning Compared to Alternatives like [C]**\n\nOur work indeed aims to create more **generalizable and efficient** brain-language models. While the method of **unifying different participants** in the same space is one of the most important design choices, it’s not the only one. In the paper, we explored several factors (data presentation/binding, loss function, number of trainable parameters) that significantly impact performance (see **Fig. 6**, Appendix A.3 for the effect of shared vs. separate stimuli presentation; **Fig. 4** for loss and LoRA rank ablations). Here, we provide **new additional analysis** comparing our approach to alternative multi-participant handling strategies:\n\n1. **Separate projection heads per participant:** We trained a variant of our model where each participant has their own projection layer (similar to the conditional projections in [C]). This drastically increases the number of parameters as participants increase.\n2. **Shared Response Model (SRM):** We applied the classic SRM [H] to the fMRI data (i.e., instead of FreeSurfer), projecting all participants’ data into a shared space as a pre-processing step, and then trained a model on this aligned data.\n3. **Non-linear projection head:** We replaced our simple linear projection with a multi-layer perceptron to test if a non-linear mapping of brain features improves alignment.\n\nWe measured the brain alignment improvement (relative to the pretrained model) on training and held-out participants for each variant:\n\n|  | **Ours (Multi-brain-tuned)** | **Separate Heads** | **SRM-aligned** | **Non-linear Head** |\n| --- | --- | --- | --- | --- |\n| **Train (mean ± SE)** | **47.22 ± 3.15** | 23.57 ± 3.80 | 21.60 ± 2.55 | 31.29 ± 5.01 |\n| **Held-out (mean ± SE)** | **33.66 ± 3.41** | 23.20 ± 6.76 | 9.80 ± 1.60 | 30.12 ± 5.73 |\n\n**Analysis:** Our Multi-brain-tuned model outperforms these alternatives on train and held-out participants. This highlights that our proposed approach is not only better, but it’s also:\n\n1. much more **parameter-efficient** than alternatives, which blow up with the number of participants; for instance, performing Multi-brain-tuning with separate heads on the Narratives data above would require 16 projection heads.\n2. more **controllable** than SRM since we can control which brain areas to tune with, unlike SRM.\n3. easier to perform across **multiple datasets** and recording devices since FreeSurfer is widely and easily adopted and doesn’t require re-fitting with new data.\n\nFinally, we reiterate that our method aims at finding the best framework to achieve generalizability and efficiency for brain encoding, which we show **does not depend on a single design choice (e.g., using LoRA)** but rather many design choices, for which we test each one against strong alternatives.\n\n## **More Clarifications on Novelty and Relevant Literature.**\n\nThank you for pointing out the relevant recent literature on multi-participant generalization in brain decoding. While our work is primarily concerned with brain encoding, we agree that discussing the recent approaches in brain decoding can help contextualize the gap in the literature and highlight our contributions better. We promise to add that in the updated version of the paper, along with the new results mentioned earlier. \n\nHowever, we want to emphasize that the generalization in brain encoding by adapting the same methods in brain decoding literature can be challenging and might not even make sense at times, as discussed in [I]. Unlike decoding, brain encoding models often fail to generalize effectively to new participants. Popular multi-participant unification approaches, such as SRM and FreeSurfer, substantially degrade encoding performance when applied across subjects (**~40%** drop as shown below; cross-participant encoding models trained on 3 participants and tested on a 4th). Even recent fine-tuning-based approaches like BrainWavLM [J] struggle with cross-subject generalization.\n\n\n| **Method** | **Single participant** | **Cross participant** | **Avg % Change** |\n| --- | --- | --- | --- |\n| SRM (brain alignment) | 0.104 ± 0.013 | 0.059 ± 0.012 | -42.5% |\n| FreeSurfer (brain alignment) | 0.12 ± 0.021 | 0.072 ± 0.009 | -40.2% |\n\nTo the extent of our knowledge, our approach is the first to demonstrate substantial improvements in speech and language encoding performance and generalization by fine-tuning with neural data, while simultaneously beating single-participant models on training participants' performance.\n\nWe appreciate the reviewer’s positive feedback, insightful comments and for helping us improve the paper. We hope that our revisions and new results have addressed your concerns, and we would be grateful if the reviewer could reflect this in their score.\n\n## **References:**\n\n[H] Chen et al. (2015). *A Reduced-Dimension fMRI Shared Response Model.*NeurIPS* 2015.\n\n[I] Beliy et al. (2025). The wisdom of a crowd of brains: A universal brain encoder. arXiv preprint arXiv: 2406.12179\n\n[J] Vattikonda et al. (2025). BrainWavLM: Fine-tuning speech representations with brain responses to language. arXiv preprint arXiv:2502.08866\n\n[K] Nastase et al. (2021). The “Narratives” fMRI dataset for evaluating models of naturalistic language comprehension. Sci Data 8, 250.\n\n."}, {"review_id": "hYnP1U0pMS", "reviewer": "Reviewer_W1Kc", "summary": "This paper proses a novel approach to improve model prediction accuracy of naturalistic fMRI data. This approach leads to two notable improvements over more traditional methods. The first is a decrease in the amount of fMRI data for new participants and the second is an increase in brain alignment. The authors accomplish them by using 2 different pre-trained speech models (wav2vec2 and HuBert) and fine tuning these models on (1) the individual participants data by adding a layer onto the model (2) on standardized data across participants.", "questions": "1. How do the advances of this paper compare to Tang & Huth and BrainWavLM? I think it would be beneficial for the author to make it clearer why this verison of the method is significant.\n\n2. Have the authors spent any time interpreting the weights from the aligned model? Is it more interpretable than other models? It seems likely that this will be a harder architecture to interpret which will limit its significance and usability in many/most research settings?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 1, "strengths_and_weaknesses": "The submission is technically sound and robust. The quality and clarity of this work is definitely it strongest point. However the originality and significance of this work is reduced by both recent publications in other outlets and the limited scholarship of the work. This papers primary results seem to highly overlap with Tang & Huth 2025. Which while a relatively new work shows many of the same results that aligning participants data decreases the amount of data required per person. The tang paper used language models instead of speech models. Similarly this paper is highly overlapping with BrainWavLM (Vattikonda et al, 2025). The primary difference being the generalizability across participants. However both of these improvements seem relatively trivial from previous work in the field. \n\nNotably, the paper heavily cites a single author's previous work, while omitting relevant related research from other groups. This raises concerns about scholarly balance and the framing of the work within the broader field. A more thorough literature review that situates this contribution among alternative or complementary approaches would strengthen the paper and make its novelty clearer to readers outside the authors’ own research lineage. This work shows a strong bias towards ML papers in its citations over neuroscience papers, which will limit its reach. In particular this work fails to cite Tang & Huth 2025 (as mentioned above) which shows similar decreases in the amount of fMRI data needed to predict brain data and an increase in robustness when aligning across participants. Thus the primary results of this paper have been shown already in the literature with another method. The authors do not at any point address this redundancy or highlight differences. Similarly the authors claim that accurately estimating brain alignment has been shown to require a large amount of per-participant data (line 66-67) when this has been shown in work by authors other than Antonello first (e.g. LeBel et al 2023, Lebel et al 2021). \n\nAs a small note, the authors refer to LMs as \"model organisms\" (line 24). \"model organisms\" is already a standard term in the field referring to biological organisms (i.e. flies or mice). Using it here to suggest LMs is likely to be misleading. The term \"models\" is likely clearer and more accurate.", "was_revised": true, "final_justification": "I think this is a technically solid paper that lacks novelty. However given most reviewers seem to recommend accept and that the paper does appear to be technically solid, I have chosen to raise my recommendation to a borderline accept. The authors have also done a good job integrating and addressing comments from reviewers.", "rebuttal": "We thank the reviewer for their review of our work and literature suggestions. Below, we clarify the distinctions of our contributions compared to the mentioned previous work (further supporting it with new results), address interpretability concerns, and discuss the suggested relevant literature.\n\n## **Q1: Novelty Compared to Recent Work**\n\n\n### **A. Distinction from Brain decoding work by Tang & Huth (2025)**\n\nWe appreciate the pointer to the recent decoding work by Tang & Huth (2025). However, we want to highlight that this study investigates generalization in brain decoding rather than brain encoding (and is not the first to show decoding generalization). The two domains differ a lot, and transferring methods between them can be challenging.\n\nTang & Huth (2025) demonstrate strong generalization across stimulus modalities, extending previous decoding successes in speech [1], language [2, 3, 4], and vision [5, 6, 7] in unifying participant responses and generalizing to new participants. Some of these methods show impressive generalization [8, 9], including zero-shot decoding [10]. Despite these advances, achieving similar generalization in brain encoding remains **challenging and unresolved**. Previous attempts typically resulted in substantial performance drops compared to single-subject models (see below). Adapting decoding approaches directly may not be straightforward or even feasible for encoding tasks, as discussed in [11]. This emphasizes the non-triviality of generalization in brain encoding,  an important gap that our work addresses.\n\n### **B. Challenges in Brain Encoding Generalization**\n\nUnlike decoding, brain encoding models typically fail to generalize effectively to new participants. Common participant-alignment methods, such as SRM[22] and FreeSurfer, substantially degrade encoding performance (**~40%** drop, as shown below; cross-participant encoding models trained on 3 participants and tested on a 4th). Even recent fine-tuning approaches like BrainWavLM [15] show around a **30%** drop in cross-participant performance (where the speech model is fine-tuned on one participant, but the encoding model is trained and evaluated on others).\n\n| **Method** | **Single participant** | **Cross participant** | **Avg performance drop** |\n| --- | --- | --- | --- |\n| SRM (brain alignment) | 0.104 ± 0.013 | 0.059 ± 0.012 | 42.5% |\n| FreeSurfer (brain alignment) | 0.12 ± 0.021 | 0.072 ± 0.009 | 40.2% |\n\nOur approach is, to our knowledge, the first to demonstrate significant improvements and generalization in speech and language encoding by fine-tuning with neural data. Crucially, our method simultaneously improves performance on both trained and held-out participants, addressing an important gap unresolved by prior work.\n\n### **C. Comparison with BrainWavLM (Vattikonda et al., 2025)**\n\nBoth Brain-tuning [14] and BrainWavLM [15] introduce a framework for fine-tuning speech models with fMRI data, with Brain-tuning [14] being the first such work. However, neither paper offers a way to adapt to multi-subject or multi-dataset settings; they also show limited transferability across subjects. We emphasize that successful generalization across participants requires extensive, careful design choices and **is not a trivial extension of those works**. Specifically, our results demonstrate:\n\n1. **The method of aggregating and presenting multiple participant responses** critically influences generalization (Fig. 6, Supp A.3).\n2. **Loss function and model capacity** substantially impact performance and generalization ability (Fig. 4).\n3. ***new results,* Investigating different methods for Multiple participant handling**: In order to further demonstrate the effectiveness of our design choices, we explore the performance of different multiple participants handling methods. Specifically, we compare our Multi-brain-tuning against three alternatives: (i) separate participant-specific projection heads (as in prior work [23]), (ii) a non-linear projection network (MLP instead of linear), and (iii) SRM-based participant alignment (instead of FreeSurfer). The table below summarizes alignment improvements for training and held-out participants.\n\n|  | **Our Multi-brain-tuned** | **With Separate Heads** | **With SRM** | **With Non-linear Projection** |\n| --- | --- | --- | --- | --- |\n| Train (mean ± STE) | **47.22 ± 3.15** | 23.57 ± 3.8 | 21.6 ± 2.55 | 31.29 ± 5.01 |\n| Held-out (mean ± STE) | **33.66 ± 3.41** | 23.2 ± 6.76 | 9.8 ± 1.6 | 30.12 ± 5.73 |\n\nOur Multi-brain-tuned model outperforms all alternatives while being:\n\n- more parameter-efficient compared to separate heads or non-linear alternatives, which scale poorly with participant number (e.g., 16 separate projection heads for Narratives).\n- more controllable in selecting targeted brain areas unlike SRM.\n- easier to apply across multiple datasets due to FreeSurfer’s widespread adoption.\n\n## **Q2: Interpretability of Multi-Brain-Tuned Models**\n\nWe argue that our multi-brain-tuned models are **at least as interpretable** as their pretrained counterparts. Our method fine-tunes the full architecture, unlike BrainWavLM [15], which removes some of the model layers. Thus, existing interpretability analyses for speech models (layer-probing, attribution, representation analysis) remain fully applicable.\n\nMoreover, understanding and interpreting why models align with the brain in the first place is an active research area ([16, 18, 19, 20, 21]. Methods developed in this context apply equally to our Multi-brain-tuned models, potentially yielding further insights into the effect of increased neural alignment. For instance, recent work [16] demonstrates that speech brain-tuned models’ layers reflect hierarchical brain processing stages better than pretrained models. Similar analyses can be easily employed on our Multi-brain-tuned models, and we hope that our work inspires such future work.\n\nIn essence, since our models can be used, analyzed, and deployed in the same manner as their popular pretrained counterparts, our approach wouldn’t complicate interpretability analyses of these models. \n\n## **Q3: Revising Literature and Terminology**\n\nWe appreciate the reviewer’s feedback regarding scholarly literature balance. We note, nevertheless, that our related work discussion covers relevant brain encoding work, and that our repeated comparison to [14, 15] is because they are the only recent works related to brain-tuning of speech models. However, we agree that expanding our literature review to integrate more brain decoding and neuroscience-focused work (such as Tang & Huth, 2025, and others mentioned earlier) will better contextualize our contributions.\n\nRegarding terminology, our use of \"model organisms\" aimed at highlighting parallels with biological research models, distinguishing our usage from general AI \"models.\" This term has been similarly employed in recent survey literature [17]. Nevertheless, we understand the potential confusion and will clarify this usage in the manuscript.\n\nWe greatly appreciate the reviewer’s feedback and believe these clarifications significantly strengthen the paper. We hope these revisions clearly highlight the contributions, novelty, and interpretability of the work, and hope these improvements are reflected positively in your evaluation.\n\n## **References**\n\n[1] Défossez et al. (2023). Decoding speech perception from non-invasive brain recordings. *Nature Machine Intelligence*.\n\n[2] Lévy et al. (2025). Brain-to-Text Decoding: A Non-invasive Approach via Typing.\n\n[3] d’Ascoli et al. (2024). Decoding individual words from non-invasive brain recordings across 723 participants.\n\n[4] Liu et al. (2025). Learning interpretable representations leads to semantically faithful EEG-to-text generation.\n\n[5] Benchetrit et al. (2024). Brain decoding: toward real-time reconstruction of visual perception. *ICLR*.\n\n[6] Liu et al. (2023). BrainCLIP: Bridging brain and visual-linguistic representation via CLIP for generic natural visual stimulus decoding.\n\n[7] Chen & Wei (2024). Mind’s Eye: Image recognition by EEG via multimodal similarity-keeping contrastive learning.\n\n[8] Thual et al. (2023). Aligning brain functions boosts the decoding of visual semantics in novel subjects.\n\n[9] Dahan et al. (2025). SIM: Surface-based fMRI analysis for inter-subject multimodal decoding from movie-watching experiments. *ICLR*.\n\n[10] Aristimunha et al. (2025). EEG Foundation Challenge: From cross-task to cross-subject EEG decoding.\n\n[11] Beliy et al. (2024). The wisdom of a crowd of brains: A universal brain encoder.\n\n[12] Schwartz et al. (2019). Inducing brain-relevant bias in natural language processing models. *NeurIPS*.\n\n[13] Lu et al. (2024). Achieving more human brain-like vision via human EEG representational alignment.\n\n[14] Moussa et al. (2025). Improving semantic understanding in speech language models via brain-tuning. *ICLR*.\n\n[15] Vattikonda et al. (2025). BrainWavLM: Fine-tuning speech representations with brain responses to language.\n\n[16] Moussa & Toneva (2025). Brain-tuned speech models better reflect speech processing stages in the brain. *Interspeech*.\n\n[17] Oota et al. (2023). Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey). *TMLR*.\n\n[18] Caucheteux & King (2022). Brains and algorithms partially converge in natural language processing. *Communications Biology*.\n\n[19] Merlin & Toneva (2022). Language models and brains align due to more than next-word prediction and word-level information.\n\n[20] Oota et al. (2023). Joint processing of linguistic properties in brains and language models.\n\n[21] Oota et al. (2024). Speech language models lack important brain-relevant semantics. *ACL*.\n\n[22] Chen et al. (2015). A Reduced-Dimension fMRI Shared Response Model. *NeurIPS*.\n\n[23] Jayalath et al. (2025). The Brain’s Bitter Lesson: Scaling Speech Decoding With Self-Supervised Learning. *ICML*."}]

Author rebuttal:

Meta-review: This paper presents a simple approach for improving Model-Brain alignment in speech models, which involves using the representation from the speech model to predict fMRI responses from multiple participants. This type of "multi-task" alignment is shown to improve performance and data efficiency in two popular speech models, namely Wav2Vec and HuBERT.

This is a well-executed paper that shows that a simple technique helps improvement alignment. It was especially impressive to see that this approach does not hurt downstream performance, and sometimes even helps. While some reviewers initially noted that the use of the same stimuli is a strict restriction, during the rebuttal the authors showed that the method can generalize to new stimuli. One potential weakness identified by a reviewer was with regard to originality. The authors (in my opinion) adequately responded to this point, but I encourage the authors to incorporate this in the camera ready.

Overall, this paper is a clear accept.
Decision: Accept (poster)