Brain-Informed Fine-Tuning for Improved Multilingual Understanding in Language Models
Authors: Anuja Negi
SUBBA REDDY OOTA
Anwar O Nunez-Elizalde
Manish Gupta
Fatma Deniz
Venue: NeurIPS 2025 poster ()
Keywords: fMRI
bilingualism
fine-tuning
language models
brain-tuning
brain encoding
multilingual language models
downstream tasks
Primary area: neuroscience_and_cognitive_science

Abstract: Recent studies have demonstrated that fine-tuning language models with brain data can improve their semantic understanding, although these findings have so far been limited to English. Interestingly, similar to the shared multilingual embedding space of pretrained multilingual language models, human studies provide strong evidence for a shared semantic system in bilingual individuals. Here, we investigate whether fine-tuning language models with bilingual brain data changes model representations in a way that improves them across multiple languages. To test this, we fine-tune monolingual and multilingual language models using brain activity recorded while bilingual participants read stories in English and Chinese. We then evaluate how well these representations generalize to the bilingual participants’ first language, their second language, and several other languages that the participants are not fluent in. We assess the fine-tuned language models on brain encoding performance and downstream NLP tasks. Our results show that bilingual brain-informed fine-tuned language models outperform their vanilla (pretrained) counterparts in both brain encoding performance and most downstream NLP tasks across multiple languages. These findings suggest that brain-informed fine-tuning improves multilingual understanding in language models, offering a bridge between cognitive neuroscience and NLP research. We make our code publicly available.

Brain-Informed Fine-Tuning for Improved Multilingual Understanding in Language Models
Anuja Negi"
Subba Reddy Oota"
Technical University of Berlin Berlin, Germany 10623
Technical University of Berlin Berlin, Germany 10623
anuja. megiittu-eeriing
subba.reddy www.atttberrlnnn........
Anwar O Nunez-Elizalde anvarnunzZQmail..com com
Manish Gupta Microsoft Research Hyderabad, India 500032 gman gmanish0mi crosoft.o....oooo com
Fatma Deniz
Technical University of Berlin Berlin, Germany 10623
deniz@tu-bers certttuuberiii..
Abstract
Recent studies have demonstrated that fine-tuning language models with brain data can improve their semantic understanding, although these findings have far been limited to English. Interestingly, similar to the shared multilingual em- bedding space of pretrained multilingual language models, human studies provide strong evidence for a shared semantic system in bilingual individuals. Here, we investigate whether fine-tuning language models with bilingual brain data changes model representations in way that improves them across multiple languages. To test this, we fine-tune monolingual and multilingual language models using brain activity recorded while bilingual participants read stories in English and Chinese. We then evaluate how well these representations generalize to the bilingual partici- pants' first language, their second language, and several other languages that the participants are not fluent in. We assess the fine-tuned language models on brain encoding performance and downstream NLP tasks. Our results show that bilingual brain-informed fine-tuned language models outperform their vanilla (pretrained) counterparts in both brain encoding performance and most downstream NLP tasks across multiple languages. These findings suggest that brain-informed fine-tuning improves multilingual understanding in language models, offering bridge between cognitive neuroscience and NI.P research. We make our code publicly available.
Introduction
Recent research has demonstrated that representations extracted from text-based Transformer lan- guage models be used to accurately predict human brain activity evoked during language processing, suggesting parallels between artificial and brain language representations (Wehbe et al.) 2014b Jain & Huth, 2018, Toneva & Wehbe 2019; Schrimpt al. 2021 Caucheteux & King
2022; Goldstein et al. [2022] Karamolegkou et 2003; Oota et al. 2025). Although these mod- els accurately predict patterns of brain activity, they have not been originally pretrained do so. Several previous studies hypothesized that fine-tuning language models with brain data can lead to representations that are better aligned with brain activity (Schwartz et al. 2019, Moussa al.
2025
Vattikonda et al. 2025). However, these efforts have largely focused on monolingual language
Equal contribution
http://ttth..c/eeeeeeeeee///aaaa tuning
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Training stimulus
Brain-informed Fine-tuning
Predicted
fMRI activity
"theeeie some scientets who soy the universe that we inhobit i
Unvon Proprroooo Poppetooo
OR
ANB- tie
Pooling
Language model
Deloy
185
weight update
+
Loss
TRS
Recorded fMRI activity
Figure 1: Brain-informed fine-tuning pipeline. Participants read naturalistic stories in English or Chinese while fMRT responses were recorded. The corresponding transcripts were fed into a pretrained language model. Representations from the final token of the last hidden layer were downsampled and temporally delayed to align with the fMRI acquisition. These were then projected to voxel space via linear layer to predict brain activity. The loss between predicted and recorded responses was backpropagated through all layers for full model fine-tuning.
models and monolingual brain data, usually trained and evaluated only in English. This overlooks the widespread prevalence of bilingualism and multilingualism in the human population (Grosjean)
2024). The limitation is particularly significant given recent neuroscientific evidence revealing that bilingual individuals have shared semantic representations across languages (Chen al. 2024b
Francis 2005), hinting at a shared component for the processing of different languages in the human brain (de Varda al. 2025). Complementary NLP research has shown that multilingual language models operate in a shared, banguage-agnoste conceptual space (Wendler et al.) [0244] Schut et al.) 2025). This raises the question of whether fine-tuning language models with bilingual brain data can elicit multilingual capabilities in language models.
brain recordings of bilingual participants reading the same naturalistic stories in English and Chinese, from (Chen et al. 2024b), and investigate whether fine-tuning language models with brain data from bilingual participants elicit multilingual capabilities in language models. We first introduce novel, end-to-end brain-informed fine-tuning pipeline (as shown in Fig. [T. which can be trained using data from the whole-brain, or from language- or sematiiclllyy selective brain regions. Second, we use brain recordings of bilingual participants to fine-tune two monolingual language models (BERT for English and Chinese (Devlin al.) 2019) and four multilingual language models (mBERT (Devlin et al. 2019), XLM-R (Conneau et al. 2020), XGLM (Lin et al., (022). LLaMA-3.2 (Touvron al. 2023)). Third, we investigate how monolingual and multilingual language models change after fine-tuning with bilingual brain data. We evaluate this along two axes: brain encoding and downstream NLP task performance. To assess generalization across languages, we evaluate the brain-informed fine-tuned language models on downstream NLP benchmarks not only in the models' fine-tuned language but also the bilingual participants' other language (cross-language transfer between known languages) and several other languages that the participant is not fluent in (and also not used in fine-tuning mne-tuning; referred to as 'unseen languages') ) c.g., German and French. Fine-tunin fine-tuning is done separately for each participant, and we assess encoding performance both within the same participant and across the other participants.
In this study,
Brain- dein--informed fine-tuning with bilingual brain data reveals several key conclusions: (1) Brain encoding performance improves after fine-tuning, even when evaluated on participants not used in the training set. This suggests that the brain bias introduced is not specific to a particular participant but rather reflects a shared component of bilingual representations.
(2) Both monolingual and multilingual language models show improved performance on downstream NL.P tasks in the fine- tuned language, the participant's 's other language, and in several other unseen languages, indicating that fine-tuning elicits generalizable semantic structure not tied to any one language. (3)

---

To ascertain whether our results are specific to fine-tuning with bilingual brain data, and not general outcome of fine-tuning language models with brain data, we performed the same analyses using brain data from monolingual individuals. Results suggest that the observed effects are indeed driven by bilingual brain representations. These findings suggest that bilingual brain-informed fine-tuning improves multilingual understanding in text-based language models. Our results contribute to the alignment between brain and artificial multilingual language representations, offering insights into the development of brain-inspired multilingual NLP systems.
We make the following contributions: (1) To the best of our knowledge, this the first study to perform informed fine-tuning using bilingual brain data, applying it to both monolingual and multilingual language models. (2) We introduce a novel brain brain-informed fine-tuning pipeline that explicitly models the temporal component of brain activity. This contrasts with previous fine-tuning studies, where these are implemented as preprocessing steps before fine-tuning the model. (3) We evaluate the performance of brain-informed fine-tuned monolingual and multilingual language models on downstream NLP tasks in both English and Chinese. The code is publicly available 2
2 Related Work
Fine-tuning of language models with naturalistic brain data. Our work builds the brain- tuning approach introduced by /Schwartz et al. (2019): [Moussa et al. (2025); Vattikonda et al. (2025). )
which fine-tunes pretrained Transformer-basee language models using brain data to integrate brain-
relevant information. [Scwwarzz et al. (2019) demonstrated improved brain encoding and NLP task performance using brain data from monolingual English readers, while Moussa et al. (2025);
Vattikonda et al. (2025) extended this to speech-based models to enhance semantic representations.
fine-tuning and analyzing how monolingual and multilingual models change when trained with bilingual brain data.
Our study complements these by exploring bilingual
Multilingual language models and brain alignment. Our work aligns with growing body of research examining the alignment between human brain activity and language models. Several studies have shown that text-based models can predict brain responses to written and spoken stimuli with
high accuracy (Wehbe et al.] 2014a] Jain & Huth 2018 Toneva & Wehbe [019 Deniz et al. 2019
Abdou et al. 2222 Toneva al. [0022] Antonello al.
2021 Oota et al. [022] Aw & Toneva 2023
Oota et al. 2024b Lamarre et al. 2022 Chen et al.
2024a). Recent efforts have extended this to
multilingual Transformer-based models using brain data from tasks in multiple languages (de Varda
cct al. 2025), though most studies remain monolingual in design, with the exception of Chen et al. (2024a), who explored bilingual processing using similar English and Chinese stimuli. Our study supplements this by investigating bilingual brain alignment through brain-informed fine-tuning and its impact on downstream NLP tasks in both languages. Detailed related work is in Appendix [A
3 Methodology
3.1 Naturalistic Brain Imaging Dataset
Bilingual [MRI dataset. Blood oxygen level-dependent (BOLD) responses were collected using MMRI from six bilingual participants (three males, three females), all native speakers of Mandarin Chinese and fluent in English as a second language. Participants read 11 narrative stories from The Moth Radio Hour (Huth et al. 2016). presented word-by-word in separate scanning sessions for English (en) and Chinese (zh). This dataset is taken from Chen et al. (2024b). Each story is 10-15 minutes long. Of the 11 stories, seven were used for fine tuning the language models (covering 2756 TRs l. three were used for training voxelwise encoding models (1117 TRs), and one story was reserved for testing (291 TRs). The same of stories were used in each language ensure comparability across languages.
Monolingual [MRI dataset. To test whether the effects of brain-informnn fine-tuning are specific to bilingual brains, we compare models fine-tuned with IMRI data from three English-monolinge participants (all male; one from Deniz et al. (2019) and two from LeBel et al. (2023)). The participants read or listened to the same English narrative stories under similar experimental designs.
3 3TR"" stands for repetition time, which is the acquisition time for each IMRI volume; in our case, the TR
was 2.0045 seconds.
3

---

This allows direct comparison between models fine-tuned with bilingual and monolingual brain data. If similar improvements are seen with monolingual brain fine-tuning, it would suggest that the effects are driven by general brain representations rather than shared semantic representations in bilinguals. More details about datasets and preprocessing are Appendix B
3.2 Text-based Language Models
We fine-tuned monolingual and multilingual text-based Transformer language models investigate the effects of brain-informed fine-tuning. All pretrained model checkpoints were obtained from Hugging Face (Wolf et al., 2020).
Monolingual pretrained language models. We fine-tuned two monolingual models: English BERT (BERT-en) and Chinese BERT (BERT-zh) (Devlin et al.) 2019). Both models share the same architecture, comprising 12 Transformer layers with a hidden dimension size of 768, differing only in the language of their pretraining corpora.
Multilingual pretrained language models. We fine-tuned four multilingual Transformer- based models: mBERT (Devlin et al., 2019), XLM-R (Conneau al., 2020), XGLM (Lin et al. (022). and LLaMA-3.2 Touvrro et al. 2023). These models represent three distinct architecture types: encoder-based (mBERT, XLM-R), R), cross- poos-lingual pretrained (XGLM), and decoder- based (LLaMA- 3.2). Representations were extracted from the base versions of mBERT, XLM-R, and XGLM, and from the IB parameter version of LLaMA-3.2. mBERT and XLM-R consist of 12 layers with hidden dimension size of 768, XGLM has 24 layers with a hidden dimension size of 1024, and LLaMA-3.2 has 16 layers with hidden dimension size of 2048. Results and analyses using XLM-R, XGLM and LLaMA-3.2 are reported in the Supplementary sections 2 and 3.
3.3 Brain-informed Fine-Tuning
We performed supervised full fine-tuning of pretrained language models using fMRI BOLD responses as targets. We employ small-N design, where language model is fine-tuned and evaluated independently for each participant, enabling robust within paatiii-atiiiiannt inference (Smith & Little 2018). An overview of the fine-tuning pipeline is shown in Fig. [ We fine-tune language models using fMRI responses from either the whole-brain language- selective firom/Fedorenke et al. (2010), waaaaniicllllslleetiiiee brain regions (see Appendix Fig. B) Details on fine-tuning with language- selective and comantically-selettivv brain regions are described in Appendix C
Model architecture. Text transcripts of the narrative stimuli were provided as inputs to the pretrained language model, using a sequence length 20 tokens. We then extract the representations of the last token from the last hidden layer for each word. This representation is given as input a dropout layer (a dropout of 0.2 is used to mitigate overfitting on limited brain data). This is followed by differentiable 3-lobe Lanczos (Huth et al. 2016) interpolation to downsample the text to the fMRI sampling rate. To model the temporal hemodynamic response, we learn a finite impulse response (FIR) filter by concatenating representations delayed by 2, 4. 6, and 8 seconds. The resulting representations are then passed through a linear projection layer to predict voxelwise BOLD responses. Previous brain-based fine-tuning studies (Moussa al. 2025 Vattikonda et al. 2025) typically treated downsampling and temporal-debe modeling as separate preprocessing steps before model fine-tuning. In contrast, our method integrates these components directly into the model architecture, enabling end-to-en optimization. This design supports flexibility in handling variable-length inputs (as word timing per TR vary) and allows for broader generalization across modalities and tasks. Comparison with previously proposed fine-tuning pipeline is provided in the Supplementary section 4.
Training protocol. We fine-tuned the models using the AdamW optimizer (Loshchilov & Hutter 2017) with a learning rate of lc-4 and weight decay of 1c-3 for 30 epochs with a batch size of 32. A PiduceLLOnPPltteer scheduler was used to adjust the learning rate based validation loss. We used mixed-precision training for computational efficiency and applied early stopping with patience of epochs based on validation performance. Implementation details in Appendix D
Training objective. Our training objective minimized the NT-Xent (Normalized Temperature-Scales Cross-entropy) loss (Sohn) 2016) between the predicted and actual BOLD responses. We also experimented with alternative loss functions such as MSE, ridge loss, spatial loss, and a combination of all (referred to hybrid), and found NT-Xent to perform best across models and experiments.

---

Results from other losses are reported in Supplementary section 1. language models and propagated the loss backward to update both the projection and transformer layers.
We fine-tuned all layers of the
In addition to comparing our brain-informed fine-tuned models their pretrained counterparts in both monolingual and multilingual settings, we include several baselines for a comprehensive comparison: TR-shuffled IMRI, multilingual model representations, and monolingual brain data as fine-tuning largets. This experiment was done to clarify the specific impact of bilingual brain data on the resulting fine-tuned model representations. Details about the baselines are reported in Appendix
3.4 Voxelwise Encoding Modeling
To evaluate whether brain-informed fine-tuning alters the alignment between language model repre- sentations and brain activity, we assess voxelwise encoding performance before and after fine-tuning. Voxelwise encoding models (VEM) estimate, for each voxel, a linear mapping from an embedding to the observed fMRI responses (Huth et al., 2016 Deniz et al. 2019). If fine-tuning with bilingual brain data improves encoding performance. we interpret this as evidence that the language model's internal representations have become more brain- like, indicating successful alignment. If encoding performance remains unchanged, this suggests that fine-tuning preserves the existing brain alignment of the representations without introducing degradation. A decrease in performance would imply that fine-tuning introduces distortions that reduce the language model's s ability to predict brain responses, thus reducing its alignment with the brain.
We used representations extracted from layer (found to yield the highest encoding performance based on validation data) as the embedding. Embeddings and brain responses were z-scored across time separately for each of the three stories not used for fine-tuning Embeddings were downsampled to match the fMRI acquisition rate using 3-lobe Lanczos interpolation. Next, to account for the delayed hemodynamic response, each embedding was passed through a finite impulse response (FIR) filter with four delays. Specifically, delayed copies of each feature at 1, 2, 3, and 4 TRs (2, 4, 6, and 8 seconds) were concatenated. Ridge regression was used to determine how the embedding is represented in each voxel (Wu et al. 2006 Naselaris et al.) 2011). We performed 5-fold cross-validation to find optimal regularization parameters. Encoding performance was quantified by calculating the Pearson correlation coefficient (r) between predicted and recorded BOLD responses using a held-out story. A separate VEM was fit for each voxel, participant, and language model variant (before and after grain-informed fine-tuning). No sample size calculations were performed, as each participant serves as a full replication of the results.
Cross- passsppaticipant transfer of encoding performance. We evaluated whether informenformed fine- tuned model representations generalize across participants. We fine-tuned the language model using brain data from one bilingual participant and evaluated its encoding performance using VEMs on other participants. Improved transfer would suggest that changes introduced by brain-informed fine-tuning are not participant-specific but reflect shared representations across bilingual individuals.
3.5 Downstream NLP tasks
To evaluate how brain-informed fine-tuning changes language model representations, we assessed its performance on standard NLP benchmarks before and after fine-tuning. For English, we used tasks from the GLUE benchmark (Wang et al., 2018), and for Chinese, we used 7 tasks from the CLUE benchmark (Xu et al. 2020). Both monolingual and multilingual models were evaluated on these benchmarks. To investigate cross-linguistir generalization to unseen languages. we additionally evaluated multilingual models on 3 tasks from the XGLUE benchmark (Liang ct al. (020)) and tasks from the XTREME benchmark (Hu et al. 2020). Details about these benchmarks and task metrics are in Appendix B
Fine-tuning and evaluation in the same language. To assess how brain-informed fine-tuning of language models affects performance on NLP tasks within the same language, we fine-tuned language models using fMRI data in English or Chinese. We then evaluated these language models on NLP benchmarks the corresponding language. English and Chinese brain data are used to fine-tune monolingual (BERT-en, BERT-zh) and multilingual (mBERT) language models, which are then evaluated on GLUE and CLUE, respectively. Any performance increase observed in this setting would imply that brain informedd fine-tuning not only changes language model representations to
5

---

English (en)
Chinese (zh)
BERT
cererte
MBERT e
langage repuro
wonornocseiiio DINE
Breuding
HIFS
Figure 2: Brain encoding performance before and after brain-informed fine-tuning. We evaluated the effect of bilingual brain-informed fine-tuning using voxelwise encoding models (VEMs; see Methods). VEMs were trained using representations from both the vanilla (pretrained) model and their bilingual brain-informed fine-tuned variants (using whole-brain or using language, or cemantically-seletii brain regions). Flattened cortical surface for one participant is shown. Each voxel on the cortical surface is colored according to which model variant achieved the best encoding performance (r) for (a) BERT and (b) mBERT with English (left) and Chinese (right) brain data, respectively. Each point corresponds to a voxel that was consistently well predicted (r>0.1) across models. Voxel colors reflect the best-performing model: black for vanilla, blue for fine-tuned with brain
whole-brain, magenta for fine-tuned with language, and yellow for semantically-seleette regions. Across both languages and model types, bilingual brain-informed fine-tuning consistently yields better encoding performance than its vanilla counterpart. Results for other participants are in Appendix F2
better align with the brain, but that these modified representations are better for NLP tasks within trained language.
Cross-language transfer between known languages. To test whether train-inffrmmed fine-tuning transfers to bilingual participant's other language, we evaluate models on the participant's 's other language after tine-tuning on the first. For example, weights from BERT-en fine-tuned with English brain data are transferred to BERT-zh and evaluated on Chinese tasks, and vice versa. Improved performance in this setting would suggest that brain-informed fine-tuning introduces the bilingual brain's shared semantic representations into the model, enabling cross-language transfer.
Zero-shot transfer unseen languages. To evaluate the broader generalizability of brain-informed fine-tuned models, we tested multilingual models on five additional languages (German, French, Spanish, Japanese, and Korean) that were neither used for fine-tuning nor known the participant. Specifically, we fine-tuned mBERT on English brain data and evaluated its performance on the XGLUE (Liang al., (2200) and XTREME (Hu et al.) (0200 benchmarks. This setup allows us to assess whether infonmilformed fine-tuning can introduce language-agnose representations that support broader cross-linguistic transfer.
Results
4.1 Effects Brain Encoding Performance and Generalization
We tested whether brain-informed fine-tuning improves brain encoding performance and if these possible improvements generalizable across participants. We compared the VEM performance

---

of the vanilla language model against the three fine-tuned variants, trained using whole-brain or language-selective or participant-speeffie semantically-selective voxels. Fig. 2 shows flattened cortical surfaces for participant 1, visualizing which language model variant achieved the best encoding performance across well-predicted voxels (Pearson correlation r>0.1) ) for (a) monolingual and (b) multilingual models within the same language. Across both languages and model types, we found that fine-tuned language models outperform their vanilla counterparts, with the best-performing variant explaining more variance in majority of voxels (76-84% of well-predicted voxels). This was consistent across all six participants (see Fig. 5 in Appendix [+2 for VEM results on other participants). Our results suggest that language model's ability to predict brain activity benefits from -nfornen fine-tuning. Further, as shown in Appendix [..T] brain-informed fine-tum tuning outperforms control baselines (like TR-shufile fMRI responses and mBERT representations as fine-tuning targets).
We do not observe any regional preference for any particular fine-tuning approach. No specific brain region consistently preferred one fine-tuned variant over the others across participants. Although fine-tuning improved encoding performance, the overall increase was modest (maximum Ar R 0.15), which aligns with prior studies. For example, previous work has shown that current representations from language models already serve as one of the best predictors of brain responses (Schrimpi et al., 202I), and even in speech models that lack brain-relevant semantics (Oota et al. 2024a), fine-tuning yielded small improvements (Ar = 0.1) (Vattikonda et al. 2025).
To test whether these improvements were participant-specifee we checked for cross-partie cross-participant generalization. We fine-tuned a BERT-en model on one participant's 's English brain data and evaluated its VEM performance on the other participants. As shown Fig. [6] ii Appendix FF3 we observe small but consistent improvements (Ar z 0.03 - 0.05) in high-level semantic areas. These findings suggest that the improvements from brain-informed fine-tuning of language model representations are not specific to individual participants but reflect shared representations across bilingual individuals.
4.2 Performance on Downstream NLP tasks
We tested the effect of brain-informed fine-tuning (with semantically-selectvee brain regions) on language models by evaluating model performance on standard NLP benchmarks. We assess this in three settings:
Fine-tuning and evaluation in the same language. We show in Table Ta) performance on down- fine-tuning (with semantically-ssele selective brain regions) and
stream NLP tasks when evaluation are computed in the same language. For English, models were fine-tuned using English brain data and evaluated on the GLUE benchmark. We observe that the monolingual model (BERT- ft-en) outperforms its vanilla counterpart (BERT-en) on 7/9 tasks, with an average improvement of +0.80 percentage points, and a maximum gain of +3.57 on the WNLI task. The multilingual model (mBERT-ft-en) improves on 7/9 tasks, with an average gain of +0.89 percentage points, and max gain of +3.12 also on WNLI. For Chinese, models were fine-tuned using Chinese brain data and evaluated on the CLUE benchmark. Here, BERT-ft-zh achieves slight improvements on 5/7 tasks relative to BERT-zh, with an average gain of +0.65 percentage point and max gain of + +1.23 on the CSL task. The multilingual model (mBERT-ft-zl) also improves on 6/7 tasks, achieving an average improvement of +0.53 percentage point, with the largest improvement of +1.02 observed on the C task. These results indicate that brain-informed fine-tuning consistently improves downstream task performance for both monolingual and multilingual models when the fine-tuning and evaluation languages match.
Cross- Pross-language transfer between known languages. Table [[b) reports performance when brain- informed fine-tuning (with semantically-selcctin brain regions) is done in one language and eval- uation is performed in the participant's other language. On the GLUE benchmark, we evaluated BERT-zh and mBERT models fine-tuned using Chinese brain data. We observe that the monolingual model (BERT-ft-zl) ft-zz) outperforms its vanilla counterpart (BERT-zh) on 8/9 tasks, with an average improvement of +1.01 percentage points, and maximum gain of +2.82 on the WNLI task. The multilingual model (mBERT-ft-zh) improves on all tasks, with an average gain of +1.61 percentage points, and max gain of +2.96 on MRPC (Acc.). On the CLUE benchmark, we evaluated BERT-en and mBERT models fine-tuned using English brain data. BERT-ft-en shows improvement on 4/7 tasks, with mean improvement of +0.44 points overall (max gain: +0.63 on TNEWS (FI)). mBERT-ft-co
7

---

Table 1: Downstream task performance before and after bilingual brain-informed fine-tuning (with semantically-seleetive brain regions). We compared vanilla (pretrained) models, English BERT (BERT-en), Chinese BERT (BERT-zh), and mBERT, with their bilingual brain-informed fine- tuned counterparts (e.g., BERT-ft-en: BERT-en fine-tuned using English brain data from semantically- selective brain regions). For each task, the average performance and standard deviation across the six bilingual participants are reported. To assess whether brain-informed fine-tuning elicits multilingual capabilities, we evaluate downstream task performance three settings: (a) Fine-tuning and evaluation in the same language: models are fine-tuned with brain data in one language (en or zh) and evaluated on NLP tasks in the same language (GLUE benchmark for en, CLUE benchmark for zh). This tests within-languagg improvements due to train-ifformed fine-tuning. (b) Cross- lansu-lan transfer between known languages: model fine-tuned on brain data in one language and evaluated on tasks in the participants' second (not used in fine-tuning) language (e.g., fine-tuned with en brain data and evaluated on CLUE (zh benchmark) tasks). This tests whether bilingual brain-informed fine-tuning elicits the participants" shared semantic representations. (c) Zero-shot transfer to unseen languages: to assess broader multilingual transfer, mBERT-ft-en is evaluated on downstream tasks in additional languages not seen during fine-tuning (German, French, Spanish, Japanese, and Korean) using XGLUE and XTREME benchmarks. Bolded values indicate performance equal to or better than the corresponding vanilla model. We observe that bilingual irain-informed fine-tuning improves performance on several NLP tasks, with mBERT showing greater benefits than BERT across all three settings. Results for fine-tuning using whole-brain and language-selectinn regions are reported in Appendix G.I.
see Tables 9 andd] 10
a) Fine-tuning and Evaluation in the Same Language
GLUE Task
BERT-en
BERT-N--n
mBERT
mBBRRT-feen
CoLA (MCC
53.38
55.25
0.86
42.68
41.89 = 0.74
SST-2 (Acc.)
92.08
92.62
0.10
89.68
91.06
0,44
CLUE Task
BER1-zh
BtRl-ft-zh
BBERT
mBERT-H(-Zh
MRPC (Acc.)
79.41
81.43
+: 0.79
84.80
85,96
= 0.59
AFQMC (Acc.)
75.25
73.76 1 0.3 39
69.74
70.37 = 0.38
MRPC (FF)
0.49
88.56
89.22
0.66
CMNLI (Ace.)
80.50
80.93
0.20
78.66
79.22 - 0.25
STS IIS.BCO (Pears.)
88.06
87.70
1: 0.19
88.06
88.31
1 0.77
CSL (FI)
80.18
81.41
0.36
81.10
81.36
+ 0.59
STS-B (Spear.)
87.65
87.54
0.20
87.76
88.27
IFLYTEK (Acz.)
60.25
60.37
0.50
56.52
56.66
1 0.19
QQP (Aco.)
90.84
90.88
+ 0.03
90.22
90.47
= 0.04
TNEWS (Acz.)
56.24
56.50
0.14
54.77
55,02
0.17
OQP (F1)
87.70
87.79
0.02
86.70
87.15 0.07
TNEWSS (FI)
55.17
55.95
0.27
$3.69
54.07
: 0.24
MNLI (Acc.)
84,38
84.29
0.26
$2,09
82.10
0.19
(Acc.)
10.66
0.00
10.66
10.66
1
MILI-mm (Ace.)
QNLI (Acc.)
84.64
84.60 + 0.31
82.38
91.14
82.52
91.18
- 0.34
C² (Acc.)
49.74
49.79
0.42
49.42
50.44
0.96
RTF (Acc.)
67.15
67.47
0.46
67.15
66.78
1.15
WNLI (Aoc.)
49.30
52.87
0.64
53.52
56.64 0.62
b) Cross-Language Transfer Between Known Languages
BERT-sh
BERT-A-zh
mBERT
mBRRRT-ftzh
Cal .A (MCC)
51.13
52.73
0.56
40.96
41.18
0.41
SST-2 (Acc.)
92.26
92.59 + 0.13
88.27
90.51
= 0.33
CLUE Task
BERT-en
BERT-R--n
MBERT
mBERI-N--n
MRPC (Aco.)
76.96
0.45
82.84
85,80=
0.23
AFQMC (Ace.)
69.31
0.38
69.74
71.12 - 0.54
MRPC (F1)
83.38
85.08
1 0.71
87.15
89.45
1 0.47
CMNLI (Ace.)
68.34
68.63
0.21
78.66
79.09 = 0.14
STS-B (Pears.)
87.21
88.37
0.18
$7.09
88.09
(FT) (F1)
71.20
71.63
1.55
81.10
81.34
+ 0.49
STS-B (Spear))
86,73
88.16
+ 0.13
86.97
87.99
= 0.48
IFLYTEK (Aco.)
47.86
47.53
0.27
56.52
57.21
0.51
QQP (Acz.)
QQP (FI)
87.53
90.65
87.92
91.03 1: 0.36
- 0.20
89.91
86.27
87.08
90.45 0.08
0.10
TNEWS (Acc.)
TNEWS (FI)
50.15
50.92
51.48
1 0.49
0.92
$4.77
$3.69
54.93
53.97
- 0.72
MNLI-m (Acc.)
84,00
84.59
1: 0.42
81.52
82.07 = 0.09
ChID (Acc.)
10.66
10.66
0.00
10.66
10,66
- 0.00
MNLI-mm (Ace.)
QNLI (Acc.)
91.27
91.35 1 0.05
0.46
90.55
81.94
91.04
82.74
0.10
C² (Acc.)
42.64
41.05
+ 6.41
49.42
50.33 + 0.43
RTE (Acc.)
67.15
0.65
66.06
68.71
WNLI (Aes.)
53,52
56.34 + 0.00
54.93
56.34 0.00
c) Zero-Shot Transfer to Unseen Languages
Task
German (de)
French (fr)
Spanish (es) mBERT-ft-on
Japanese (ja)
Korean (ko)
mBERT mBERT-ft-cn
mBERT ImBERTft-cn mBERT
mBERT
IBBERT-tt-on mBERT
mBERT-ft-cn
XGLUE
PAWS-X (Acc.)
84,00
84,90
1 0.54
88,10
88.48
1 0.44
87.05
87.44
1 0.48
78.70
78.70 1 0.17
78.75
78.36
1. 1.43
XNLI (Ace.)
70.60
71.04
: 0.83
72.69
73.38 + 0.91
74.10
74.96
+ 0.54
86.00
88.53
+ 0.15
86.00
88.25
+ 0.12
NER (Ace.)
10.67
11.89
1:
06.67
11.78
1.68
07.33
12.40
10.89 1: 0.51
12.00
11.13
1.48
XTREME
PAWS-X (Acc.)
85.65
184.51 + 0.01
88.95
18:25 + 0.01
87.90
87.18 + 0.01
79.55
78.98 1 0.01
79.35
78.71
0.01
XNLI (Acc.)
70.06
71.32
+ 0.86
72.97
73.53
0.25
74.50
74.73 + 0.13
86.00
87.13
+
86.00
87.21
0.51
NER (Acc.)
12.33
12.94
1: 0.83
09.33
11.91 + 1.03
10.33
11,89 + 0.46
08.67
11.09 + 0.92
12.33
11.87
0.77
improves across 6/7 tasks, with an average gain of +0.68 points and maximum improvement of +1.38 on the AFQMC task.
8

---

Overall, bilingual brain- brain-informed informed fine-tuning enables cross- lons-llngual generalization in language models, improving downstream task performance even when the fine-tuning and evaluation languages differ. This suggests that bilingual brain-informed fine-tuning of language models introduces changes that reflect the bilingual brain's 's shared semantic representations, enabling longuaa transfer.
Zero-shot transfer to unseen languages. Table Tic) reports performance on downstream NLP tasks when brain-informed fine-tuning (with camantically-ssleette brain regions) is performed with English brain data and evaluated on five unseen languages (German, French, Spanish, Japanese, and Korean), that were neither used for finettuning nor known the participant. We evaluate the multilingual model (mBERT-ft--e using tasks from the XGLUE and XTREME benchmarks. On the XGLUE benchmark, mBERT-ft-en improves performance on 3/3 tasks in German, French, and Spanish, and 1/3 tasks in Japanese and Korean, with average gains of +0.85, +2.06, +2.11, +0.14, and +0.33 percentage points, respectively. On the XTREME benchmark, mBERT-ft-en improves performance on 2/3 tasks in German, French, Spanish, and Japanese and 1/3 tasks in Korean, with average gains of +0.24, +0.81, +0.36, +0.99, and +0.04 percentage points, respectively, These results fine-tuning improves language-agnostic representations in
demonstrate that bilingual multilingual models, enabling generalization beyond the language of the brain data to entirely unseen languages (zero-shot).
Performance on downstream tasks with whole-brain or language-selective voxels. We evaluated downstream task performance before and after irain-informed fine-tuning using either whole-brai or language-seless selectivee voxels. Results averaged across all participants (mean + std) for these two settings are reported in Appendix G.1 Tables and 10 respectively. Table X in Appendix further compares mBERT-en (fine-tuned with English brain data) and mBERT-zh (fine-tuned with Chinese brain data) when using either whole-brain, language-selectve, selective, or semantically-seeeete for fine-tuning. Fine-tuning with either variant improves performance across all downstream tasks compared to vanilla models.
regions
Detailed results for individual participants, including fine-tuning with whole-brain, language-selective, and semantically--eeeee seleecive regions, are provided in the Supplementary (Tables 4-6). Results and analyses for other multilingual language models (XLM-R, XGLM, and LLaMA) are also reported in the Supplementary.
4.3 Monolingual Brain data as Fine- Teee-Tnning Targets
Table 2 compares downstream task performance after brain-informed fine-tuning (with semantically- selective brain regions) with English brain data from either bilingual or monolingual participants. Monolingual brain-informed fine-tuning improves downstream performance relative to the baseline across several tasks. However, bilingual fine-tuning yields even greater gains. On the GLUE bench- mark (within-language evaluation), bilingual brain-informed fine-tuning outperforms monolingual tuning 7/9 tasks, especially in all inference-related tasks (MNLI, QNLI, and WNLI). On the CLUE benchmark (cross-language evaluation), bilingual brain-informed fine-tuning leads to higher performance on 5/7 tasks. These results suggest that while monolingual brain-informed fine-tuning also improves downstream tasks within-language, fine-tuning with bilingual brain data enhances cross-linguistic generalization. Results for fine-tuning using whole-brain brain data reported in Appendix Table
Discussion and Conclusion
In this work, we perform brain-informed fine-tuning of monolingual and multilingual language models using brain data from bilingual participants reading the same naturalistic stories in English and Chinese. We show that bilingual brain-informed tine-tuned language models outperform their vanilla (pretrained) counterparts in both brain encoding performance and most downstream NLP tasks. Our analysis of brain-informed fine-tuning reveals several key conclusions.
First, representations from brain-informe fine-tuned language models are shared across bilingual in- dividuals. Both monolingual and multilingual language models show improved encoding performance after brain-informed fine-tuning, in both English and Chinese (Fig These models outperform baseline controls (Appendix Fig 4 and generalize across participants (Appendix Fig This high- lights the potential of bilingual brain data rich supervisory signal for grounding language model

---

Table 2: Downstream task performance before and after bilingual monolingual brain- informed fine-tuning (with semantically-sseeetiiy brain regions). We perform brain-informed fine-tuning of mBERT with English brain data (mBERT-ft-en) from semantically-seledte brain regions, from either bilingual or monolingual participants. For each task, the average performance and standard deviation across the participants reported. We evaluate downstream task performance in two settings: (a) Fine-tuning and evaluation in the same language: the model is evaluated English with GLUE tasks, (b) Poos--lag transfer between known languages: the model is evaluated in Chinese with CLUE tasks.
a) Fine-tuning and Evaluation in the Same Language
b) Cross-Language Transfer Between Known Languages
mBEKT-ft-cn
mBERT-n-em
Task
mBEKT
Biingual
Monolingual
CLUE Task
mBERT
Bilingual
Monolingual
CoLA (MCC)
42.68
41.89 + 0.74
42.16 = 5.94
ARJMC (Acc.)
69.74
70.37 = 0.38
70,87 1: 1.42
SST-2 (Acc.)
91.06 + 0.44
90.54 0.57
(MNLI(Acc)
78.66
79.22
0.25
79.02
1: 0.12
MRPC (Acc.)
84.80
85.96
1 0.59
84,68
+ 1.22
(FI)
81.10
81.36
0.59
81:07
1: 0.28
(FII
89.22
88.90
0.62
IFLYTEK (Acc.) )
46.79
56.66 = 0.19
56.56
+ 0.65
STS-B (Pearson)
88.06
88.31
+ 0.77
88.05 = 0.99
TNEWS (Acc.)
54.77
55.02
STS B (Spearman)
87.76
88.27
1 0.69
87.94
0.91
TNEWS (FI)
$3.69
54.07
0.24
51.54
1 0.15
QQP (Acc.)
90.22
90.47
1: 0.04
90.38
0.06
ChiD (Ace.)
10.66
10.66 0.00
10.66
+: 0.00
QQP (FI)
86.70
87.15
+: 0.07
86.94 = 0.15
C (Acc.)
49.42
50.44
0.96
50.36 0.26
(Acc.)
82.10
82.04
MNLI mmm (Acc.)
82.38
82.52
1 0.34
82.64
1 0.11
ONLI (Acz.)
91.14
91.18
91.18 0.05
RTE (Acc.)
67.15
66.78
+ 1.15
66.96
1.28
WNLI (Acc.)
53.52
56.64
1 0.62
56.34
1 0.00
representations with biologically meaningful representations. Second, brain-informed tunningning enables cross-language transfer in language models. Downstream performance improves across within-language, cross-language, and unseen language settings (Table [ This holds for fine-tuning with whole-brain language-selective or semantically-selective regions (Appendix Tables 9 and [) This effect is likely driven by shared semantic representations in bilingual individuals (Chen et al. 2024b). This highlights the potential of leveraging bilingual brain representations for developing language-agnostic language models. Lastly, the observed improvements are driven specifically by brain-informed fine-tuning with bilingual brain data, not brain data in general. Fine-tuning with monolingual brain data does not yield similar cross-linguuiste transfer (Table
Our results contribute to growing body of work exploring the representational alignment between brain and artificial language models. We demonstrate that bilingual brain-informed fine-tuning improves multilingual understanding in language models. To the best of our knowledge, this is the first work to do so. This study offers a bridge between neuroscience research on cross-linguistic semantic representation and NLP efforts to interpret and improve multilingual language models.
6 Limitations and Future Work
key limitation of this study the relatively small number of data samples per participant, due to limited bilingual brain recordings with naturalistic stimuli. This constrains the observed performance gains. Additionally, our analysis is restricted to only two languages (English and Chinese). Expanding to larger and more diverse multilingual brain datasets would allow broader evaluation of cross- linguistic transfer. Future work could also examine what linguistic properties are captured by brain-informed fine-tuning (such as syntax, morphology, or discourse-level structure). This can potentially be tested by targeting functionally selective brain regions and analyzing fine-tuned model representations across layers.
Acknowledgment
We thank Lily Gong for collecting and preprocessing the fMRI data, and Mathis Lamarre for valuable discussions. This work was funded by grants from the German Federal Ministry of Education and Research (BMBF; Grant no. 01GQ1906) and the European Research Council (ERC; Grant no. 101042567).
10

---

References
Mostafa Abdou, Ana Valeria González, Mariya Toneva, Daniel Hershcovich, and Anders Søgaard. Does injecting linguistic structure into language models lead to better alignment with brain recordings? arXiv preprint wrXiv:2101.12608 2021.
Richard Antonello, Javier S Turek, Vy Vo, and Alexander Huth. Low-dimensional structure in the space of language representations is reflected in brain responses. Advances in Neural Information Processing Systems, 34:8332- 184432--344, 2021.
Khai Loong Aw and Mariya Toneva. Training language models to summarize narratives improves brain alignment. In The Eleventh International Conference on Learning Representations, 2023.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. Transactions of the Association for Computational Linguisties, 5:135-146, 2017.
Charlotte Caucheteux and Jean-Rémi King. Brains and algorithms partially converge in natural language processing. Communications Biology, 5(1):134, 2022.
Catherine Chen, Tom Dupré Tour, Jack L Gallant, Daniel Klein, and Fatma Deniz. The cortical representation of language timescales is shared between reading and listening. Communications Biology, 7(1 7(1):284, 2024a.
Catherine Chen, Xue L Gong, Christine Tseng, Daniel L Klein, Jack L Gallant, and Fatma Deniz. Bilingual language processing relies on shared semantic representations that are modulated by each language. bioRxiv, pp. 2024-06, 2024b.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Fran- cisco Guzmán, Édouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8440-8451, 2020.
Andrea Gregor de Varda, Saima Malik-Moraleda, Greta Tuckute, and Evelina Fedorenko. Multilingual computational models reveal shared brain responses to 21 languages. bioRxiv, pp. 2025-02, 2025.
Fatma Deniz, Anwar O of semantic information across human cerebral cortex during listening versus reading is invariant to stimulus modality. Journal of Neuroscience, 2019.
Alexander G Huth, and Jack L Gallant. The representation
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, volume 1 (long and short papers). pp. 4171-4186, 2019.
Evelina Fedorenko, Po-Jang Hsieh, Alfonso Nieto-Castaîón, Susan Whiffelld-Gabrilli and Nancy Kanwisher. New method for fmri investigations of language: defining rois functionally in individual subjects. Journal of Neurophysiology, 104(2):1177-119) 2010.
Wendy Francis. Bilingual semantic and conceptual representation. Handbook of Bilingualism: Psycholinguistic Approaches, pp. 251-267, 2005.
Ariel Goldstein, Zaid Zada, Eliav Buchnik, Mariano Schain, Amy Price. Bobbi Aubrey, Samuel Nastase, Amir Feder, Dotan Emanuel, Alon Cohen, et al. Shared computational principles for language processing in humans and deep language models. Nature Neuroscience, 25(3):369-380, 2022.
François Grosjean. On bilinguals and bilingualism. Cambridge University Press, 2024.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation. In International Conference on Machine Learning, pp. 4411-4421. PMLR, 2020.

---

Alexander G Huth, Wendy A De Heer, Thomas L Griffiths, Frédéric E Theunissen, and Jack L Gallant. Natural speech reveals the semantic maps that tile human cerebral cortex. Nature, 532 (7600):453 458, 2016.
Shailee Jain and Alexander Huth. Incorporating context into language encoding models for fmni. Advances in Neural Information Processing Systems, 31, 2018.
Antonia Karamolegkou. Mostafa Abdou, and Anders Søgaard. Mapping brains with language models: A survey, In Findings of the Association for Computational Linguistics: ACL 2023, pp. 9748-9762, 2023.
Mathis Lamarre, Catherine Chen, and Fatma Deniz. Attention weights accurately predict language representations in the brain. In Findings of the Association for Computational Linguistics: EMNLP 2022. pp. 4513-4529, 2022.
Amanda LeBel, Lauren Wagner, Shailee Jain, Aneesh Adhikari-Desai, Bhavin Gupta, Allyson Morgenthal, Jerry Tang, Lixiang Xu, and Alexander G Huth. A natural language fmri dataset for voxelwise encoding models. Scientific Data, 10(1):555, 2023.
Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang. Guihong Cao, et al. Xglue: A new benchmark datasetfor cross-lingual are-training. understanding and generation. In Proceedings of the 2020 Conference on Empirical Methods in Nat- ural Language Processing (EMNLP), pp. 6008-6018. Association for Computational Linguisties, 2020.
Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian OoHor,, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, and Xian Li. Few-shot learning with multilingual generative language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 9019-9052, 2022.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2017.
Omer Moussa, Dietrich Klakow, and Mariya Toneva. Improving semantic understanding in speech language models via brain-tuning. In The Thirteenth International Conference on Learning Representations, 2025. URL htt.::ttt/://o... htttps:://ppeneeeee...eeeee wtt.:/nneeetttooodiddddddasaaa
Thomas Naselaris, Kendrick N Kay, Shinji Nishimoto, and Jack L Gallant. Encoding and decoding in fmri. Neuroimage, 56(2):400-410, 2011.
Subba Reddy Oota, Jashn Arora, Veeral Agarwal, Mounika Marreddy, Manish Gupta, and Bapi Surampudi. Neural language taskonomy: Which nlp tasks are the most predictive of fmni brain activity? In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2237- 2022.
Subba Reddy Oota, Emin Çelik, Fatma Deniz, and Mariya Toneva Speech language models lack important brain-relevant semantics. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 8503-8528. Association for Computational Linguistics, 2024a.
Subba Reddy Oota, Manish Gupta, and Mariya Toneva. Joint processing of linguistic properties in brains and language models. Advances in Neural Information Processing Systems, 36, 2024b.
Subba Reddy Oota, Zijiao Chen, Manish Gupta, Bapi Raju Surampudi, Gael Jobard, Frederic Alexandre, and Xavier Hinaut. Deep neural networks and brain alignment: Brain encoding and decoding (survey). Transactions on Machine Learning Research, 2025. ISSN 2835-8856. URL Survey Certification.
https:///penrrevie: wtt.:ewwwwwee.ooom///d//ei/iiiia
Martin Schrimpf, Idan Asher Blank, Greta Tuckute, Carina Kauf, Eghbal A Hosseini, Nancy Kan- wisher, Joshua B Tenenbaum, and Evelina Fedorenko. The neural architecture of language: Integrative modeling converges on predictive processing. Proceedings of the National Academy of Sciences, 2021.
12

---

Lisa Schut, Yarin Gal, and Sebastian Farquhar. Do multilingual Iims think in english? In ICLR 2025 Workshop on Building Trust Language Models and Applications, 2025.
Dan Schwartz, Mariya Toneva, and Leila Wehbe. Inducing brain-relevant bias natural language processing models. Advances in Neural Information Processing Systems, 32, 2019.
Philip Smith and Daniel R Little. Small is beautiful: In defense of the small-n design. Psychonomic Bulletin & Review, 25(6):2833-25)) 2200,,21112222 2018.
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. Advances in Neural Information Processing Systems, 29, 2016.
Mariya Toneva and Leila Wehbe. Interpreting and improving natural-language processing (in (in the brain). Advances in Neural Information
machines) with natural Processing Systems, 32, 2019.
Mariya Toneva, Jennifer Williams, Anand Bollu, Christoph Dann, and Leila Wehbe. Same cause; different effects in the brain. Causal Learning and Reasoning, 2022.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini wrxiv:2307.09288: 2023.
Nishitha Vattikonda, Aditya R Vaidya, Richard Antonello, and Alexander G Huth. Brain- wavlm: Fine-tuning speech representations with brain responses to language. arXiv preprint arXiv:2502.08866 2025.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations, 2018
Leila Wehbe, Brian Murphy, Partha Talukdar, Alona Fyshe, Aaditya Ramdas, and Tom Mitchell. Simultaneously uncovering the patterns of brain regions involved in different story reading subpro- cesses. PloS One, 11, 2014a.
Leila Wehbe, Ashish Vaswani, Kevin Knight, and Tom Mitchell. Aligning context- based statistical models of language with brain activity during reading. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 233-243, 2014b.
Chris Wendler, Veniamin Veselovsky. Giovanni Monea, and Robert West. Do llamas work in english?
on the latent language of multilingual transformers. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties ( (Volume I: Long Papers), pp. 15366-15399 2024.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State-of of of-the-art are natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38 45, 2020.
Michael C-K Wu, Stephen V David, and Jack L Gallant, Complete functional characterization of sensory neurons by system identification. Annual Review Neuroscience, 29(1 :977757755 2006.
Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, et al. Clue: A chinese language understanding evaluation benchmark. In Proceedings of the 28th International Conference on Computational Linguistics, pp. 4762-4772, 2020.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have ensured that the main claims made in the abstract and introduction are directly correlating to the research findings and the methods we have employed.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper discusses the main limitations of the work performed by the authors in the discussion section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiitiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
14

---

Answer: [NA]
Justification:
Our paper does not require any explicit theorems and proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: The paper has delineated all the information related to the experimental setup in the experimental setup section and model parameters details in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general, releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
15

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We provide a GitHub repository in the abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ priips.cc/ubbliccguuueee CoodeSumeeieeeeee ssi iessonPolll mollicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide all the training and test details in the experimental setup (See Section 3.3 3.4 3.5
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We conducted our experiments multiple times across participants and took the average results for baseline settings.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
16

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have included the specifications of the hardware and software environments ensure the reproducibility of our results (See Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in this paper fully conforms with the NeurIPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: This paper could benefit society by advancing the development of NLP systems to be more human-like.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
17

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our research does not pose any risks for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have explicitly cited the NLP benchmark datasets, code, and models used.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
18

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We release the code, model weights, and complete documentation in the GitHub repository linked in the abstract.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: fMRI data recorded from human subjects. All ethical regulations relevant to human research participants were followed, and all procedures were approved by the Committee for the Protection of Human Subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethies, workers involved in data collection, curation,
or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: All subjects signed a consent form where risks were disclosed, and IRB approval was obtained for this experiment. Further details can be found in Appendix B.2
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
19

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification:
We have used LLM only for grammar correction.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy bottts://neurre http://errip..cccooeeeereeee//// cc//Conferencce//2225/LLL for what should or should be described.
20

---

Overview of Appendices
Section
A
Detailed Related Work
Section
Naturalistic Brain Imaging Dataset and Preprocessing
Section
Brain-Informed Fine-tuning with Subsets of Cortex
Section D
Implementation details for reproducibility
Section
Details of Downstream Tasks
Section
Additional results for Voxelwise encoding performance
Section
G
Models fine-tuned with different training objectives
Section
Bilingual vs. Monolingual with English whole-brain data
A Detailed Related Work
Fine-tuning of language models with naturalistic brain data, Our work is most closely related to that of SScwwrrzz et al. (2019); Moussa et al. (2025); Vattikonda et al. (2025), who proposed the brain-tuning approach to fine-tune pretrained Transformer-basse language models using brain data. This approach integrates brain-relevant information into the language models and examines whether and how brain-informed language models effect brain encoding performance and downstream task performance. Specifically, /Schwartz et al. (2019) introduced brain-tuning using brain data recorded (recifically,SScwwartz
during naturalistic reading tasks monolingual English speakers, observing improved brain encoding performance after fine-tuning along with enhanced performance downstream NLP tasks. More recently, [Moussa et al. (2025); Vattikonda et al. 2025)) applied a similar approach to explore whether brain-tune tuning of speech- based language models could enhance brain-relevant semantic representations, thus improving encoding performance and speech-tase task performance. Our study complements these previous studies by investigating bilingual prain-informe fine-tuning and exploring how monolingual and multilingual language models change when trained with bilingual brain data.
Multilingual language models and brain alignment. Our work also relates to growing body of literature investigating alignment between human brain activity and language models. Several studies have successfully used text-based language models to predict brain activity evoked by both written and spoken stimuli, achieving impressive levels of alignment between language models and
brain activity (Wehbe et al., 2014a; Jain & Huth 2018 Toneva & Wehbe 2019 Deniz al.] [0199
Abdou et al., [021] Toneva et al., 2022 Antonello et al.) 2021 Oota et al.) [022] Aw & Toneva
2023; Oota al. 2024b; Lamarre et al. 2022 Chen et al. 2024a). More recently, research has
focused on multiliguall Transformer- based language models using brain data from reading and listening tasks across multiple languages evaluate their brain encoding performance (de Varda et af. 2025). However, previous multilingual studies have generally remained monolingual in their experimental design, with participants exposed exclusively to stimuli in one language. A notable exception is the recent work by Chen et al. (2024a), which examined bilingual language processing in participants who read identical stories in both English and Chinese, and find that bilingual individuals have shared semantic representations across languages. Our approach complements this research by further exploring bilingual brain alignment through brain-informed fine-tuning of language models, particularly analyzing how different brain regions respond to bilingual stimuli and fine-tuning strategies, evaluating their impact on downstream NLP tasks both English and Chinese
B Naturalistic Brain Imaging Dataset and Preprocessing
B.1 MRI data collection of the bilingual dataset
MRI data were collected on 3T Siemens TIM Trio scanner using 32-channel volume coil. Functional scans used gradient-echo EPI sequence (TR 2.0045 s, TE = 35 ms, flip angle = 74%, voxel size 2.24 x 2.24 x 4.1 mm, 30 interleaved axial slices). Motion correction and alignment were performed using the FMRIB Linear Image Registration Tool (FLIRT) from ESL. Please refer to Chen et al. (2024b) for more details.
21

---

B.2 Participants
Functional data from six healthy bilingual (in English and Chinese) participants from (Chen et al. 2024b) with normal hearing and vision was used, All participants were right-handed or ambidextrous. All potential risks and/or discomforts from MRI were disclosed to participants: 1) loud beeping and hammering sounds while the scanner is collecting measurements; 2) claustrophobia while inside the scanner; 3) nerve stimulation; 4) risk of injury from metal objects affected by the scanner (highly unlikely); 5) compromised confidentiality (highly unlikely). All participants signed consent form where these risks were disclosed. All ethical regulations relevant to human research participants were followed, and all procedures were approved by the Committee for the Protection of Human Participants. For anonymity, detailed information about the ethics committee has been omitted and will be included in the camera-reat -eadeyy version. Voxelwise encoding models and brain- irfin-informed fine-tuning are done separately for each participant, with results reported per participant. No sample size calculations were performed, each participant serves full replication of the results.
B.3 Monolingual participants from LeBel al. 2023) and Deniz et al. (2019
Functional data were collected from three healthy, monolingual English participants with normal hearing: two participants (UTS07: male, age 25; UTS08: male, age 24) from [LeBel al. (2023) and one participant (male) from (Deniz et al. 2019). Participants listened to several narrative stories from The Moth podcast. For the monolingual analysis, we used subset of the data from [eBel et al. (2023) and Deniz et al. (2019), specifically the same stories that are included in the bilingual dataset. For more details, please refer to the original publications (Deniz et al. 2019 LeBel al. 2023).
C Brain-Informed Fine-tuning with Subsets of Cortex
a) Language-selective regions
b) Semontically-seeeetive regions
Figure 3: Cortical regions used for brain-informed fine-tuning, Flattened cortical surfaces for: (a) Language-selective regions displayed on the 'fsaverage' surface, used the mask for all participants. and (b) semantically-seeectee for each participant and language. Only fMRI responses from the dark red regions were used for fine-tuning in their respective variants.
regions for English for Participant 1. This mask is computed separately
C.1 Fine-tuning with Saneuage Regions
Voxels were extracted from regions of interest (ROIs) known to support language comprehen- sion (Fedorenko al. 2010). To apply these ROIs to individual participants, we first mapped each participant's 's cortical surface to the fsaverage template using surface-based registration. The group- level language mask was then projected onto each participant's brain extract participant-specific language ROIs. The ROI mask used is shown in Appendix Fig. Bh.
C.2 Fine-tuning with Semantically-Selettin Regions
To identify voxels selective for semantics, we applied a two-stage regression procedure to isolate semantic signals while controlling for low-level sensory confounds. Semantic features were extracted using fast'Text (Bojanowski et al.) (017)) embeddings of the narrative text. This procedure was performed separately for each participant and language. First, trained VEMs for response to seven
22

---

low-level sensory feature spaces: word count, letter/character count, single phonemes, diphones, triphones, visual motion energy, and pixel-based orthographic similarity. These models were trained on the training data and used to predict responses for both train and test sets. Predicted low-level responses were subtracted from the actual BOLD signals to obtain residuals. These residual responses were then z-scored per voxel to have unit variance. Next, we trained VEMs using the semantic features on the residual signals. Voxels that showed sigmificant (one-sided permutation test, p < 0.05, FDR-corrected) prediction accuracy based on semantic features were labeled as semantically-selettive The semantically--seeetiie voxel mask for participant shown in Appendix Fig. 3b.
C.3 Baseline Models for Fine-tuning
TR-shuffled IMRI as fine-tuning target. We tested how fine-tuming language models with temporally misaligned data affects model representation. This TR-shuffled approach allows us verify that the improvements from brain-informeed fine-tuning are driven by meaningful stimulus-response alignment, rather than random brain data. Here, we randomly shuffled the brain responses, permuted in blocks of 10 contiguous TRs. This breaks the correspondence between stimulus and brain response.
Multilingual model features fine-tuning target. We tested how fine-tuning monolingual language models with multilingual model representations affects model representations. The goal of this test is compare the results between pure-language model fine-tuning with brain-informed language model fine-tuning. Here, we replaced the fMRI data associated with input stimuli with representations obtained from a multilingual language model. We then fine-tuned monolingual language models (BERT-en and BERT-zh) using these multilingual model representations (from mBERT) as targets.
Monolingual brain data fine-tuning target. We tested how tuning language models with brain data from individuals who know only one language differs from tuning language models with bilingual brain data. This test is used to clarify the impact of bilingualism the resulting model representations.
D Implementation details for reproducibility
All brain-informed fine-tuning experiments were conducted machine equipped with an NVIDIA TITAN RTX GPU (24 GB RAM), and NVIDIA RTX A6000 GPU (40GB RAM). The downstream NLP tasks were evaluated on machines with the same GPU configuration. Voxelwise encoding models were trained on the TITAN RTX GPU using banded ridge regression, with the following hyperparameters: MSE loss, L2 regularization (A) values logarithmically spaced from 10- to 10 target batch size of 1000, and 20 À values per batch.
For downstream NLP task fine-tuning, we used batch size of 64, learning rate of 2e-5, weight decay of 0.01, per-device evaluation batch size of 128, and trained for epochs.
E Details of Downstream NLP Tasks
Detailed description of the downstream NLP tasks in GLUE and CLUE benchmarks is provided in Table 3 Detailed description of the downstream NLP tasks in XGLUUE and XTREME benchmarks is provided in Table The metric ranges and the scaling for each task are provided in Tables 5 and
F Additional results for Voxelwise Encoding Performance
Please note that the bilingual dataset is from a reading experiment, where variations in visual brain areas is correlated with stimulus presentation, especially in naturalistic settings. Moreover, semantic features from language models are known to spuriously predict brain activity in visual areas even after regressing out low-level visual information (Deniz et al., 2019 Oota et al.] (024b). In our analyses, we used embeddings extracted from the language model layer for fitting the VEMs, without including any low-level sensory features. Consequently, well-predicted voxels in the reported figures do not necessarily correspond to language-selective brain regions. This distinction is important to avoid overinterpreting encoding performance in non-linguistic areas such as the early visual cortex. Importantly, our analyses using semantically selective voxels (which explicitly exclude visual regions)
23

---

Table 3: Descriptions of tasks in the GLUE and CLUE benchmarks.
Task
Description
Task Type
GLUE Benchmark
CoLA
Determine linguistic acceptability of senience
Single sentence classlication
SST-2
Sentiment analysis movic reviews
Single-sentence classification
MRFC
Determine two sentences ure semamically equivalent
Senience puir classilication
semantic similarity (0-5) between two sentences
Sentence pair regression
QQP
Identify if two questions are duplicates
Sentence-pair elassification
MNLI
Determing premise entails, contradicts, or is neutral to a hypothesis Determine if senience contains the
Sentenco-pair classification (mulli--cass))
QNLI
to question
Senience puir classiication
RTE
Determine textual entailment between two sentences
Sentence puir classification (binary)
WNLI
Coreference resolution: determine hypothesis with replaced peonoun
given a sentence
Sentence-pair
CLUE Benchmark
AFQMC
Determine Chinese sentences semantically equivalent
Sentence-pair classification
CMNLI
Multllngual version of MNLI for Chinese
Sentenoc-pair classification
CSL
Determine whether keyword list matches given abstract
Sentence-pair classification
IFLYTEK
Classify app descriptions into prodetined categories
Multi-clas classification
TNEWS
Classily news headlines into calegories
Multi-class classilication
Chil
Fill the blank multiple choice close test from Chinese literature
Cloze style multiple choice
C²
Reading comprehension with multiple-chnice answers
Multi-tum QA /Multipl--choice
Table 4: Summary of XGLUE and XTREME tasks used in this study.
Benchmark
Task Name
Task Type
Languages Evaluated de. de, ft. es. ko
Evaluation Metric
PAWS-X
Paraphrase Identiiication
kn
Accuracy
XGLUE
XNLI
Natural Language Inference
Accuracy
Fiiiiy Recognition
de.
ja. ko
Score
PAWS-X
Puraphrase Identsiccttion
de. lr.
Accuracy
XTREME
XNLI
Natural Language Inference
de, fr. es, ja. ko
Accuracy
NER
Named Entity Recognition
de,
ko
Fl Score
Table GLUE tasks, metrics, reference ranges, and scaling.
GLUE Task
Metric
Reference Range
Scaled
CoLA
Matthews Correlation
x100
SST2
Accuracy
x100
MRPC
FI Score / Accuracy
[0.
100
STS-B
Peanson Spearman Correlation
x100
QQP
F1 Score / Accuracy
[0.
X x100
Accuracy
QNLI
Accuracy
00, x100
RTE
Accuracy
[U, x100
WNLI
Accuracy
0.
x100
Table CLUE tasks, metrics, reference ranges, and scaling.
CLUE Task
Metrie
Reference Range
Scaled
AFQMC
Accuracy
0,
x 100
CMNLI
Accuracy
0. 1
x 100
CSL
Score
0. 1
x 100
IFLYTEK
Accuracy
0. |
1100
TNEWS
Accurncy FI Score
00, 1]
x 100
ChID
Accuracy
[0,
C
Exact Match/ Accuracy
[0, 1] x 100
Table 7: XGLUE tasks, metrics, reference ranges, and scaling.
XGLUE Task
Metrie
Reference Range
Scaled
x100
PAWS-X
Accuracy Accuracy
0.
x 100
XNLI
x100
show consistent performance gains from brain-informed fine-tuning, indicating that the observed improvements are not driven by visual areas but reflect genuine alignment with higher-level semantic processing.
F1 Baseline Models
Appendix Fig. shows results from baseline experiments where brain-informed fine-tuning was performed using (a) TR-shuffled fMRI responses and (b) mBERT representations fine-tuning

---

targets. For BERT-en across participants (considering well- voxels with > 0.1), the average difference in encoding performance was A maanill-TRssv 0. 0.136. In both cases, we observe that VEM predictions are limited to some sensory regions and scattered across the cortex, with no systematic predictions in higher-level semantic areas. This confirms that the observed improvements in our main experiments are not driven by spurious correlations but rather due to biologically plausible representations bilingual brains.
0.133 and A novitaa- BERT
a) TR-shuffled fMRI fine-tuning target
b) MBERT representations as fine-tuning
target
BERT BERT-en en
Encoding Performance
6.0
0.5
030
322
BERT-2 -h
Figure 4: Encoding performance of baseline fine-tuned models. Voxelwise encoding performance visualised for two fine-tuning baselines: (a) using TR-shuffled fMRI data, and (b) using mBERT representations as fine-tuming target. Flattened cortical surfaces with encoding performance are shown for Participant 1, with BERT-en (on top) and BERT-zh (on the bottom) fine-tuned with the baseline targets. In both baselines, the well predicted voxels do not correspond to semantically regions and appear scattered in low-level sensory regions.
F.2 Results Participants 2-6
Fig. 5 shows brain encoding performance before and after brain-informed fine-tuning for all partici- pants. This figure shows voxelwise encoding model performance for the best-performing fine-tuned variants across the cortex for all participants, using (a) BERT-en, (b) BERT-zh, and (c-d) mBERT models. This replicates the format of Fig. 2. and reports it for all participants. For details voxel color coding, refer to the caption of Fig. 2 Across participants, model types, and languages, brain- informed fine-tuning consistently improves encoding performance relative it's vanilla counterpart.
F.3
Cross-participant Transfer
To examine whether brain-informed fine-tuning generalizes encoding performance across partici- pants, we evaluated wrain-informed fine-tuned models on one participant's 's brain data and test it on the brain recordings of other participants. Fig. 6 presents the encoding performance comparisons between vanilla and brain-tuned models in this prosspparticiaat setting. Across participants (con- sidering prediced voxels with r > 0.1), the average difference in encoding performance was 00.0055 = 0.00029.
AF. -uriilaa-prtiicpanttteee
We observe that there is also no difference in overall encoding performance between the vanilla and brain-tuned models when tested on unseen participants. This pattern holds consistently across all participants, both English and Chinese brain data, and for both monolingual and multilingual BERT models. These findings suggest that the representational changes introduced by brain-specifis fine-tuning are not participant-specifi but reflect some shared representations across bilingual individuals.
25

---

(6) BERT-dwwithhhhtmoommmmm
(6)
Port-pponts
Partipp
Nurtepponts
Porteipante
15
Fine-tuned
Varilaa
with Whole brait
Fine ned with larguage reg
rne-ttned with visill
Encod Perfomancc (1)
Figure 5: Brain encoding performance before and after brain-informed fine-tuning for all participants. This figure shows voxelwise encoding model performance for the best-performing fine-tuned variants across the cortex for all participants, using (a) BERT-en, (b) BERT-zh, and (c-d) mBERT models. This replicates the format Fig. and reports it for all participants. For details voxel color coding. refer to the caption of Fig. Across participants, model types, and languages, brain-ine informinfrrmed fine-tuning consistently improves encoding performance relative to it's vanilla counterpart.
Participant
Participant
Participant
Participant
Participant
Diffrrenceln
Vanills mondell
Encoding Performance
Modde Fine horwed with Portoppont Is better
Webtter
Figure 6: Cross- Pross-participant generalization of brain-informed fine-tuned language models. Flat- tened cortical surfaces show changes encoding performance for Participants 2-6 when comparing BERT-en (vanilla) with BERT-ft-en fine-tuned using English brain data from Participant 1. Each point represents a voxel that was consistently well-predicted (r>0.11 across both models. Across participants, encoding performance in higher-level semantic regions shows either degradation or slight improvements with the fne-tumed model.
26

---

G Additional results on Downstream NLP Tasks Performance
G.1 Vanilla vs. Fine-tuned with Whode-brrammLanggug---sleece
regioos/Semnnticlll-sseeee voxels
Table g shows downstream task performance before and after bilingual brain-informed fine-tuning (with whole-brain, sanguage-selective and semantically-selectee voxels). We compared vanilla (pretrained) models and mBERT, with their billiguual brain-informed fine-tuned counterparts (e-g.. mBERT-ft-en and mBERT-ft-zh). To assess whether brain-informed fine-tuning elicit multilingual capabilities, we evaluate downstream task performance in two settings: (a) Fine-tume Fine-tuning and Evaluation in the Same Language: models are fine-tuned with brain data in one language (English (en) Chinese (zh)) and evaluated on NLP tasks the same language (GLUE benchmark for en, CLUE benchmark for zh). This tests within-language improvements due to brain-informer fine-tuning. (b) Cross- Language Transfer Between Known Languages: model is fine-tuned on brain data in onc language and evaluated tasks in the participants's ' second (not used in fine-tuning language (e.g., fine- tuned with en brain data and evaluated on CLUE (zh benchmark) task. This tests whether bilingual brain-informed fine- tuning elicits the participants' shared semantic representations. Bolded values indicate equal or the best performance. We observe that bilingual brain-informed fine-tuning improves performance several NLP tasks, with either language- or ceannnicalllsssleettee region variants showing more improvement than their vanilla counterpart.
Table 8: Downstream task performance before and after bilingual brain-informed fine-tuning (with whole-brain, language-selecting semantically-ssleee regions).
a) Fine-tuning and Evaluation the Same Language
Task
mBB.RT-en Whole-Brain
Language
Semantic
CLUE Task
mBERT-zzh Whole-Brain
Language
Semantic
CoLA (MCC)
42.68
40.05
41.80
42.81
AFQMC (Avc.)
69.74
70.55
72.03
71.11
SST-2 (Acc.)
90.14
90.25
90.60
CMNLI (Ace.)
78.66
79.02
78.75
79.01
MRPC (Acc.)
84.80
84.80
86,76
86.03
CSL (FII
81.10
80.71
82.27
81.24
MRPC (FI)
88.56
89.01
90.49
89.73
IFLYTEK (Acc.)
56.52
56.83
56.79
56.91
STS-B (Pearson)
88.06
88.42
88.46
88.42
TNEWS (Acc.)
54.77
55.12
55.27
54.91
STS-B (Spcarman)
87.76
88
$8.38
88.34
TNEWS (FI)
53.69
53.74
53.82
53.98
QQP (Ace.)
90.48
(Acc.)
QQP (FI)
86.70
87.07
87.02
87.11
C (Acc.)
49.42
30.21
49.76
49.95
MNLI-m (Acc.)
82.09
82.66
82.27
82.07
MNLI (Acz.)
82.38
82,97
82.92
82.68
QNII (Acc.)
91.14
91.18
91.10
91.31
WNLI (Acc.)
53.52
56.34
56.34
56.34
RTE (Ace.)
67.15
65.70
68.95
67.87
(b) Cross-Language Transfer Between Known Languages
Task
mTE Whole-Brain
Language
Semantic
CLUE Task
mBERT-P
Whole-Brain
Language
Semantic
CoLA (MCC)
40.96
41.22
40.82
41.60
AFQMC (Acc.)
69.74
70.62
71.29
70.76
SST-2 (Acc.)
88.27
$9.25
91.17
90.37
CMNLI (Acc.)
78.66
78.92
79.09
MRPC (Acc.)
82.84
83,82
85,05
87.01
CSL (FI)
81.10
81.30
81.53
81.57
MRPC (FI)
87.15
87.88
89.09
90.62
ILLYTEK (Acc.)
46.79
56.52
56.60
56.33
STS-B (Pearsan)
87.09
$8.11
87.00
88.54
TNEWS (Acc.)
54.77
54.81
54.95
53.47
STS-B (Spearman)
86.97
87.83
87.27
88.43
TNEWS (FI)
$3.69
53.84
54.01
53.75
QQP (Acc.)
(Acc.)
QQP (FI)
86.27
86.89
87.06
87.00
C (Acc.)
49.42
49.48
49.84
49.76
MNLI-m (Acc.)
81.52
82.13
82.54
82.94
MNLI-mm (Acc.)
81,94
82.36
82.74
83,12
QNLI (Ace.)
90.55
90.92
91.27
91.38
WNLI (Acc.)
54.93
56.34
56.34
56.34
RTE (Acc.)
66.06
66.43
68.59
66.06
27

---

Table 9: Downstream task performance before and after bilingual brain-informed fine-tuning (with whole-brain). We compared vanilla (pretrained) models, English BERT (BERT-en), Chinese BERT (BERT-zh), and mBERT, with their bilingual brain-informed fine-tuned counterparts (e.g.. BERT-ft-en: BERT-en fine-tuned using English brain data). For each task, the average performance and standard deviation across the six bilingual participants are reported. To assess whether brain- informed fine-tuning elicits multilingual capabilities, we evaluate downstream task performance in three settings: (a) Fine-tuning and evaluation in the same language: models are fine-tuned with brain data in one language (en or zh) and evaluated on NLP tasks in the same language (GLUE benchmark for en, CLUE benchmark for zh). This tests within- language improvements due brain-informed fine-tuning. (b) Cross-languan
transfer between known languages: model is fine-tuned on brain data in one language and evaluated on tasks in the participants's 's second (not used in fine-tuning) language (e.g., fine-tuned with en brain data and evaluated on CLUE (zh benchmark) tasks). This tests whether bilingual prain-informed fine-tuning elicits the participants shared semantic representations. (c) Zero-shot transfer to unseen languages: to assess broader multilingual transfer, mBERT-ft-n is evaluated on downstream tasks in additional languages not seen during fine-tuning (German, French, Spanish, Japanese, and Korean) using XGLUE and XTREME benchmarks. Bolded values indicate performance equal to or better than the corresponding vanilla model. We observe that bilingual brain-informed fine-tuning improves performance greater benefits than BERT across all three settings. Results for fine-tuning using whole-brain data for individual participants reported in the Supplementary (see Table 4).
several NLP tasks, with mBERT showing
a) Fine-tuning and Evaluation the Same Language
GLUE
BERT-en
BERT-f--n
mBERT
mBERT-ft-en
CoLA (MCC)
53.38
55.11
0.75
42,68
34.58
1.82
SST-2 (Acc.)
92.08
92.58 + 0.39
89.68
90.64
0.54
CLUE Task
BERT-2h
BERT-n-Zh
MBERT
MBERI-I--2
MRPC (Acc.)
79.41
1.01
$4.80
85,39
AFQMC [Acc.)
75.25
74.50
0.48
69.74
70.70 = 0.99
MRPC (F1)
86.27
86.91
1 0.75
88.56
89,20
0.27
CMNLI (Ace))
80.50
80.87 - 0,04
78.66
78.99 + 0.14
STS-B (Pears.)
88.10
1: 0.07
$8.06
88.47
CSL( (FI)
80.18
80.82
0.42
81.10
81.60
1 0.56
STS-B (Spear))
87.65
87.55
11.55=00 0.18
87.76
88.28
= 0.12
IFLYTEK (Acs.)
60.25
60.48
0.25
56.52
56.92
0.17
QQP (Acc.)
90.84
90.79
0.03
90.22
90.43
0.04
TNEWS (Acc.)
56,24
56.48
: 0.08
54.77
55.08
- 0.16
QQP (FI)
87.70
87.71
1. 0.19
86.70
87.08
0.02
TNEWS (FI)
53.69
53.81
1
MNL.I-m (Acc.)
84.38
84.40
1: 0.09
$2.09
82.26 - 0.27
ChID (Acc.)
10.66
10.66 = 0.00
10.66
10.66 = 0.00
MNLI- (Ace.)
QNLI (Aoc.)
91.45
91.49
1: 0.09
0.20
91.14
82.38
91.24
62.73
0.06
C² (Acc.)
49.74
49.83 + 0.39
49.42
50.25 = 0.27
RTE (Acc.)
67.15
67.32
0.17
67.15
65.63
0.51
WNLI (Acc.)
49.30
55,59
1.31
53.52
56.34 0.00
b) Cross-Languare
Transfer Between Known Languages
GLUE Task
BERT-sh
BERT-R-zh
mBERT
mBERR-44-2h
CoLA (MCC)
51.13
53.82 + 1.18
40.56
40.99 = 1.08
SST-2 (Acc.)
0.15
88.27
90.30=
0.66
CLUE Task
BERT-en
BERT-N--n
MBERT
mBERT-flecn
MRPC (Acz.)
76.96
77.53
1: 0.28
82.84
85.20
0.93
AFQMC (Aoc.)
69.00
69.02 - 0.03
69.74
71.25 = 0.97
MRPC(FF) (FI)
84.39
0.17
87.15
89.16
(Ase)
68.34
68.78
0.13
78.66
78.87 : 0.07
STS-B (Pears.)
87.21
87.96
+ 0.12
87,09
88.30
= 0.17
(FI)
71.20
72.33
0.66
81.10
81.51
1 0.20
STS-B (Spear.)
86.73
87.67
0.18
86.97
88.16
0.23
IFLYTEK (Ace.)
47.86
47.27
1 0.56
56.32
56.92
- 0.31
QQP (Aco.)
QQP (FI)
90.65
87.53
87.76 1: 0.04
90.86
:: 0.12
89.91
86.27
90.39
87.05 = 0.09
0.13
TNEWS (Acc.)
TNEWS (FI)
50.15
50.68
1 0.07
0.16
53.69
54.77
53.76
54,99
- 0.04
1
MNLI-m (Acc.)
81.52
81.97
0.14
ChID (Acc.)
10.66
10.66
0.00
10.66
10.66
= 0.00
MNLI mm (Ace.)
QNLI (Acc.)
83.91
91.27
91.42
84.16 1 0.10
$1.94
90.55
91,23
82.78
0.28
C (Acc.)
1.49
49.42
49.72
RTE (Acc.)
67.15
67.91
+ 0.08
66.06
67.21
0.67
WNLI (Ace.)
53.52
56.34
0.00
54.93
55.25
0.84
Zero-Shot Transfer to Unseen Languages
Task
German (de) MBERT mBB:RT-f--
French (fr)
Spanish (es)
Japanese (ja)
Korean (ko)
MBERT mBBRRI-fteen
mBERT
ImB:RT-f-enn MBERT mBLRRI-Denn
MBERT MBERT- ft-en
XGLUE
PAWS-X (Acc.)
84.00
83.62 + 1.10
88.10
87.17 1: 0.81
87.05
86.50 + 0.42
78.70
76.55 4: 2.12
78.75
76.90
+: 0.35
XNLI (Acc.)
70.60
70.94
0.14
72.69
73.30
0.80
74.10
74.62
0.05
0.05
86.00
87.82
0.25
NER (FI)
10.67
12.85
+: 2.02
6.67
11.09 1: 1.09
7.33
12.11
+ 0.67
13.00
11.34 1: 2.11
12.00
12.25
+: 1.71
XTREME
PAW'-X
85.65
84.85 + 1.34
88.95
88.00
0.99
87,99
80 = 0.39
79.55
78.35
01.35:100 0.85
79.35
78.70 = 0.64
XNLI
70.06
71.75
1 0.94
72.97
73.48 1: 1.73
74.50
74.85 + 0.35
86.00
87.61 1: 0.86
86.00
87.24
= 0.33
NER
12.33
12.17 + 1.18
09.33
13:02 1.39
10.33
11.12 + 0.30
08.67
13.00
0.00
12.33
11.17 + 1.18
28

---

Table 10: Downstream task performance before and after bilingual brain-informed fine-tuning (with language-selective). We compared vanilla (pretrained) models, English BERT (BERT-en), Chinese BERT (BERT-zh), and mBERT, with their bilingual brain-informed fine-tuned counterparts (e.g., BERT-ft-en: BERT-en fine-tuned using English brain data). For each task, the average perfor- mance and standard deviation across the six bilingual participants reported. To assess whether brain-informed fine-tuning elicits multilingual capabilities, we evaluate downstream task performance in two settings: (a) Fine-tuning and evaluation in the same language: models are fine-tuned with brain data in one language (en or zh) and evaluated on NLP tasks in the same language (GLUE benchmark improvements due brain-informed fine-tuning, (b) Cross-language transfer between known languages: model is fine-tuned on brain data in one language and evaluated on tasks in the participants's 's second (not used in fine-tuning) language (e.g., fine-tuned with en brain data and evaluated on CLUE (zh benchmark) tasks). This tests whether bilingual wwain-informed fine-tuning elicits the participants shared semantic representations. Bolded values indicate performance equal to or better than the corresponding vanilla model. We observe that bilingual brain- inninninformed fine-tuning improves performance on several NLP tasks, with mBERT showing greater benefits than BERT across all two settings. Results for fine-tuning using language-selective regions for individual participants are reported in Supplementary (see Table 5)
for en, CLUE benchmark for zh). This tests within-
a) Fine-tuning and Evaluation in the Same Language
GLUE Task
BERT-en
BERT-It-en
mBERT
mBEK7-f--n
CoLA (Acc.)
53.38
55.25
0.85
42.68
41.71 - 0.76
SST-2 (Asz.)
92.08
92.62
0.10
89.68
91.15
0.48
CLUE Task
BERT-sh
BERT-R-zh
MBERT
mBFRT-ft-zh
MRPC (Acc.)
79.41
81.43
0.80
84.80
85.76
0.59
AFQMC (Acc.)
75.25
73.92 1 0.46
69.74
70.38
: 0,50
MRPC (F1)
86.27
87.47
0.57
88.56
89,28
0.46
CMNLI (Ace.)
0.19
78.66
78.99
STS-B (Pearson)
88.06
87.70
0.19
88.06
88.14
0.91
(FI)
80.18
81.22 + 0.40
81.10
81.84
= 0.27
STS (Spearman)
0.19
87.76
87.86
IFLYTEK (Acc.)
60.25
60.41
0.33
56.52
56.59
1 0.68
QQP (Acc.)
90.84
90.88
0.03
90.22
90.47
0.05
TNEWS (Aco.)
56.24
56.49
0.17
54.77
55.12
- 0.18
QQP (FFI
87.70
87.79
0.02
86.70
87.11
0.08
TNEWS (FI)
55.17
55.12 + 0.33
53.69
54.14
= 0.19
MNLI-m (Acc.)
84.38
84,29
0.23
82.09
82.10
0.18
Chil (Acc.)
0.00
10.66
10.66
MNLI-mm (Aco.)
QNLI (Aco.)
84.64
84.60
= 0.30
0.05
91.14
82.38
91.04
82.49
0.29
e (Acc.)
49.74
49.77
0.33
49.42
50.15
0.59
RTE (Acc.)
67.15
67.47
0.45
65.15
67.75
1.15
(Acc.)
52.87
0.62
53.52
56.62
b) Cross-Languuaee
Transfer Between Known Languages
GLUE Task
BERT-sh
BERT-R--h
mBERT
mBB.R-ff-zh
(MCC)
0.56
40.96
42.50
SST-2 (Aco.)
92.26
92.39
0.13
88.27
90.89
0.19
CLUE Task
BERT-en
BERT-ft-n
MBERT
mBERT-fteen
MRPC (Ace.)
76.96
77.74
0.51
82.84
85.55
0.23
AFQMC (Acc.)
69.00
69.31
0.47
69.74
70.78 1: 0.67
MRPC (FF)
83.38
84.42
0.89
87.15
89.35
0.24
CMNLI (Acz.)
68.34
68.34
0.63
78.66
78.88 1 0.09
STS-B (Pearson)
87.21
87.63
0.74
87.09
88.11 + 0.41
(FI) (FI)
71.20
71.46
1: 0.43
81.10
81.25
- 0.29
STS-B (Spearman)
86.73
87.88
0.38
86.97
88.11
0.42
IFLYTEK (Acc.)
47.86
48.05
1.44
56.52
57.02
0.29
QQP (Acc.)
90.65
91.01
0.31
89.91
90.44 - 0.05
TNEWS (Acc.)
50.92
50.69
1 0.18
$4.77
55.23
: 0.24
QQP (F1)
86.27
87.07
TNEWS (F1)
53.69
54.11
MNLI-m (Acc.)
84.00
84.19
0.42
81.52
82.09 = 0.17
ChiD (Ace.)
10.66
10.66
0.00
10.66
10.66
= 0.00
MNLI-mm (Acc.)
83.91
84.17
0.47
81.94
82.47 : 0.34
C (Acc.)
42.64
55.18 :: 0.61
49.42
50.24 - 0.33
(Ace.)
91.27
91.33
0.10
90.55
91.12 0.19
RTE (Ace.)
67.15
66.81
- 0.67
66.06
66.63 - 1.28
WNLI (Acc.)
53.52
1.8
54.93
55,49
1.26
29

---

Bilingual vs. Monolingual with English whole-brain data
Table 11: Downstream task performance before and after bilingual monolingual brain- informed fine-tuning, We perform brain-informed fine-tuning of mBERT with English whole-brain brain data (mBERT-ft-en) from either bilingual (participant 1) monolingual participant, We evaluate downstream task performance in two settings: (a) Fine-tuning and evaluation in the same language: the model is evaluated in English with GLUE tasks. (b) Cross-language transfer between known languages: the model is evaluated in Chinese with CLUE tasks.
a) Fine-tuning and Evaluation in the Same
b) Cross-Languass Transfer Between Known Languages
Language
mBEKT-It-n
mBERT f cm
Task
mBERT
Bilingual
Monolingual
CLUE
Bilingual
Monolingual
CoLA (MCC)
42.68
40.05
35.74
AFQMC (Acc.)
69.74
70.62
70.64
(Acc.)
90.14
CMNLI (Acc.)
78.66
78.83
78.48
MRPC (Acc.)
84.80
84.80
78.85
CSL
81.10
81.30
81.18
(FI)
[FLYTEK (Acc.)
46,79
56.52
57.06
STS-B (Pearson)
88.06
88.42
88.42
'NEESS (Acc.)
$4.80
STS-B (Spearman)
87.76
58. 22
88.34
TNEWS (FI)
53.69
53.84
$3,50
QQP (Acc.)
90.22
90.47
90.54
(Acc.)
QQP (F1)
86,70
87.07
87.19
C (Acc.)
49.42
49.46
(Ase.)
MNLI mm (Aoc.)
82.38
82.97
$2.78
QNLI (Acc.)
91.18
RTE (Acc.)
67.15
65,70
65.34
WNLI (Acc.)
53.52
56.34
56.34
30


Reviews:
[{"review_id": "5ShG6FixIf", "reviewer": "Reviewer_jJEB", "summary": "The paper uses fMRI data obtained during story reading to fine-tune an LLM. The paper suggests that this type of fine-tuning improves voxelwise encoding modeling compared to the baselines. Additionally, the paper claims that fine-tuning the LLM on bilingual brain data improves downstream task performance on standard NLP benchmarks.", "questions": "The paper uses some terminology that may not be known to the broad audience of NeurIPs without defining it: What is brain encoding?What is a TR?\n\nA few design decisions that are unclear to me:\n1. What is the motivation for including language and semantically-selective regions into the analysis?\n2. One of the contributions of the paper is providing a novel brain-informed fine-tuning pipeline, while prior work has performed the same steps as part of pre-processing. Why is this important? Is the pipeline more efficient, performant, etc? Some comparison to prior work or motivation would be nice to have if the pipeline itself is listed as a novel contribution.\n\nVoxelwise encoding modeling: The baselines (temporally mis-aligned data and mBERT embeddings) seem a bit too weak of a comparison for fine-tuning with the actual fMRI data. I’d encourage the authors to consider a stronger baseline.\n\nPerformance on downstream tasks: \n1. The fMRI dataset contains 6 bilingual and 1 monolingual participants. How was bilingual fine-tuning performed — was one model fine-tuned on 1 participant data? That is, are the post-fine-tuning numbers in Table 1 averages over 6 fine-tuned models? Or was it a randomly chosen participant? Or something else entirely?\n2. I find it difficult to interpret the findings in Table 1 and Table 2 without knowing how much variability there is across models trained on different participants since the differences are very small.\n3. Similarly, the comparison between the bilingual and monolingual performance (Table 2) shows very small differences. Since we’re essentially looking at 2 participants in this table, my summary of the findings is that bilingual and monolingual brain fine-tuning performs broadly similarly. Given the current evidence, I'm not convinced that bilingual brain fine-tuning outperforms monolingual one.\n\nModel choices: the paper states that it fine-tunes 4 multilingual models: mBERT, XLM-R, XGLM, LLaMA-3.2-1b. However, I can only find the results from the BERT models either in main text or appendices. Is the paper missing the Supplementary (the authors refer to the Supplementary in multiple spots but it doesn’t exist. I assumed it was another name for the appendix but I’m no longer sure). Either way, I’d like to see how stable the results are across different architectures.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Clarity: The paper is very difficult to understand — after 2 reads of the paper and going through all of the appendices, a lot of aspects of the data, task and other design decisions are still unclear to me. There are a lot of appendices, including one on extended related work, which are all necessarily to be able to follow the paper. This makes me wonder whether a journal submission where there’s more space for discussion is a better format for the current work.\n\nQuality: the main claims of the paper are only weakly supported. The baselines for the voxelwise modeling are fairly weak and the differences between bilingual and monolingual models are very small and hard to interpret without understanding the variability between models (see Questions for more details).\n\nSignificance and originality are hard for me to assess since I’m not an expert in the field. The paper tackles a very interesting question — whether fine-tuning LLMs with fMRI data obtained on language tasks improves model’s performance on standard NLP benchmarks, asking specifically whether models benefit from bilingual data over monolingual data. Based on the Related Work section, we already know that brain encoding and NLP performance improves after fine-tuning on monolingual fMRI data. Thus, the main contribution of the current work is evidence of the difference in performance between monolingual and bilingual brain data, which is hard to assess given the provided evidence.", "was_revised": true, "final_justification": "The authors have conducted additional analyses to address my concerns (including adding monolingual participants in their study) and clarified the points that were unclear in the original submission. My only remaining concern is how many of the clarifications/elaborations provided in the response to the reviewers would make it into an already tight paper to improve the clarity given the space constraints.", "rebuttal": "## Clarification on the Main Objective of the Paper\nThank you for your comment. Prior studies have fine-tuned speech-based language models with brain data and shown improvements in encoding performance (Moussa et al., 2025; Vattikonda et al., 2025). In contrast, our study focuses on text-based language models, which already exhibit strong semantic capabilities and have been shown to align well with brain activity. The key question we address is whether bilingual brains can serve as a useful supervisory signal to enhance the multilingual capabilities of language models. Specifically, we ask whether fine-tuning with a bilingual brain, which is known to encode a shared semantic space across languages (Chen et al., 2024), can benefit both monolingual and multilingual language models.\n\nThe comparison between fine-tuning with monolingual and bilingual brain data serves primarily as a control to assess whether the observed improvements in multilingual performance are attributable to brain data in general, or specifically to the bilingual brain.\n\n## Clarification on Methods\nThank you for the helpful feedback. We will include additional citations and elaborations in the revised version to explain the foundational concepts more clearly for a broader audience. To clarify a few terms: “brain encoding” refers to the prediction accuracy of voxel-wise timecourses, where each voxel in our data corresponds to a 2.24 × 2.24 × 4.1 mm³ volume of the brain. “TR” stands for repetition time, which is the acquisition time for each fMRI volume; in our case, the TR was 2.0045 seconds.\n\n## Clarification on design decisions\n### Response for Q1\nFor speech-based models, Vattikonda et al. (2025) showed that fine-tuning with different brain regions can yield differences, as regions may compete during fine-tuning and dominate the loss (see also Antonello et al., 2023). To avoid noise or voxels from non-language/semantic areas dominating the loss, we fine-tuned two complementary models for each participant: one using only brain responses from language regions and another using semantically-selective regions. The semantically-selective regions were chosen in a data-driven manner for each participant using encoding performance, while the language regions were defined using a group-level template (Fedorenko et al., 2010). These variants showed slightly better performance than the whole-brain model on some tasks, as reported in Appendix Section F.\n\n### Response for Q2\nThank you for the insightful question. Typically in fMRI studies, text stimuli are presented either in a controlled manner for example as a fixed number of words per repetition time (TR) (e.g., four words per TR, each lasting 0.5 seconds used in (Schwartz et al. 2019)) or as continuous input later downsampled during preprocessing. This limits flexibility, especially in settings where word timing is uneven. Performing stimulus-TR alignment as a preprocessing step (prior to fine-tuning) likely discards valuable temporal and contextual information.\n\nIn contrast, our pipeline introduces stimulus-TR alignment during fine-tuning, which has three key advantages:\n- variable-length input (since word timing per repetition time (TR) can vary)\n- more flexible and generalizable alignment applicable across modalities and tasks as we pass raw text as input \n- dynamic stimulus-repetition time (TR) alignment as part of the learning process, not a static preprocessing step\n\nIn response to your suggestion, we implemented an alternative version of our pipeline that mirrors the preprocessing-based approach used in prior work; stimulus-TR alignment is performed prior to fine-tuning. Specifically, we constructed stimulus-TR pairs similar to Schwartz et al. (2019). We then fine-tuned the language model using this pre-aligned input and bilingual brain data from 1 participant. The mean encoding performance is 0.13 for the preprocessing-based method, compared to 0.15 for the vanilla model and 0.16 for our fine-tuned model. The table below presents downstream task performance (on the GLUE benchmark) for this preprocessing-based method (denoted as BERT-ft-en (Ablation)). This allows direct comparison with our proposed end-to-end brain-informed fine-tuning pipeline.  We observe a substantial drop in performance for the preprocessing-based method (BERT-ft-en (Ablation)). This suggests that stimulus-TR alignment performed prior to fine-tuning procedure likely discards valuable temporal and contextual information. In contrast, our approach integrates brain alignment during fine-tuning, enabling robust downstream performance.\n\n|GLUE Task|BERT-en|BERT-ft-en (Mean±Std)|BERT-ft-en (Ablation)\n|-|-|-|-|\n|CoLA|53.38|**55.11 ± 0.75**|0.00|\n|SST-2|92.08|**92.58 ± 0.39**|50.92|\n|MRPC (Acc.)|79.41|**80.55 ± 1.01**|68.38|\n|MRPC (F1)|86.27|**86.91 ± 0.75**|81.22|\n|STS-B (Pears.)|88.06|**88.10 ± 0.07**|0.008|\n|STS-B (Spear.)| **87.65**|87.55 ± 0.18|0.028|\n|QQP (Acc.)|**90.84**|90.79 ± 0.03|75.28|\n|QQP (F1)|87.70|**87.71 ± 0.19**|62.81|\n|MNLI-m| 84.38|**84.40 ± 0.09**|41.74|\n|MNLI-mm|**84.64**|84.55 ± 0.20|41.48|\n|QNLI|91.45|**91.49 ± 0.09**|58.32|\n|RTE|67.15|**67.32 ± 0.17**|52.71|\n|WNLI|49.30|**55.59 ± 1.31**|43.66|\n\n## Clarification on baseline for Voxelwise encoding modeling\nWe agree that strong baselines are important for comparing the value of fine-tuning with bilingual brain data. Our current baselines are designed to test two specific hypotheses. First, to assess whether the fMRI signal alone, without meaningful temporal alignment to language, is sufficient. For this, we fine-tuned with temporally misaligned fMRI data. Second, to test if encoding improvements could stem from a different multilingual representation without brain data, we used mBERT embeddings as a baseline.\n\n## Clarification on Downstream tasks\n### Response for Q1\nThank you for this helpful question. Models were fine-tuned separately for each participant and language (note Methods 3.3). We will explicitly highlight this in the paper to ensure clarity. The numbers reported in Table 1 correspond to Participant 1. We have revised Table 1 to report the average performance across all six bilingual participants, along with standard deviations. Due to space constraints, please see the *updated table 1* included in our response to reviewer hVbZ. As shown in the updated results, the overall trends remain consistent with those originally reported in the paper. We appreciate the suggestion since this change will help make the findings more interpretable and robust. \n\n### Response for Q2 and Q3\nWe agree with the concern regarding the small differences and sample size reported in Tables 1 and 2. \n\nRegarding Table 1, we have revised Table 1 to report the average performance across all six bilingual participants, along with standard deviations (see comment above). \n\nRegarding Table 2, we agree that the small sample size limited the strength of the conclusions. In response to your feedback, we identified two additional monolingual participants from another dataset (LeBel et. al, 2023) and have now included them in the analysis. Due to space constraints, please see the *updated table 2* included in our response to reviewer hVbZ. With this expanded comparison group (3 monolingual participants), the results are consistent across participants: bilingual brain fine-tuning consistently outperforms the mBERT baseline across tasks and offers slightly stronger generalization than fine-tuning on individual monolingual participants. Bilingual brain fine-tuning outperforms monolingual fine-tuning on 7 out of 8 tasks on the CLUE benchmark, with equal performance on the remaining task (ChID). We thank the reviewer for this suggestion as we think this extended analysis better supports the contribution of our paper and makes our results more convincing.\n\n[LeBel et al., 2023] A natural language fMRI dataset for voxelwise encoding models \n\n## Clarification on model choice\nPlease note that the supplementary material is indeed different from the appendix; it is separately downloadable. The supplementary material (referred to in the main text) includes comparison results for various model architectures: across mBERT, XLM-R, and XGLM. We apologize for the oversight and have now also included results for LLaMA-3.2-1B.\n\nDue to space constraints, please see the results and comparison for mBERT, XLM-R, and XGLM (*updated Supplementary Table 2*) in our response to reviewer xVwE. This table shows that fine-tuning improves performance on most tasks: mBERT (7/9 tasks), XLM-R (9/9), and XGLM (6/9), with especially large gains for XLM-R (e.g., RTE: 54.15 → 62.71, MRPC: 69.61 → 77.94).\n\nDue to space constraints, please see the results for LLaMA-3.2-1B in our response to reviewer 4NET. Bilingual brain fine-tuning outperforms the vanilla model on 2 out of 3 tasks. Please note that, due to computational constraints, LLaMA was fine-tuned using LoRA only at layer 9. As a result, the observed performance gains on downstream tasks are low compared to the other models.\n\nThank you for highlighting the points we had overlooked. Your feedback has helped us improve the clarity of the paper. We will incorporate all the tables and explanations provided above into the camera-ready version."}, {"review_id": "ah0vbHp98i", "reviewer": "Reviewer_hVbZ", "summary": "This work is part of a broader research effort exploring the potential benefits of incorporating brain data into the training of large language models (LLMs). Specifically, the authors empirically examine the effects of fine-tuning both monolingual and multilingual LLMs (e.g., BERT and mBERT) using bilingual (English and Chinese) brain data acquired through functional Magnetic Resonance Imaging (fMRI). The evaluation focuses on two aspects: voxel-wise brain activity reconstruction and performance on downstream natural language processing (NLP) tasks.\n\n### Contributions\n\n1. *Introduction of a novel data modality*: This study is the first to leverage bilingual brain data for fine-tuning LLMs.  \n2. *Development of an end-to-end brain-informed fine-tuning pipeline*: The authors propose a novel pipeline that improves upon previous, less integrated approaches.  \n3. *Comprehensive evaluation*: The study presents an extensive assessment of both monolingual and multilingual models fine-tuned with bilingual brain data, evaluating their performance on voxel-wise brain activity reconstruction and downstream NLP tasks in both English and Chinese.\n\n### Technical summary\n\nThis is primarily an empirical study, and its methodology involves the following components:\n\n1. *Data collection*: The authors collected a new bilingual fMRI dataset recorded while participants read naturalistic narratives in both English and Mandarin Chinese.  \n2. *Model fine-tuning pipeline*: The brain recordings were preprocessed and used to fine-tune several LLMs, including monolingual BERT and multilingual models such as mBERT, XLM-R, XGLM, and LLaMA-3.2. The fine-tuning procedure involves supervised learning, using token sequences from the narratives as inputs and the corresponding BOLD (Blood Oxygen Level Dependent) fMRI responses as the target. A projection layer maps the model representations to brain activity.\n\n### Experimental design/evaluation\n\nThe fine-tuned models are compared to their non-fine-tuned counterparts across two “dimensions”:\n\n1. *Voxelwise encoding modeling*: This analysis evaluates whether brain-informed fine-tuning enhances alignment between model representations and brain activity. The primary question is whether the fine-tuned model can better predict brain responses compared to the original (vanilla) model.  \n2. *Downstream NLP task performance*: Models are evaluated on standard NLP benchmarks, GLUE (for English) and CLUE (for Chinese), under several setups:  \n   a. Fine-tuning and evaluation in the same language;  \n   b. Cross-lingual transfer between English and Chinese (both languages are known to participant); \n   c. Zero-shot transfer to unseen languages, including French, German, Japanese, Spanish, and Korean (XGLUE/XTREME benchmarks).\n\n### Main findings\n\nAccording to the authors’ interpretation, the main findings are as follows:\n\n1. Fine-tuning improves voxelwise encoding performance.  \n2. Fine-tuning also improves performance on downstream NLP tasks in most cases, including within-language, cross-lingual, and zero-shot transfer scenarios.  \n3. Fine-tuning on bilingual brain data (English and Chinese) results in better performance in both known and unseen languages compared to fine-tuning on monolingual brain data (English only).", "questions": "I would like to thank the authors for the interesting paper and the considerable effort invested in this work. However, there are several points that I believe require further attention/work. I have divided these into *major* issues, which should be prioritized, and minor ones, which should be addressed if time permits.\n\n# Major Comments/Questions\n\nI will consider increasing the quality score and overall score (rating) if the following points are addressed:\n\n1. *Sample size*: While I fully understand the complexity associated with empirical research involving human participants, the sample size in this study appears to be very limited. I strongly encourage the authors to consider increasing the sample size by recruiting additional participants.  If expanding the entire dataset is only partially feasible, I suggest targeting a subset of the experiments for additional data collection and explaining that explicitly. In the case that further recruitment is not possible at all, please provide a clear justification for the current sample size and its sufficiency for the conclusions drawn.\n\n2. *Reporting variability*: Please report standard deviations (or standard errors) alongside mean values to provide a clearer picture of variability. Even with a small sample size (e.g., 6–7 participants), such measures are crucial for assessing the robustness and reliability of the results. Although the standard deviation may be less stable with a small sample size, its inclusion contributes to transparency.\n\n3. *Clarity in Table 1*: Table 1 presents some ambiguities regarding how performance metrics are reported. For instance, in the STS-B task from the GLUE benchmark, results are typically evaluated using Pearson or Spearman correlation coefficients, which range from [-1, 1]. However, the values reported fall in the 85–90 range, which is unclear.  \nIn addition to incorporating standard deviations (see point 2), I recommend the following:  \n- Clearly specify the evaluation metric used for each task (currently done only for tasks with more than one metric).  \n- Include reference ranges for chosen metrics.\nThese changes will improve clarity and facilitate a more accurate interpretation of both absolute and relative performance.\n\n4. *Monolingual participant experiments*: The experiment involving a monolingual (English-speaking) participant currently includes only one individual. I strongly recommend increasing the number of monolingual participants to match the bilingual sample size for comparability.  If additional recruitment is not feasible, I suggest re-positioning these results as preliminary, moving them to the appendix, and referencing them in the main manuscript. Additionally, revise the main text to ensure that claims are appropriately moderated. Should you decide to include more monolingual participants, please also apply recommendations from points (2) and (3) to this subset of the data.\n\n5. *Clarification on brain region analysis (Section 4.2)*: In Section 4.2, the paragraph titled *“Performance on downstream tasks with language- or semantically-selective voxels”* mentions the use of selective voxels. Does this imply that the results reported in Tables 1 and 2 are based on whole-brain data?  If so, and unless I have missed something, this distinction should be made explicit in the manuscript. Clarifying whether the reported results are based on whole-brain activity or specific voxel subsets is important for interpreting the findings.\n\n# Minor Comments/Questions\n\nWhile addressing the following points may not be critical to the paper’s core contributions, doing so would enhance the overall quality. I consider increasing the quality score and overall score (rating), but strongly encourage the authors to prioritize the major comments above in case of time limitations:\n\n1. *Line 137*: I would appreciate the inclusion of additional visual elements to clarify the model architecture, specifically the roles of the 3-lobe Lanczos, FIR filtering, and BOLD processes. Although these components are not introduced in this work, they are integral to the proposed method. Visual illustrations would be especially helpful for readers unfamiliar with these elements. Ideally, such visuals should be included in the main manuscript; at the very least, they should appear in the appendix.\n\n2. *Line 148*: Consider relocating the entire *Training Protocol* section to Appendix G to conserve space. This section does not provide method-specific insights central to the main narrative and could be referenced as needed.\n\n3. *Line 153*: First, please include an explicit reference to the appendix section where the loss functions are compared. Second, a brief explanation as to why NT-Xent appears to perform better would be helpful. If the reason is speculative or not yet understood, stating so would improve transparency and avoid ambiguity.\n\n4. *Figure 2*: The names of the brain regions are not clearly legible. Please improve the visibility of these labels. Additionally, consider referencing Figures 3 and 4 earlier in the text to clarify the distinction between language-selective and semantically-selective brain regions. Without a clear visual cue, it is difficult to interpret statements from Figure 2 captions such as: “Voxel colors reflect the best-performing model: black for vanilla, blue for fine-tuned with whole-brain, magenta for fine-tuned with language, and yellow for semantically-selective brain regions.”\n\n# General Advice\nThe manuscript presents a range of experimental design choices that are intellectually compelling and, in my view, constitute one of the strengths of the work (as I indicated above). However, these choices also lead to a combinatorial explosion of experimental configurations, which can be time-consuming to conduct properly. I recommend that the authors prioritize the most principled setups and place greater emphasis on improving statistical rigor and clarity in the reporting of results, in case of time limitations.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n\nI found this work to have the following strengths:\n\n*Clarity*: The manuscript is well-written and well-structured. The narrative is logical and easy to follow, with a clear separation between primary and secondary information, the latter being appropriately placed in the appendix. The experimental design is extensive and broad, covering various setups. \n\n*Originality*: To the best of my knowledge, the idea of using brain data from bilingual individuals, while simple, is quite novel conceptually. Since bilingual brains may encode conceptual information in a less language-specific way, this approach could plausibly enhance skill transfer and generalization when used to fine-tune large language models (LLMs).\n\n*Significance*: This work is significant in that it contributes to a better understanding of the parallels/interconnection between language regression models (such as LLMs) and neuroscientific findings from studies of bilingual individuals. It helps bridge the gap between artificial and biological representations of language. Furthermore, I believe this work opens promising directions for future research, such as exploring other brain modalities or studying individuals with additional cognitive abilities (e.g., multilingualism), that may further enhance conceptual understanding in LLMs.\n\n# Weaknesses\n\nFrom my perspective, the primary weaknesses of this study arise from the interpretation and presentation of results:\n\n1. *Small sample size*: The most significant limitation is the small number of participants, which the authors acknowledge in the limitations section. While this is understandable, given the complexity of the experimental setup, it limits the generalizability and statistical power of the findings.\n\n2. *No statistical significance reports*: The results are reported using mean values only, without accompanying measures of variability such as standard deviation or standard error. These metrics are critical for evaluating the robustness of the findings. Although this omission is transparently noted in the NeurIPS checklist (Section 7), it nonetheless weakens the statistical interpretation of the results.\n\n3. *Ambiguity in reported metrics*: Tables 1 and 2 lack clarity regarding the exact metrics reported and their corresponding ranges. For example, it is unclear in some cases which metric is being reported (unless explicitly specified), and the absence of reference ranges makes interpretation more difficult.\n\nFor a complete and detailed account of both major and minor issues, please refer to the “Questions” section.", "was_revised": true, "final_justification": "The authors have addressed my concerns point by point, and I am satisfied with their responses. I have accordingly raised the quality score and overall ranking.", "rebuttal": "## Major comments\n### Q1 sample size\nThank you for raising this important concern. Our study shows that brain-informed fine-tuning can reliably enhance multilingual understanding. We employ a small-N design, where a language model is fine-tuned and evaluated independently for each participant, enabling robust within-subject inference (Smith & Little, 2018). Importantly, each participant in our dataset contributes two hours of naturalistic brain recordings, which are critical for capturing the complexity of language representations (Hamilton & Huth, 2018) in the brain and for enabling meaningful learning through fine-tuning. While we recognize that larger sample sizes would strengthen the generalizability of our findings, data collection is not feasible at this stage due to current resource constraints (four 2.5 hours sessions for data collection, data analysis, preprocessing and subject recruitment time). That said, we fully agree with the reviewer’s suggestion and currently plan to expand the bilingual dataset and extend our main findings to new participants in future work. \n\nWe have increased the number of monolingual participants, please see response to Q4 below.\n\n[Smith & Little, 2018] Small is beautiful: In defense of the small-N design\n\n[Hamilton & Huth, 2018] The revolution will not be controlled: natural stimuli in speech neuroscience\n\n### Q2 Reporting variability\nModels were fine-tuned separately for each participant and language (note Methods 3.3). We have now revised Table 1 to report the average performance across all six bilingual participants, along with standard deviations. As shown in the updated results (see below), the results remain consistent with those originally reported in the paper. We thank the reviewer for this helpful feedback as this change should help better demonstrate the robustness of our findings.\n\n**Updated Table 1, a) Fine-tuning and Evaluation in the Same Language**\n|GLUE Task|BERT-en|BERT-ft-en (Mean ± Std)|mBERT|mBERT-ft-en (Mean ± Std)|\n|-|-|-|-|-|\n|CoLA (MCC)|53.38|**55.11 ± 0.75**|**42.68**|39.58 ± 1.82|\n|SST-2 (Acc.)|92.08|**92.58 ± 0.39**|89.68|**90.64 ± 0.54**|\n|MRPC (Acc.)|79.41|**80.55 ± 1.01**|84.80|**85.39 ± 0.51**|\n|MRPC (F1) | 86.27 | **86.91 ± 0.75** | 88.56 | **89.20 ± 0.27**|\n|STS-B (Pears.)| 88.06|**88.10 ± 0.07**|88.06|**88.47 ± 0.09**|\n|STS-B (Spear.)|**87.65**|87.55 ± 0.18|87.76|**88.28 ± 0.12**|\n|QQP (Acc.)|**90.84**|90.79 ± 0.03|90.22|**90.43 ± 0.04**|\n|QQP (F1)|87.70|**87.71 ± 0.19**|86.70|**87.08 ± 0.02**|\n|MNLI-m (Acc.)|84.38|**84.40 ± 0.09**|82.09|**82.26 ± 0.27**|\n|MNLI-mm (Acc.)|**84.64**|84.55 ± 0.20|82.38|**82.73 ± 0.14**|\n|QNLI (Acc.)|91.45|**91.49 ± 0.09**|91.14|**91.24 ± 0.06**|\n|RTE (Acc.)|67.15|**67.32 ± 0.17**|**67.15**|65.63 ± 0.51|\n|WNLI (Acc.)|49.30|**55.59 ± 1.31**|53.52|**56.34 ± 0.00**|\n\n|CLUE Task|BERT-zh|BERT-ft-zh (Mean ± Std)|mBERT|mBERT-ft-zh (Mean ± Std)|\n|-|-|-|-|-|\n|AFQMC (Acc.)|**75.25**|74.50 ± 0.48|69.74|**70.70 ± 0.99**|\n|CMNLI (Acc.)|80.50|**80.87 ± 0.04**|78.66|**78.99 ± 0.14**|\n|CSL (F1.)|80.18|**80.82 ± 0.42**|81.10|**81.60 ± 0.56**|\n|IFLYTEK (Acc.)| 60.25|**60.48 ± 0.25**|56.52|**56.92 ± 0.17**|\n|TNEWS (Acc.)|56.24|**56.48 ± 0.08**|54.77|**55.08 ± 0.16**|\n|TNEWS (F1.)|55.17|**55.39 ± 0.08**|53.69|**53.81 ± 0.20**|\n|ChID (Acc.)|10.66|10.66 ± 0.00|10.66|10.66 ± 0.00|\n|C³ (F1.)|49.74|**49.83 ± 0.39**|49.42|**50.25 ± 0.27**|\n\n**Updated Table 1, b) Cross-Language Transfer Between Known Languages**\n\n|GLUE Task|BERT-zh|BERT-ft-zh (Mean ± Std)|mBERT|mBERT-ft-zh(Mean ± Std)|\n|-|-|-|-|-|\n|CoLA (MCC)|51.13|**53.82 ± 1.18**|40.96|**40.99 ± 1.08**|\n|SST-2 (Acc.)|92.26|**92.77 ± 0.15**|88.27|**90.30 ± 0.66**|\n|MRPC (Acc.)|76.96|**77.53 ± 0.28**|82.84|**85.20 ± 0.93**|\n|MRPC (F1)|83.38|**84.39 ± 0.17**|87.15|**89.16 ± 0.78**|\n|STS-B (Pears.)|87.21|**87.96 ± 0.12**|87.09|**88.30 ± 0.17**|\n|STS-B (Spear.)|86.73|**87.67 ± 0.18**|86.97|**88.16 ± 0.23**|\n|QQP (Acc.)|90.65|**90.86 ± 0.12**|89.91|**90.39 ± 0.13**|\n|QQP (F1)|87.53|**87.76 ± 0.04**|86.27|**87.05 ± 0.09**|\n|MNLI-m (Acc.)|84.00|**84.16 ± 0.05**|81.52|**81.97 ± 0.14**|\n|MNLI-mm (Acc.)|83.91|**84.16 ± 0.10**|81.94|**82.78 ± 0.28**|\n|QNLI (Acc.)|91.27|**91.42 ± 0.05**|90.55|**91.23 ± 0.26**|\n|RTE (Acc.)|67.15|**67.91 ± 0.08**|66.06|**67.21 ± 0.67**|\n| WNLI (Acc.) | 53.52 | **56.34 ± 0.00** | 54.93 | **55.25 ± 0.84** |\n\n| CLUE Task | BERT-en | BERT-ft-en (Mean ± SD) | mBERT | mBERT-ft-en (Mean ± SD) |\n|-|-|-|-|-|\n|AFQMC (Acc.)|69.00|**69.02 ± 0.03**|69.74|**71.25 ± 0.97**|\n|CMNLI (Acc.)|68.34|**68.78 ± 0.13**|78.66|**78.87 ± 0.07**|\n|CSL (F1)|71.20|**72.33 ± 0.66**|81.10|**81.51 ± 0.20**|\n|IFLYTEK (Acc.)|**47.86**|47.27 ± 0.56|56.52|**56.92 ± 0.31**|\n|TNEWS (Acc.)|50.92|**51.42 ± 0.16**|54.77|**54.99 ± 0.20**|\n|TNEWS (F1)|50.15|**50.68 ± 0.07**|53.69|**53.76 ± 0.04**|\n|ChID (Acc.)|10.66|10.66 ± 0.00|10.66|10.66 ± 0.00|\n|C³ (F1)|42.64|**42.99 ± 1.49**|49.42|**49.72 ± 0.35**|\n\n**Updated Table 1, c) Zero-Shot Transfer to Unseen Languages**\n\n|XGLUE Task|German (de)|French (fr) | Spanish (es) | Japanese (ja) |Korean (ko)|\n|-|-|-|-|-|-|\n| | mBERT / mBERT-ft-en | mBERT / mBERT-ft-en | mBERT / mBERT-ft-en | mBERT / mBERT-ft-en | mBERT / mBERT-ft-en |\n| PAWS-X (Acc.) | **84.00** / 83.62 ± 1.10 | **88.10** / 87.17 ± 0.81 | **87.05** / 86.50 ± 0.42 | **78.70** / 76.55 ± 2.12 | **78.75** / 76.90 ± 0.35 |\n| XNLI (Acc.) | 70.60 / **70.94 ± 0.14** | 72.69 / **73.30 ± 0.80** | 74.10 / **74.62 ± 0.05** | 86.00 / **87.97 ± 0.05** | 86.00 / **87.82 ± 0.25**|\n| NER (F1)| 10.67 / **12.85 ± 2.02** | 06.67 / **11.09 ± 1.09** | 07.33 / **12.11 ± 0.67** | **13.00** / 11.34 ± 2.11 | 12.00 / **12.25 ± 1.71** |\n### Q3 Clarity in Table 1\nWe understand how this may have caused confusion. To maintain consistency across tasks, we followed the common practice used in the GLUE leaderboard and reported all metrics on a 0-100 scale. For example, a Pearson correlation of 0.88 is reported as 88.0. To improve clarity, we will add metrics adjacent to the task (like in the table above) and a footnote to all relevant tables explicitly indicating the evaluation metric range and the scaling. \n\nFor reference, please see the table below:\n| GLUE Task | Metric | Reference Range | Scaled |\n|-|-|-|-|\n| CoLA | Matthews Correlation | [-1, 1] | ×100 |\n| SST-2 | Accuracy | [0, 1] | ×100 |\n| MRPC | F1 Score / Accuracy | [0, 1] | ×100 |\n| STS-B | Pearson / Spearman Correlation | [-1, 1] | ×100 |\n| QQP | F1 Score / Accuracy | [0, 1] | ×100 |\n| MNLI | Accuracy | [0, 1] | ×100 |\n| QNLI | Accuracy | [0, 1] | ×100 |\n| RTE | Accuracy | [0, 1] | ×100 |\n| WNLI | Accuracy | [0, 1] | ×100 |\n\n| CLUE Task | Metric| Reference Range | Scaled |\n|-|-|-|-|\n| AFQMC | Accuracy | [0, 1] | ×100 |\n| CMNLI | Accuracy | [0, 1] | ×100 |\n| CSL | F1 Score | [0, 1] | ×100 |\n| IFLYTEK | Accuracy| [0, 1] | ×100 |\n| TNEWS | Accuracy / F1 Score | [0, 1] | ×100 |\n| ChID | Accuracy | [0, 1] | ×100 |\n| C³ | Exact Match / F1 Score | [0, 1] | ×100 |\n\n| XGLUE Task | Metric | Reference Range | Scaled |\n|-|-|-|-|\n| PAWS-X | Accuracy | [0, 1] | ×100 |\n| XNLI | Accuracy | [0, 1] | ×100 |\n| NER | F1 Score | [0, 1] | ×100 |\n\n### Q4 Monolingual participant experiments\nIn response to your feedback, we identified two additional monolingual participants from another dataset (LeBel et. al, 2023) and have now included them in the analysis. With this expanded comparison group (3 monolingual participants), the results show that bilingual brain fine-tuning consistently outperforms the mBERT baseline across tasks and offers slightly stronger generalization than fine-tuning on individual monolingual participants. Bilingual brain fine-tuning outperforms monolingual fine-tuning on 7 out of 8 tasks on the CLUE benchmark, with equal performance on the remaining task (ChID). We thank the reviewer for this suggestion as we think this extended analysis better supports the contribution of our paper and makes our results more convincing.\n\n**Updated Table 2. b) Cross-Language Transfer Between Known Languages**\n| CLUE Task | mBERT | Bilingual mBERT-ft-en (Mean ± SD) | Monolingual mBERT-ft-en (Mean ± SD) |\n|-|-|-|-|\n| AFQMC (Acc.) | 69.74| **71.25 ± 0.97**|70.33 ± 0.48 |\n| CMNLI (Acc.)| 78.66| **78.87 ± 0.07**|78.60 ± 0.12 |\n| CSL (F1)| 81.10| **81.51 ± 0.20**|81.09 ± 0.11 |\n| IFLYTEK (Acc.)| 56.52|**56.92 ± 0.31**|56.87 ± 0.37 |\n| TNEWS (Acc.) | 54.77| **54.99 ± 0.20**|54.61 ± 0.30 |\n| TNEWS (F1) | 53.69| **53.76 ± 0.04**|53.42 ± 0.09 |\n| ChID (Acc.)| 10.66| 10.66 ± 0.00|10.66 ± 0.00 |\n| C³ (Acc.)| 49.42| **49.72 ± 0.35**|49.28 ± 0.22 |\n\n[LeBel et al., 2023] A natural language fMRI dataset for voxelwise encoding models \n### Q5 Clarification on brain region analysis\nThe reviewer is correct that Tables 1 and 2 are based on fine-tuning with whole-brain data. This is noted in the captions but we will revise this and make this distinction more explicit both in the main text and the caption.\n\n## Minor comments\n**Q1:** We agree this would improve clarity and will include graphical illustrations of the requested components in the appendix for the camera-ready version.\n\n**Q2:** We agree with the reviewer and will move the Training Protocol section to Appendix G in the camera-ready version.\n\n**Q3:** Thank you for this suggestion. We would like to note that contrastive loss yielded better brain encoding performance (mean r = 0.163, SE = 0.0097), slightly outperforming Ridge (mean r = 0.1622) and MSE (mean r = 0.1612). Given that contrastive objectives are commonly used in prior brain decoding studies and offer strong alignment with brain representations (Défossez et al., 2023; Levy et al., 2025), we retained it as our main objective to ensure comparability with existing literature. We will revise the reference to Supplementary Table 1 and include this reasoning in the revised version for clarity.\n\n**Q4:** Thanks for highlighting this; we indeed overlooked this. We’ll improve label visibility, enlarge ROI names, and reference Figures 3 and 4 earlier to ensure clarity."}, {"review_id": "JrBfk3tSEj", "reviewer": "Reviewer_4NET", "summary": "This article investigates the consequences of fine-tuning a monolingual or multilingual language model with fMRI brain data using either monolingual or bilingual participants. The models are evaluated on their ability to predict brain activity on a held-out dataset and on natural language processing benchmarks. According to the authors, the results show that fine-tuning on brain data from bilingual subjects improves the performance of the models on all these tasks, implying that the models benefited from the neural representations of the bilingual participants.", "questions": "See Strengths And Weaknesses section above.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The paper is rather well structured, well referenced and well written, and tackles an interesting question using a large variety of tasks and control conditions. It makes use of an fMRI dataset that seems to be novel and could be of interest in itself.\n\nThe key question, \"whether fine-tuning language models with bilingual brain data can elicit multilingual capabilities in language models\", is only supported by quite fragile results that are in my view over-interpreted.\nFine-tuning using bilingual brain data is said to be the cause of the improved performance of the multilingual capabilities of language models, and this point is key to differentiate this paper from existing literature. In spite of the many experiments, the results are not so convincing and are reported in a way that favors the hypotheses of the authors, but after looking at the results with more scrutiny, the results do not strongly support the claim (see below for more details).\n\nDespite clear qualities in the document as listed above, this work is not quite polished, and there is some missing content, although it is announced in the paper that it should be there. In particular, I could not find the results with the Llama-3.2 model, which would have been interesting as it is the only decoder based model, and also the only recent model that the authors investigate (this model is announced on p.4, but is not in appendix nor in supplementary information). Moreover, the results with the alternative loss function coined \"hybrid\" in the paper are not reported in Supplementary as announced (see section 3.3 p.4).\n \nAlthough the paper produces a lot of results, these are actually not much discussed. It might have been interesting to discuss whether the type of task in which the models improve or not makes sense. More importantly, what about the other models that were tested? As said before, the Llama-3.2 results are not presented, but the Supplementary present results for XLM-R and XGLM. The results for these models are reported in a way that makes it not possible to check whether it actually works: the bold face seems to indicate which fine-tuned model wins, but the point is about base model vs fine-tuned model, so these results should be reported, and they are missing. Only then can we appreciate whether fine-tuning these models on brain data improves their NLP abilities.\n\nGiven the main point of the paper, it would be nice to include a control task where we would expect to see no difference after fine-tuning, such as a non-language task like arithmetic. \n\nThe submission would strongly benefit from sharing the fMRI data that was collected, as the scientific community working on multilingualism in the brain and LLMs might benefit from it.\n\nHere are some more detailed comments and suggestions.\n\np.4\nWhy use the last hidden layer for the representation? \nIt is well know that peak predictivity between hidden states of language models and brain activity comes from middle layers (see  e.g. Jain & Huth, 2018; Toneva & Wehbe, 2019; Caucheteux & King 2022, among others). In fact this is what is done in section 3.4, for the voxel-wise encoding (by the way, the precise layer is given for BERT, layer 7, but not for all the other models that were studied). It seems that the goal is the same: predict brain activity using the representations learned by the LLM. This is also important given the paper's focus on multilingualism and existing works that show the interplay between layer depth and multilingual computations (e.g. Wendler et al, 2024; Zhao et al, 2024).\n\np.4\nIn order to facilitate the comparison between the different losses that were studied, it would be helpful to put all of them in the same table (see table in Supplementary; by the way, the so-called \"hybrid\" loss is missing here, contrary to what is stated on p.4). Moreover, the results for mBERT-ft-en with Ridge seem to be clearly better, it does not seem obvious that NT-Xent is better. Having all of them side-by-side might help the comparison.\n\np.6 Fig. 2\nThere are many improvements located in the visual areas (same for other models, see Fig. 1 in supplementary info). It looks like fine-tuning finds its way, but in a way that is not related to language, which might go counter the main argument. This deserves more discussion.\n\np.6 \nThe control baselines are appreciated but hard to read. Some comparative statistics might help, as would a visualization on a brain map of the difference between brain-informed over control. Same thing for the cross-participants results, the maps are not easy to assess objectively (see e.g. participant 5), and descriptive statistics might help (e.g. mean $\\Delta$r per subject). \n\np.7 \nThere seems to be an issue with the results relative to the WNLI task. All the results on fine-tuned models (but not the base vanilla model) have a performance of 56.34, which improves over the original model, but is the exact same value whatever the conditions, for BERT-ft-en, mBERT-ft-en, BERT-ft-zh, mBERT-ft-zh (p. 8), mBERT-ft-en, bilingual or monolingual (p.9) and even the models with alternative losses (Supplementary p.1) and the other models provided in Supplementary p.3. \n\np.7 \nOverall, the results are not so clear and the effects quite small. These results would be more meaningful if they could be replicated with other participants and models. \nThe part with the models is supposed to be done, but the results are not really reported as far as I understand.\n\np.7\nConcerning the results on fine-tuning on bilingual brain data vs monolingual ones. This is an important section given the point of the paper. The reporting of the results seems misleading. Ignoring the WNLI results (see above), if one looks at Table 2, the bilingual vs monolingual battle is 4/8 vs 4/8 for English (rather than 5/7 as reported) and 4/7 for Chinese (rather than 5/7; one is actually a tie). So the results are fragile, which is concerning given the importance of this in the general point the authors are making (see Discussion notably).\nIt seems important to study these effects on other participants and other models.\n\nMore minor points.\n\np. 4 What is the dropout value that you use? How was it chosen, and how does it affect the results?\n\nFig. 2 p.6 There is an error when calling appendix D: it is actually E.2\n\nTable 5 p.26 Ordering the tasks as in the table of the main paper would ease the comparisons.", "was_revised": true, "final_justification": "I believe that the proposed changes, with the new results and discussion, will improve the manuscript. The effects that are observed, although interesting, are quite subtle, which limits the overall impact of the work, but there is still value in this original work. As a consequence, I raise my score to 4.", "rebuttal": "## Inclusion of control task\nThank you for the helpful suggestion. We ran a control experiment on the GSM8K dataset (arithmetic reasoning task), and as expected found no performance gains; both vanilla and fine-tuned models achieved 0 accuracy and F1.\n\n## p.4 Clarification on usage of last hidden layer for the representation\nThank you for this important question. We think the confusion arises from the assumption that both analyses, brain-informed fine-tuning and voxel-wise encoding have the same goal, i.e., predicting brain activity from pretrained LLM representations. \n\nThis is not the case. We have three scenarious in our analysis. First, in brain-informed fine-tuning (Section 3.3) our aim is to induce a brain-informed bias into the language model’s representations by performing full end-to-end fine-tuning of all layers using brain data, without any layer selection. Second, we use the representations from the last hidden layer,following standard practice for downstream NLP tasks. Third, Once the model is fine-tuned, our goal is to evaluate how these brain-tuned language model representations perform, particularly in how well they predict brain activity. In this third case, we follow established provedures from the literature and focus on intermediate layers (e.g., layer 7 for BERT). We will revise the section to make this explicit and ensure clarity.\n\n## p.4 Comparison of different losses\nThank you for this important point, and apologies for the oversight regarding the missing hybrid loss. Please see the updated Supplementary Table 1 below for a full side-by-side comparison of all five loss functions, including the newly added hybrid loss, for mBERT-ft-en across GLUE tasks.\n\n**Updated Supplementary Table 1 a) Fine-tuning and Evaluation in the Same Language**\n| GLUE Task| mBERT| Contrastive | MSE| Ridge| Spatial | Hybrid|\n|-|-|-|-|-|-|-|\n| CoLA| 42.68| 40.05| 43.28| **44.40**| 41.14| 40.36| \n| SST-2| 89.68| 90.14| 88.53| **91.17**| 90.94| 90.37| \n| MRPC (Acc.) | 84.80| 84.80| 84.10| **86.27**| 84.56| 84.07| \n| MRPC (F1)| 88.56| 89.01| 88.59| **89.96**| 88.69| 88.33| \n| STS-B (Pearson)| 88.06| **88.42**| 86.14| 88.36| 85.84| 87.88| \n| STS-B (Spearman)| 87.76 | 88.22| 86.10| **88.25**| 85.96| 87.82| \n| QQP (Acc.)| 90.22| 90.47| 90.36| **90.57**| 90.40| 90.55| \n| QQP (F1)| 86.70| 87.07| 87.01| **87.24**| 86.97| 87.23| \n| MNLI-m| 82.09| **82.66**| 81.27| 82.33| 82.24| 82.31| \n| MNLI-mm | 82.38| 82.97| 82.23| **82.98**| 82.69| 82.83|\n| QNLI| 91.14| 91.18| 91.38| 91.34| 91.38| **91.41**| \n| RTE | 67.15| 65.70| 67.51| 66.06| **68.59**| 65.62| \n| WNLI| 53.52| 56.34| 56.34| 56.34| 56.34| **57.75**|\n\nWe agree that Ridge loss achieves the strongest downstream performance on several tasks. We would like to note that contrastive loss yielded the better brain encoding performance (mean r = 0.163, SE = 0.0097), slightly outperforming Ridge (mean r = 0.1622) and MSE (mean r = 0.1612). Given that contrastive objectives are commonly used in prior brain decoding studies and offer strong alignment with brain representations (Défossez et al., 2023; Levy et al., 2025), we retained it as our main objective to ensure comparability with existing literature.\n\nWe will revise the manuscript to clarify this rationale and include both within-language and cross-language results for all five loss functions (including the hybrid loss) in the Appendix to provide a comprehensive side-by-side comparison.\n\n[Défossez et al., 2023] Decoding speech perception from non-invasive brain recordings\n\n[Levy et al., 2025] Brain-to-Text Decoding: A Non-invasive Approach via Typing\n## p.6 Clarification on improvements in visual areas\nThank you for this thoughtful observation. We agree that improvements in visual areas warrant further discussion. Please note this is a reading experiment, where variation in visual brain areas is correlated with stimulus presentation, especially in naturalistic settings. In addition, semantic features from language models are known to spuriously predict brain activity in visual areas even after regressing out low-level visual information (Oota et al., 2024; Deniz et al., 2019). Importantly, our analyses using semantically selective voxels (which explicitly exclude visual regions) show consistent performance gains from brain-informed fine-tuning. This indicates that the improvements are not driven by visual areas but reflect genuine alignment with higher-level semantic processing. We will add a discussion of this point in the revised manuscript.\n\n[Oota et al., 2024] Speech language models lack important brain relevant semantics\n\n[Deniz et al., 2019] The Representation of Semantic Information Across Human Cerebral Cortex During Listening Versus Reading Is Invariant to Stimulus Modality \n## p.6 Comparative statistics for control baselines\nThank you for the helpful suggestion. We will include visualizations of differences in encoding performance in the brain to better highlight improvements from brain-informed fine-tuning over controls. Additionally, we’ll provide descriptive statistics, including mean Δr per subject, across both baselines, models, and cross-participant analyses in the camera-ready version. For now, here is a summary statistic for BERT-en, across participants (for well-predicted voxels; r>0.1) to aid interpretation:\n- Δr̄ (vanilla – fine-tuned): -0.037 ± 0.00037\n- Δr̄(vanilla – participant transfer): -0.00055 ± 0.00029\n- Δr̄(vanilla – control (TR shuffle)): 0.133\n- Δr̄(vanilla – control (mBERT)): 0.136\n\nAs can be seen from the summary statistics reported above, our results remain consistent taking Δr in account.\n\n## p.7 Clarification on WNLI task\nThank you for pointing this out. The identical score (56.34) across some fine-tuned models on WNLI is due to the small dataset size and integer-valued confusion matrices (please refer to GLUEBenchmark FAQ). Note that our fine-tuned models do yield slightly different predictions, but the resulting accuracy rounds to the same value. For example, here are the confusion matrices for three different fine-tuned models (for 1 participant):\n- BERT-ft-en: [[33, 18], [13, 7]]\n- mBERT-ft-en: [[32, 17], [14, 8]]\n- mBERT-ft-zh: [[34, 19], [12, 6]]\nWe've updated Table 1 to report mean ± standard deviation across participants to better highlight subtle differences and clarify variability.\n\n## p.7 Comparison across participants and models\nThank you for the helpful feedback. We have now revised Table 1 to report the average performance across all six bilingual participants, along with standard deviations. All models were fine-tuned separately for each participant and language. The overall trends remain consistent with those initially reported, helping strengthen the robustness of the findings. Due to space constraints, please see the *updated supplementary table 2* included in our response to reviewer xVwE. Regarding model comparisons, the current supplementary material includes fine-tuned results for mBERT, XLM-R, and XGLM. We apologize for the oversight in not including the performance of their corresponding vanilla models and LLaMA-3.2-1B. Please see the updated table below. This table shows that fine-tuning improves performance on most tasks: mBERT (7/9 tasks), XLM-R (9/9), and XGLM (6/9), with especially large gains for XLM-R (e.g., RTE: 54.15 → 62.71, MRPC: 69.61 → 77.94).\n\nPlease see below for results with LLaMA‑3.2‑1B fine-tuned using our brain-informed approach with LoRA adaptation applied to layer 9 (similar to Antonello et al., 2024). Models were evaluated on three downstream datasets (Wang et al. 2019, Sakaguchi et al., 2019). Brain-informed fine-tuning yields improvements over the base model across two datasets, further supporting the generalizability of our approach to different architectures and low-parameter adaptation methods (LoRA). We note that improvements are small, likely due to the parameter-efficient adaptation (LoRA on a single layer). Unfortunately, we couldn’t perform full fine-tuning on this model due to computational constraints\n| Dataset | LLaMA‑3.2‑1B | LLaMA‑3.2‑1B‑ft-en|LLaMA‑3.2‑1B‑ft-zh|\n|-|-|-|-|\n| BoolQ | 0.5790 | **0.5848** | **0.5830** |\n| COPA | 0.5534 | 0.5534 | 0.5534 |\n| Winogrande | 0.4945 | **0.4961** | **0.4961** |\n| **Average** | 0.5423 | **0.5447** | **0.5441** |\n\nWe will update the supplementary section to include these results for completeness.\n\n[Wang et al. 2019] SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems, NeurIPS-2019\n\n[Sakaguchi et al. 2019] WinoGrande: An Adversarial Winograd Schema Challenge at Scale, 2019\n\n## p.7 Clarification and extension of bilingual vs monolingual brain data\nThank you for pointing this out. To address the reviewer’s concern about robustness, we increased the number of monolingual participants by adding two additional participants from another dataset (LeBel et al., 2023). Due to space constraints, please see this extended comparison with 3 monolingual participants (*updated table 2*) included in our response to reviewer hVbZ. The results show that bilingual brain fine-tuning consistently outperforms the mBERT baseline across tasks and offers slightly stronger generalization than fine-tuning on individual monolingual participants. Bilingual brain fine-tuning outperforms monolingual fine-tuning on 7 out of 8 tasks on the CLUE benchmark, with equal performance on the remaining task (ChID). We will correct the misleading wording and be more explicit in the final version.\n\n[LeBel et al., 2023] A natural language fMRI dataset for voxelwise encoding models \n\n**Minor point; p.4:** We use a dropout of 0.2 for brain-informed fine-tuning to mitigate overfitting on limited brain data, and 0.1 for downstream tasks, following standard BERT fine-tuning defaults. We will include this information in the camera ready version.\n\n**Minor point; Fig. 2 p.6 and Table 5 p.26:** Thanks for catching this. We’ll correct the appendix reference and reorder tasks accordingly."}, {"review_id": "UDPNz1CB4b", "reviewer": "Reviewer_xVwE", "summary": "The paper introduces a novel approach for improving multilingual understanding in language models by finetuning them using brain activity recorded from bilingual individuals reading the same stories in English and Chinese. The proposed method aligns model representations with bilingual brain data to induce shared semantic representations. The authors finetune both monolingual and multilingual models using fMRI data and evaluate their performance on brain encoding and standard NLP benchmarks across multiple languages. Results show consistent improvements: finetuned models better predict brain responses and outperform their vanilla counterparts in downstream tasks, not only in the finetuned language but also in the participants’ second language and even in unseen languages. Importantly, these gains are specific to bilingual brain data and not replicable with monolingual data, suggesting that shared bilingual neural representations contribute to broader language generalization.", "questions": "1. Why did you focus mainly on encoder-based models (e.g., BERT, mBERT) for brain-informed finetuning? Did you observe any challenges or limitations when attempting to finetune decoder-only models like LLaMA?\n2. You integrated downsampling and temporal-delay modeling directly into the model architecture. Did you test the impact of this integration via an ablation (e.g., using fixed preprocessing instead)?\n3. Given the relatively small number of stories used per language, how sensitive is the finetuning process to the quantity of brain data? Would be interesting to try some sort of scaling laws for the proposed finetuning method.\n4. Do you have any insight into what aspects of language (e.g., semantics, syntax) are being shaped or improved by the brain-informed finetuning?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths**\n1. This paper presents a timely and compelling contribution to the NeuroAI community. It is the first to explore brain-informed fine-tuning using bilingual brain data, demonstrating its benefits not only for brain predictivity but also for performance on downstream NLP tasks such as GLUE. \n2. The proposed fine-tuning pipeline is technically novel, integrating both temporal delay modeling and downsampling directly into the model architecture, which sets it apart from prior approaches that treat these as preprocessing steps. \n3. The authors also provide a thorough and well-documented explanation of their voxelwise encoding methodology, employing 3-lobe Lanczos interpolation, finite impulse response (FIR) filtering, validation-based layer selection, and evaluating on held-out stories, all of which enhance reproducibility and exemplify best practices for measuring brain-model alignment.\n4. The paper includes several insightful analyses, such as comparisons of training objectives, evaluations across same-language, cross-language, and zero-shot generalization settings, and examinations of finetuning effects using language-selective, semantically-selective, and whole-brain regions which I personally found quite interesting.\n\n\n**Weaknesses**\n1. The main limitation of the work lies in the use of relatively outdated model architectures for fine-tuning, encoder-only models. The authors mention that results for LLaMA-3.2-1B (a more modern decoder-based model) are included in the supplementary materials (line 129), no such results are actually provided. Including such results would be informative to know whether the benefits of brain-informed finetuning scales or not with more recent larger models.\n2. While the proposed finetuning pipeline is novel, the paper does not include ablations to assess the individual contribution of components like temporal delay modeling or the specific interpolation method. This makes it hard to isolate what aspects of the pipeline are most critical to the observed improvements.\n3. The presentation of results could be significantly improved. The tables are difficult to parse, with multiple subtables embedded within a single table definition, which hampers readability. \n4. The paper does not include any quantitative tables comparing the brain encoding performance across different models, relying instead on qualitative inspection of brain surface plots. To strengthen the evaluation, I recommend reporting average Pearson correlation scores (either across all voxels or within specific ROIs) to allow for more rigorous and interpretable model comparisons.\n5. Finally, many critical results that substantiate the paper's core claims are relegated to the appendix. It would strengthen the paper to move such results into the main text to better support the narrative and improve clarity for the reader.\n\n**Typo**\n\nLine 341: captures → captured", "was_revised": true, "final_justification": "The authors have conducted additional experiments and provided reasonable responses to some of my remaining concerns. However, the paper still lacks more experiments for a stronger accept (e.g., running on larger scale models) and improving the clarity of the paper.", "rebuttal": "## Q1.1\nThank you for this question. We focused primarily on encoder-based models (BERT, mBERT) for two reasons:\n- Consistency with prior work: Previous brain-informed tuning approaches with monolingual english brain data have predominantly used encoders (e.g., BERT, Wav2Vec, Whisper, WavLM) due to their strong alignment with text and speech features.\n- Bilingual dataset requirements: Our dataset contains bilingual brain recordings. To ensure architectural uniformity across English and Chinese experiments, we used BERT-base, BERT-Chinese, and mBERT, which share a comparable encoder backbone.\n\nHowever, our experiments are not limited to encoders. We also fine-tuned XLM-R (encoder) and XGLM (decoder), showing that our brain-informed fine-tuning approach generalizes across architectures (see Supplementary Table 2). This table shows that fine-tuning improves performance on most tasks: mBERT (7/9 tasks), XLM-R (9/9), and XGLM (6/9), with especially large gains for XLM-R (e.g., RTE: 54.15 → 62.71, MRPC: 69.61 → 77.94).\n\n**Updated Supplementary Table 2. a) Fine-tuning and Evaluation in the Same Language**\n| **GLUE Task** | **mBERT** | **mBERT‑ft‑en** | **XLM‑R‑en** | **XLM‑R‑ft‑en** | **XGLM‑en** | **XGLM‑ft‑en** |\n|---|---|---|---|---|---|---|\n| CoLA            | **42.68** | 40.05 | 46.30 | **50.20** | 40.11 | **41.22** |\n| SST‑2           | 89.68 | **90.14** | 92.16 | **92.55** | **90.94** | 90.48 |\n| MRPC (Acc.)     | 84.80 | **84.80** | 69.61 | **77.94** | 71.32 | **71.81** |\n| MRPC (F1)       | 88.56 | **89.01** | 81.55 | **84.10** | 81.52 | **82.39** |\n| STS‑B (Pearson) | 88.06 | **88.42** | 84.96 | **85.18** | 81.43 | **82.36** |\n| STS‑B (Spearman)| 87.76 | **88.22** | 84.84 | **85.10** | 81.47 | **82.41** |\n| QQP (Acc.)      | 90.22 | **90.47** | 90.28 | **90.61** | 88.93 | **90.11** |\n| QQP (F1)        | 86.70 | **87.07** | 87.05 | **87.45** | 83.11 | **85.33** |\n| MNLI‑m          | 82.09 | **82.66** | 84.16 | **84.46** | 76.62 | **78.85** |\n| MNLI‑mm         | 82.38 | **82.97** | 84.22 | **84.40** | 78.80 | **79.01** |\n| QNLI            | 91.14 | **91.18** | 90.17 | **90.24** | **88.94** | 88.44 |\n| RTE             | **67.15** | 65.70 | 54.15 | **62.71** | **53.79** | 53.43 |\n| WNLI            | 53.52 | **56.34** | 43.66 | **56.34** | 50.34 | **52.41** |\n\n## Q1.2\nWhen applying brain-tuning to decoder-only models such as XGLM and LLaMA, we observed two challenges: \n- data efficiency; these larger generative models require more neural data and are prone to overfitting\n- task mismatch; our downstream evaluations focus on understanding rather than generation. \n\nDespite this, we observed small gains, suggesting that the method is architecture-agnostic but currently more effective on encoder models given data and task constraints.\n\nPlease see below for results with LLaMA‑3.2‑1B fine-tuned using our brain-informed approach with LoRA adaptation applied to layer 9 (similar to Antonello et al., 2024). Models were evaluated on three downstream datasets (Wang et al. 2019, Sakaguchi et al., 2019). Brain-informed fine-tuning yields improvements over the base model across two datasets, further supporting the generalizability of our approach to different architectures and low-parameter adaptation methods (LoRA). We note that improvements are small, likely due to the parameter-efficient adaptation (LoRA on a single layer). Unfortunately, we couldn’t perform full fine-tuning on this model due to computational constraints.\n\n| Dataset | LLaMA‑3.2‑1B | LLaMA‑3.2‑1B‑ft-en|LLaMA‑3.2‑1B‑ft-zh|\n|-|-|-|-|\n| BoolQ | 0.5790 | **0.5848** | **0.5830** |\n| COPA | 0.5534 | 0.5534 | 0.5534 |\n| Winogrande | 0.4945 | **0.4961** | **0.4961** |\n| **Average** | 0.5423 | **0.5447** | **0.5441** |\n\n## Q2\nIn response to your suggestion, we implemented an alternative version of our pipeline that mirrors the preprocessing-based approach used in prior work; stimulus-TR alignment is performed prior to fine-tuning. Specifically, we constructed stimulus-TR pairs similar to Schwartz et al. (2019). We then fine-tuned the language model using this pre-aligned input and bilingual brain data from 1 participant. The mean encoding performance is 0.13 for the preprocessing-based method, compared to 0.15 for the vanilla model and 0.16 for our fine-tuned model. The table below presents downstream task performance (on the GLUE benchmark) for this preprocessing-based method (denoted as BERT-ft-en (Ablation)). This allows direct comparison with our proposed end-to-end brain-informed fine-tuning pipeline.  We observe a substantial drop in performance for the preprocessing-based method (BERT-ft-en (Ablation)). This suggests that stimulus-TR alignment performed prior to fine-tuning procedure likely discards valuable temporal and contextual information. In contrast, our approach integrates brain alignment during fine-tuning, enabling robust downstream performance.\n\n|GLUE Task|BERT-en|BERT-ft-en (Mean±Std)|BERT-ft-en (Ablation)\n|-|-|-|-|\n|CoLA|53.38|**55.11 ± 0.75**|0.00|\n|SST-2|92.08|**92.58 ± 0.39**|50.92|\n|MRPC (Acc.)|79.41|**80.55 ± 1.01**|68.38|\n|MRPC (F1)|86.27|**86.91 ± 0.75**|81.22|\n|STS-B (Pears.)|88.06|**88.10 ± 0.07**|0.008|\n|STS-B (Spear.)| **87.65**|87.55 ± 0.18|0.028|\n|QQP (Acc.)|**90.84**|90.79 ± 0.03|75.28|\n|QQP (F1)|87.70|**87.71 ± 0.19**|62.81|\n|MNLI-m| 84.38|**84.40 ± 0.09**|41.74|\n|MNLI-mm|**84.64**|84.55 ± 0.20|41.48|\n|QNLI|91.45|**91.49 ± 0.09**|58.32|\n|RTE|67.15|**67.32 ± 0.17**|52.71|\n|WNLI|49.30|**55.59 ± 1.31**|43.66|\n\n## Q3 \nThank you for this thoughtful question. We analyzed scaling behavior on high-performing voxels using up to 6 stories (~300 TRs each). While there is a mild upward trend in encoding performance (mean r: 0.151 → 0.162), the effect is small relative to the standard errors, likely due to the limited dataset size. We expect clearer scaling trends to emerge with larger brain datasets (Antonello et al., 2023).\n\n| No. of Stories | 1   |  2   | 3   |  4   | 5   | 6   |\n|----------------|--------|--------|--------|--------|--------|--------|\n| Mean (r)    | 0.1515 | 0.1508 | 0.1500 | 0.1571 | 0.1624 | 0.1625 |\n| SE             | 0.0052 | 0.0054 | 0.0053 | 0.0102 | 0.0149 | 0.0150 |\n\n[Antonello et al., 2023] Scaling laws for language encoding models in fMRI\n\n## Q4\nThank you for this interesting question.  We observe that language-selective and semantic-selective voxels provide stronger gains in downstream task performance compared to using whole-brain voxels, suggesting that brain-informed fine-tuning primarily enhances semantic representations. Please see Appendix Table 5 for details.   Our current work focused on improving encoding and task performance of bilingual brain-informed fine-tuning. However, we agree that interpreting which specific aspects of language (semantics, syntax, etc.) are altered is an important next step. We plan to explore attribution-based analyses and causal graph methods (Lindsey et al. 2025) in future work to better understand changes introduced with fine-tuning.\n\n[Lindsey et al. 2025] On the Biology of a Large Language Model\n\nWith the extra page allowance, we will move key results from the appendix into the main text to improve clarity as suggested."}, {"review_id": "bXgQ9Odbpp", "reviewer": "Reviewer_ZxVC", "summary": "This paper presents a method for fine-tuning language models using bilingual brain data to enhance their semantic understanding in multilingual contexts. The core idea leverages neural signals to adapt language models, aligning their representations more closely with human brain activity during language processing. This approach demonstrates potential for improving cross-lingual generalization by grounding model training in biologically inspired signals.", "questions": "1. Is it because the performance is not satisfactory that the results of fine-tuning other models are placed in the appendix? I suggest placing the results of multiple models in the main text, which could reveal whether the approach generalizes across architectures or is specific to large-scale pretrained models.\n\n2. The flat map visualizations in Figure 2 appear cluttered. What specific insights do they intend to convey? Clarifying the key takeaways (e.g., spatial patterns of cross-lingual alignment) would strengthen interpretability.\n\n3. Could the authors explicitly articulate the practical contributions of this work? Given the modest performance gains (Table 1), what justifies its potential for acceptance beyond the dataset release?", "limitations": "See Weaknesses.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. The paper is clearly written, with straightforward experimental setups, and Table 1 presents intuitive results.\n\n2. The release of a benchmark dataset constitutes a tangible contribution to the community.\n\nWeaknesses:\n\n1. The proposed method—naïve full-model fine-tuning of a pretrained language model—lacks technical novelty, significantly limiting its conceptual contribution.\n\n2. As shown in Table 1, fMRI-guided fine-tuning yields marginal and inconsistent performance gains across tasks (particularly in subtables (a) and (c)). These results undermine the framework’s robustness and call into question its practical utility.", "was_revised": true, "final_justification": "Although I still don't fully agree with what the author said about the points of novelty, I have decided to raise my score to 3.", "rebuttal": "## Clarification of fine-tuning methodology\nWe thank the reviewer for their valuable comment. We would like to highlight that the novelty of our work lies in the design of the brain-informed fine-tuning pipeline. Our pipeline is a general, end-to-end framework that addresses key limitations of prior brain-to-model alignment methods and enables direct integration of brain data into language model fine-tuning.\n\nMost previous approaches aligned stimulus features to brain responses via preprocessing steps based on rigid assumptions; for instance, using fixed word presentation rates or downsampling (e.g., Schwartz et al., 2019). These designs impose strong constraints. In contrast, our pipeline allows for:\n- variable-length input (since word timing per repetition time (TR) can vary),\n- more flexible and generalizable alignment applicable across modalities and tasks as we pass raw text as input \n- dynamic stimulus-repetition time (TR) alignment as part of the learning process, not a static preprocessing step\n\nTo illustrate this advantage, we implemented the preprocessing-based alignment (as in prior work) and fine-tuned a new model on bilingual data from one participant. The mean encoding performance is 0.13 for the preprocessing-based method, compared to 0.16 for our brain-informed fine-tuned model. We hope this highlights the practical and conceptual contributions of our approach. We will add this reasoning in the camera-ready to ensure clarity.\n\n## Clarification on gains in Table 1\nThank you for raising this point. The initial Table 1 reported downstream task performance for a single participant, which may have appeared inconsistent. We have revised Table 1 to show average performance across all six bilingual participants, with standard deviation. The overall trends remain consistent with those initially reported, helping strengthen the robustness of the findings. Due to space constraints, please see the *updated table 1* included in our response to reviewer hVbZ. This aggregated analysis strengthens the robustness and practical utility of our approach, showing consistent benefits across participants and models. Please note that models were fine-tuned separately for each participant and language.\n\n## Q1\nThank you for the question. We placed the results in the appendix due to space constraints, not due to lack of performance. In fact, all three models (mBERT, XLM-R, and XGLM) show improvements (see below) following brain-informed fine-tuning. This table shows that fine-tuning improves performance on most tasks: mBERT (7/9 tasks), XLM-R (9/9), and XGLM (6/9), with especially large gains for XLM-R (e.g., RTE: 54.15 → 62.71, MRPC: 69.61 → 77.94).\n\n**Updated Supplementary Table 2. a) Fine-tuning and Evaluation in the Same Language**\n| **GLUE Task** | **mBERT** | **mBERT‑ft‑en** | **XLM‑R‑en** | **XLM‑R‑ft‑en** | **XGLM‑en** | **XGLM‑ft‑en** |\n|---|---|---|---|---|---|---|\n| CoLA            | **42.68** | 40.05 | 46.30 | **50.20** | 40.11 | **41.22** |\n| SST‑2           | 89.68 | **90.14** | 92.16 | **92.55** | **90.94** | 90.48 |\n| MRPC (Acc.)     | 84.80 | **84.80** | 69.61 | **77.94** | 71.32 | **71.81** |\n| MRPC (F1)       | 88.56 | **89.01** | 81.55 | **84.10** | 81.52 | **82.39** |\n| STS‑B (Pearson) | 88.06 | **88.42** | 84.96 | **85.18** | 81.43 | **82.36** |\n| STS‑B (Spearman)| 87.76 | **88.22** | 84.84 | **85.10** | 81.47 | **82.41** |\n| QQP (Acc.)      | 90.22 | **90.47** | 90.28 | **90.61** | 88.93 | **90.11** |\n| QQP (F1)        | 86.70 | **87.07** | 87.05 | **87.45** | 83.11 | **85.33** |\n| MNLI‑m          | 82.09 | **82.66** | 84.16 | **84.46** | 76.62 | **78.85** |\n| MNLI‑mm         | 82.38 | **82.97** | 84.22 | **84.40** | 78.80 | **79.01** |\n| QNLI            | 91.14 | **91.18** | 90.17 | **90.24** | **88.94** | 88.44 |\n| RTE             | **67.15** | 65.70 | 54.15 | **62.71** | **53.79** | 53.43 |\n| WNLI            | 53.52 | **56.34** | 43.66 | **56.34** | 50.34 | **52.41** |\n\n\n## Q2\nThank you for the helpful feedback. We acknowledge that Figure 2 has a lot of information. The main takeaway from Figure 2 is that across both languages and model types (monolingual and multilingual), bilingual brain-informed fine-tuning consistently yields better encoding performance than the vanilla models in several brain regions. \n\nWhile spatial patterns vary due to inter-individual differences (see Appendix Figure 5), we observe improvements in encoding performance in several high-level language areas.. Given this variability, the improvements in downstream task performance from fine-tuning are still robust. We will provide separate flatmaps for each fine-tuning condition in the supplementary material to improve clarity.\n\n## Q3\nThank you for this important question. Our work demonstrates that brain-informed fine-tuning can reliably enhance multilingual language model performance, even with limited data. Please note that each model was fine-tuned using 2.5 hours of naturalistic brain data per participant. We adopt a small-N design (Smith & Little, 2018), conducting full replication across participants by fine-tuning and evaluating models individually. While the performance gains may appear modest, they are robust across participants. Our results contribute to work on aligning brain and language model representations. To our knowledge, this is the first study to leverage bilingual brain data for fine-tuning, showing gains in multilingual understanding. This work offers a step toward bridging neuroscience and NLP to interpret and model multilingual semantic representations."}]

Author rebuttal:

Meta-review: Overview:

This paper studies the impact of fMRI data during story reading for finetuning LLMs, showing that finetuning on bilingual brain data finetuning on this data improves voxelweise encoding modeling. The primary experiments are conducted with English adn Chinese.  Results show consistent improvements on standard NLP tasks including NLI, CLUE, and MCQ tasks. 


Strengths:

-  The paper is well written taking time to illustrate concepts from cognitive science that are helpful for the reader.  The overall idea is presented clearly, and the experiments are all thorough.  

- The paper shows clear improvements on downstream tasks and also does studies with human participants.  In addition, while the authors have not said they will release the dataset, the paper makes contributions with a new dataset that can be used to study LLM performance with fMRI data.  

Weakness:

- This paper is primarily a study on feasibility and potential improvement of finetuning LLMs on fMRI dataset.  Many reviewers emphasized showing improvements with fMRI data as the main contribution of the work as well as the creation of the dataset. However, the paper has several limitations in its study including limited numbers of participants and experiments only on small models.  It is difficult to know how these results would extend to larger models and for other downstream tasks in addition to the standard NLP benchmarks considered.  

- The paper is missing some comparisons with prior work for which it claims an improved fine-tuning pipeline with less preprocessing, some additional analysis, and evaluations with more participants.  These were added in the rebuttal and should appear in the final version of the paper.

- The paper could benefit from editing the final set of Tables in pages 8, 9 as they are all aggregated and it is difficult to read through these last pages.

Main Reasons to Accept or Reject:

- Accept: An interesting setup that shows improvements for bilingual scenarios and potential release of a dataset to aid in studying this.

- Reject: Lack of comparison on modern models. It is hard to say that this will be useful for larger decoder models that are more conventionally studied.

Rebuttal Period:
All reviewers engaged in the rebuttal and many of the reviewers increased scores.  There is still some outstanding concern about the novelty from one reviewer.  Overall, the main concern is about whether the approach will work on more current architectures, and whether the data will be released (unclear from the checklist).  Without the dataset, I don't think this work will be very reproducible.

Final Recommendations:

The paper conducts an interesting study into the finetuning of LLMs on fMRI data.  The paper is overall strong, but can still benefit from incorporating reviewer feedback around clarity of contributions, and discussing the limitations.
Decision: Accept (poster)