When Language Models Lose Their Mind: The Consequences of Brain Misalignment
Authors: Gabriele Merlin
Mariya Toneva
Venue: ICLR 2026 Poster (2026)
Keywords: language models
brain alignment
brain misalignment
linguistic competence
neuroscience
fMRI
Primary area: foundation or frontier models, including LLMs

Abstract: While brain-aligned large language models (LLMs) have garnered attention for their potential as cognitive models and for potential for enhanced safety and trustworthiness in AI, the role of this brain alignment for linguistic competence remains uncertain. In this work, we investigate the functional implications of brain alignment by introducing brain-misaligned models--LLMs intentionally trained to predict brain activity poorly while maintaining high language modeling performance. We evaluate these models on over 200 downstream tasks encompassing diverse linguistic domains, including semantics, syntax, discourse, reasoning, and morphology. By comparing brain-misaligned models with well-matched brain-aligned counterparts, we isolate the specific impact of brain alignment on language understanding. Our experiments reveal that brain misalignment substantially impairs downstream performance, highlighting the critical role of brain alignment in achieving robust linguistic competence. These findings underscore the importance of brain alignment in LLMs and offer novel insights into the relationship between neural representations and linguistic processing.

Published as a conference paper at ICLR 2026
WHEN LANGUAGE MODELS LOSE THEIR MIND: THE CONSEQUENCES OF BRAIN MISALIGNMENT
Gabriele Merlin MPI-SWS Saarbrücken, Germany gmerl pmerlinempiswwe org
Mariya Toneva MPI-SWS
Saarbrücken, Germany mt mtonevaenpi-sws. org
ABSTRACT
While brain-aligned large language models (LLMs) have garnered attention for their potential as cognitive models and for potential for enhanced safety and trust- worthiness in AI, the role of this brain alignment for linguistic competence re- mains uncertain. In this work, we investigate the functional implications of brain alignment by introducing orain-misaligned models-LLMs intentionally trained to predict brain activity poorly while maintaining high language modeling perfor- mance. We evaluate these models on over 200 downstream tasks encompassing diverse linguistic domains, including semantics, syntax, discourse, reasoning, and morphology. By comparing brain-misaligned models with well-matched brain- aligned counterparts, isolate the specific impact of brain alignment on language understanding, Our experiments reveal that brain misalignment substantially im- pairs downstream performance, highlighting the critical role of brain alignment in achieving robust linguistic competence. These findings underscore the importance of brain alignment in LLMs and offer novel insights into the relationship between neural representations and linguistic processing.
INTRODUCTION
growing body of work studies the intriguing parallels between pretrained large language models (LLMMs) and the human brain, demonstrating substantial degree of alignment between brain activity patterns and I.I.M activations when humans and L.MSs are presented with the same linguistic input
Toneva & Wehbe, 2019 Caucheteux & King 2020 Schrimpf et al. 2021; Goldstein et al., 2022
Aw & Toneva 2023 Merlin Toneva 2024 Karamolegkou et al. 2023). This existing LLM
alignment has excited both cognitive scientists and Al researchers. From cognitive perspective, brain-aligned LLMS can serve as model organisms for studying natural language processing in the human brain, offering insights into mechanisms that may underlie human-like linguistic behavior and representation [Toneva] (02T). From an Al perspective, researchers posit that brain-aligned L.MMs may be safer and more trustworthy (Mineault et 2024). Relatedly, recent study demon- strated the first substantial downstream benefits of improving brain alignment of speech language model. by showing that brain-tuning tuning model significantly improves its performance on downstream semantic tasks (Moussa et al. 2025 Vattikonda et al. 2025).
Despite this promise of brain-LLM alignment, its necessity for model performance remains an open question. It is unclear whether alignment with the human brain is inherently required for LLMs to perform well on linguistic tasks, or whether the relationship between brain alignment and model behavior is more nuanced. To address this gap, it is essential to understand not only the presence of alignment but also functional implications.
In this work, we take direct approach to investigate the effect of brain alignment on LLM perfor- mance. We introduce brain-misaligned models-languaae models specifically trained to predict brain activity poorly while maintaining robust language modeling performance the same linguistic in- puts. We evaluate these models across more than 200 downstream tasks spanning a broad spectrum of linguistic capabilities, including semantics, syntax, discourse, reasoning, and morphology. By comparing brain- miunn-mislligeed models with well-matched models that differ primarily in their ability predict brain activity rather than their language modeling proficiency, isolate the impact of

---

Published as a conference paper at ICLR 2026
LM Head
Pretrained Lenguage
Language Modeling Loss
X Reversal Layer
"SS Brin Mappinng Head
Pearson Correlation Loss
"Harry had never believed..."
[MRI Response
Figure 1: A schematic of the proposed approach. Our method is based fine-tuning pretrained language model with two simultaneous objectives: maintaining its language modeling ability while reducing its alignment with brain recordings. Language modeling performance is preserved by continuing training on fine-tuning dataset using the standard language modeling objective. Brain alignment is reduced by introducing second prediction head and gradient reversal layer, which encourages the model to produce representations that are uninformative about the corresponding brain activity.
brain alignment on downstream linguistic performance. Our results reveal that brain ceeii-mmsaligmment significantly impairs the ability of LLMs to perform linguistic tasks. These findings suggest that alignment with the human brain is crucial for LLMs to achieve strong linguistic performance, shed- ding light on the functional relevance of brain alignment in modern language models.
Our main contributions can be summarized as follows:
We develop brain-misaligned models that allow us to investigate the effect of brain align- ment on the linguistic competence of language models.
2. We evaluate the effect of brain misalignment on comprehensive set of linguistic tasks, comprising more than 200 datasets. These tasks are designed to assess various linguistic subfields (syntax, semantics, discourse, reasoning, and morphology) and linguistic phe- nomena (e.g., part speech, protoroles, coreference resolution).
3. Via comparisons with well-matched controls, show that brain misalignment signifi- cantly decreases linguistic competence. This suggests that brain alignment necessary to maintain linguistic competence in language models.
4. We find that the competence drop is especially pronounced in semantic and syntactic tasks, demonstrating the importance of brain alignment for language models.
5. To further validate our findings, we also finetune model using brain recordings, showing that the model improves in every linguistic subfield with respect to other finetuned models, and also better than pretrained models, in particular for semantics and syntax tasks.
RELATED WORKS
A growing body of research investigates the alignment between pretrained language models and hu-
man brain activity during language comprehension (Wehbe et al. 2014b) Jain & Huth 2018 Toneva
& Wehbe 2019 Abdou al. [02T Schrimpl al. 2021 Hosseini et al. 2024h. Other studies have
focused on understanding the factors that drive this alignment, identifying model characteristics or
representational properties that correlate with neural responses (Goldstein et al. 2022 Toneva al.
2022a Oota et al. 2024a b) Caucheteux et al. 2021 Reddy & Wehbe 2021] Toneva et al. 2022b Kauf cf al. 2023] Gauthier & Levy 2019) Aw & Toneva 2023 Merlin Toneva 2024). Addition
ally. previous work have started to use brain data for finetuning language models (Schwartz et al.
2019) showing that is possible to improve downstream performance pretrained language model
(Negi et al. 2025). Our work extends these findings by investigating whether this alignment is not
only observed but also functionally relevant for language processing, specifically, whether brain alignment is necessary for maintaining linguistic competence in language models.
2

---

Published as a conference paper at ICLR 2026
In fact, substantial body of work has focused evaluating the linguistic competencies of lan- guage models. These studies aim to systematically assess the extent to which models capture vari- ous linguistic phenomena, including syntax, semantics, morphology, and discourse-level reasoning (Amouyal et al., 2024, Blevins et al., 2023). Benchmarks such as BLiMP (Warstadt et al., 2020 GLUE (Wang et al. 2018), SuperGI.UR (Wang al., 20T9), and more recently Holmes (Waldss ett al. 2024) have been used to evaluate models' understanding of language. Our study contributes
this line of research by examining how these linguistic competencies are affected when the align- ment between language model representations and brain activity is manipulated.
Additionally, a growing line of work Causal NLP aims to uncover causal relationships between model components, training signals, or representations and downstream performance (Feder et al. 2021 2022 Liu et al., 2025 Ortu et al. 2024). These studies design interventions or counterfactual setups to test whether certain features are causally implicated in model predictions or behaviors. Our approach is aligned with those works. We intervene on brain alignment, training models preserve or disrupt alignment, and estimate its causal role in supporting linguistic competence.
METHODOLOGY
3.1
PRETRAINED MODELS
We use BERT-based [Dvlin et al. 2019 GPT2-based (Radford ct al. 2019 and Llama-based (Lu
et al. 2024 language models. In particular, focus the bert www.bbaaeeeeeee cased,
small and meta- 11 ama /Ima/llann ama-3333.3. 2-1B provided by Hugging Face (Wolf et al. 2020). BERT, GPT2 and Llama have achieved strong performance various NLP tasks, such question answering and sentence classification. Moreover, they have been extensively studied in prior work on brain
alignment (Tonvaa & Wehbe 2019 Caucheteux al. 2021 Oota al. 2024b).
3.2 FMRI DATA
We use two publicly available fMRI datasets to measure the brain alignment of language model rep- resentations. The data included in the first dataset, provided by Wehbe et al. 2014a were collected from eight participants as they read Chapter 9 of Harry Potter and the Sorcerer's 's Stone (Rowling al. 1998) word by word. The chapter was divided into four runs of similar length, each separated by short break. Each word was presented for 0.5 seconds, and one IMRI image (TR) was ac- quired every 2 seconds, resulting in 1211 brain images per participant. The IMRI data in the second dataset, made publicly available by Deniz et al. 2019 consist of recordings from six participants who read and listened the same 1I stories from The Moth Radio Hour. For each modality, the dataset includes 4028 fMRI images. During reading, each word was presented for exactly the same duration as in the audio recording. In our analysis, we used only the reading data, These datasets are among the largest publicly available collections in terms of the amount of data per participant, which is crucial for obtaining accurate estimates of brain alignment.
3.3 CONTROLLING BRAIN ALIGNMENT
To investigate the effect of brain alignment of a language model on its downstream linguistic com- petence, we develop three models: the Brain Misaligned model, the Brain Preserving model and the Brain Tuned model. The Brain Misaligned model is trained to reduce alignment with brain record- ings, while the Brain Preserving model serves as comparison baseline that preserves brain align- ment while controlling for possible confounding factors. We also designed Brain Tuned model that is trained to improve alignment with brain recordings. This model serves to further validate our analysis.
3.3.1 BRAIN MISALIGNED MODEL
To evaluate the influence of brain-related information, which is an abstract concept for which no clear counterfactual input exists, must develop methods that allow us remove such informa- tion directly from the language model representations. In this study, we address this challenge by designing intervention to language models that aims remove brain-related information from
3

---

Published as a conference paper at ICLR 2026
their representations, without the need to generate counterfactual inputs. This enables to investi- gate the necessity of brain alignment for natural language processing abilities.
Our approach is based on adversarial fine-tuning (Ganin al. 2016) of language models, using a prediction head (brain mapping head in Figure ! and gradient reversal layer to remove the targeted capacity, i.e. brain prediction, while simultaneously fine-tuning second head to preserve the language modeling performance.
The model is finetuned using the stimuli from the Harry Potter fMRI dataset (Wehbe al. 2014 4a)
or from the Moth Radio Hour fMRI dataset for the language modeling loss, and the corresponding fMRI recordings for the loss of the brain mapping head. For training, select only voxels with an estimated noise ceiling > 0.05 (see Appendix C for details) belonging to regions of the brain known
process language (Fedorenko al., 2010 Fedorenko & Thompson-Schin 2014 Binder et al. 2009 Oota et al. 2024a) and used by previous works to investigate brain alignment of language models. Additional details on the prediction of brain recordings are reported Appendix The total loss is defined as:
f = Wim Lim + Liba
where Cim is the language modeling loss, Cso the brain-alignmen loss, and W/m and who are weighting factors to balance the two objectives. The language modeling loss Lim corresponds to the standard cross-entropy loss used during language model pretraining, while the brain-alignment loss Lba is defined as the mean negative squared Pearson correlation between the predicted voxels in each batch and the ground truth voxel values. W/m is fixed 0.1, value chosen based on the relative magnitude of the losses prior to fine-tuning (see Section 3.3.4 for details).
3.3.2 BRAIN PRESERVING MODEL
Similarly, we designed control condition to account for potential confounding factors and serve as comparison for the Brain Misaligned model. We finetune this model using the same procedure as for the Brain Misaligned model, but during training we permute the order of the fMRI images to disrupt the correspondence between stimuli and brain activity.
By using permuted fMRI images, our method also accounts for the effects of the adversarial removal itself, which can influence the model's representations. This controls for potential confounders such as the effect of fine-tuning on language modeling and the effect of adversarial fine-tuning. The only difference between conditions remains the correspondence between stimuli and fMRI images.
3.3.3 BRAIN TUNED MODEL
To complement our analysis, we designed Brain Tuned model. This model was finetuned using the same procedure as the Brain Misaligned model (i.e. a language modeling head and a brain mapping head), but we removed the gradient reversal layer and use as loss function Loa the mean negative Pearson correlation. This procedure actively encourages the model to increase its alignment with brain recordings while maintaining language modeling performance. This model serves as a validation tool to test the complementary hypothesis: if decreasing alignment impairs competence, does increasing it lead to performance gains?
3.3.4 MODEL SELECTION AND TRAINING
To train the models, we use training samples consisting of sequences of words corresponding to 5 TRs. The stimulus text is divided into four consecutive sections to enable cross-validation.
For training of BERT-based, GPT2-based and Llama-based Misaligned, Brain Preserving and Brain Tuned models we train applying LoRA (Hu et al. 2022) to the parameters. We train for epochs with batch size of 16, and Adam optimizer. The language modeling loss weight Wim 0.1 and Wba = 10.
Conditions for a successful comparison between models. The comparison is considered suc- cessful when the Brain Misaligned and Brain Preserving models achieve similar performance on the language modeling objective (tested using Wilcoxon signed-rand -ank test, < 0.05), while the Brain Misaligned model shows significantly lower ability align with brain recordings.
4

---

Published as a conference paper at ICLR 2026
3.4 EVALUATION
To evaluate the models, we use three types of tasks: language modeling, brain alignment, and down- stream linguistic tasks. Both language modeling and brain alignment are evaluated using the same text, which corresponds the fMRI stimulus, and is held-out during training. We assess these two tasks using overlapping sequences of words belonging to 5 TRs, following the approach of previous work (Merlin & Toneva 2024
Language modeling. For language modeling we follow the best practice for evaluation BERT- based, GPT2-based and Llama-based models. For each test example, we measure the average cross entropy across the randomly masked tokens (15% of total number of tokens, see Devlin et al. (2018) for details) for BERT- models, for GPT2-based and Llama-based models the cross entropy over all tokens (see Radford et al. (2019) for details).
Brain alignment. We measure the brain alignment between BERT, GPT2 and Llama representa- tions and fMRI recordings using a linear prediction head on top of the last transformer block. This prediction head is trained to output brain activity values from the model's representations and is widely used in previous work to assess how well language models can predict brain signals dJain & Huth 2018 Toneva & Wehbe 2019 SSchrimpf et al. 202T). We train this linear function, regu- larized with ridge penalty, using cross cross-validation and evaluate its performance on held-out data. The ridge parameter is selected via nested cross-validation. Consequently, for each participant, we train one model for each held-out run (see Section 3.2), then aggregate the predictions to compute brain alignment. Further details on the prediction head are provided Appendix B Brain alignment is quantified using Pearson correlation, computed between the predictions on held-out data and the corresponding ground truth values. Specifically, for a model q and voxel Vi with corresponding held-out fMRI values yj, brain alignment is computed as:
brain al i gineent (q, vj) = corr(y), My),
where ûú ((X) q.i X is the input text sample to model g, and Wa. are the learned prediction weights for voxel Vg-
Linguistic competence. To investigate the linguistic competence of language models, we use more than 200 datasets, designed to evaluate linguistic competence in language models via classifier- based probing (Waldis et al. 2024p. The benchmark covers datasets spanning various linguistic phenomena and subfields, including syntax, morphology, semantics, reasoning, and discourse, ex- amples of tasks reported in Appendix Table | Details about the benchmark and the included datasets are provided in Appendix A For each task, each model is evaluated using 6 seeds, which influence the probe initialization and the ordering of data during training and evaluation.
To determine whether one model outperforms the other, we not only compare the average evaluation metric (see Waldis et al. (2024) for details), but also assess whether the difference is statistically significant using a two-sample t-test. We assign a "win" to a model only for datasets where the difference reaches statistical significance. For each dataset and model pair, we thus obtain binary "win" matrix indicating whether one model significantly outperforms the other (1) or not (0). Since each subject has pair of models corresponding to different held-out runs during training, erage the resulting win matrices across runs, yielding win score for each participant, dataset, and model. The win score quantifies how consistently one model outperforms the other across different held-out runs.
av-
4 RESULTS
4.1
EFFECTS ON BRAIN ALIGNMENT
Figure A-D shows brain alignment of the BERT-based Brain Misaligned and Brain Preserving models on the Harry Potter dataset, as well as a contrast between the two, for a representative participant. Specifically, Figures 2 and 2 show the Pearson correlation between the predicted voxel values and the ground truth for the Brain Preserving model and the Brain Misaligned model, respectively. Figure C shows the contrast between the two models, i.e., the difference in Pear-
5

---

Published as a conference paper at ICLR 2026
A) Brain preserving
B) Brain Misaligned
0.3
0.3
Pearson Correlation
Pearson Correlation
C) Brain Misalignment Effeet
D) Language ROIs
AG
IFG
IFGorb
DMPEC
-0.1
0.1
Difference in Pearson Correlation
Figure 2: Brain alignment of the BERT-based Brain Preserving (A) and Brain Misaligned (B) mod- els for one participant the Harry Potter dataset (see Appendix D for all participants), and the difference between the two (C). The Brain Misaligned model exhibits substantially weaker align- ment, particularly in language regions (C, D).
son correlation for each voxel. Results for the remaining participants, other language models and datasets are consistent and reported in Appendix D
Brain Preserving Model. In Figure DA, we observe that the Brain Preserving model aligns well with brain activity across the whole brain, and in particular within language-related regions (visual- ized in Figure
2 D). as identified by previous work (Fedorenko et al., Fedorenko & Thompson-
Schill 2014 Binder al., 2009 We quantify the alignment of the Brain Preserving model in
Appendix D Since these models are explicitly finetuned to preserve brain alignment, this result is
consistent with prior studies showing that language models exhibit alignment with fMRI signals
Toneva & Wehbe 2019 Schrimpl et al. 2021; Goldstein et al. 2022 Oota et al. 2024b
Merlin &
Toneva 2024).
Brain Misaligned Model. In Figure 3 we observe that the Brain Misaligned model does not align well with brain activity across the whole brain. The Pearson correlation values are particularly low in several brain areas. We quantify the alignment of brain misaligned model in Appendix D
To show the effect of brain misalignment, in Figure 2 we show the contrast between the Pearson correlation values of the Brain Preserving model and the Brain Misaligned model. As expected, the difference in Pearson correlation is especially high in language-rede uunuagg--rlated related regions (visualized in Figure PD). This confirms that our approach is effective at removing brain-relevan information in particular in language-related areas.

---

Published as a conference paper at ICLR 2026
We assess whether the average brain correlation (computed across voxels in language regions with an estimated noise ceiling > 0.05, see Appendix O significantly differs between the two models using t-test. For example, for BERT trained on Harry Potter dataset we find that for six participants, there is a significant drop in brain alignment, while no significant difference language modeling ability is observed between the two models. Only the models corresponding these participants included in the comparison of linguistic competence.
4.2 EFFECTS ON LINGUISTIC COMPETENCE
Figures 3 B and show the performance on linguistic competence averaged across models and dataset combinations (BERT-Harry, BERT-Moth, GPT2-Harry, GPT2-Moth, Llama-Harry, Llama- Moth). Specifically, Figure E illustrates the overall effect on linguistic competence, considering all tasks. Figure 3B presents the results by linguistic subfields, while Figure ! focuses on specific linguistic phenomena. Specific results for each combinations of models and dataset are reported in Appendix G
H
Effects on the Overall Linguistic Competence. Figure BA shows the average win rate in the linguistic competence benchmark for the Brain Misaligned models and the Brain Preserving models. We observe a significant difference between the two conditions. These results indicate that removing brain alignment leads to lower performance on downstream linguistic tasks, suggesting that brain aligmment is necessary to preserve linguistic competence. Statistical tests on each individual model and dataset combinations reveal marked difference in performance between the Brain Misaligned and Brain Preserving on the overall linguistic competence. For the BERT-based and Llama- -ased models the difference is significant, although for GPT2-based models on the Harry Potter dataset the difference does not reach conventional statistical significance (p-value = 0.055), the trend mirrors the effect observed in the BERT-based models. For the GPT2-based models on the Moth Radio Hour dataset, results are not consistent due to the weaker effect of brain removal.
Effects across Linguistic Subfields. We further investigate this effect by analyzing performance across different linguistic subfields: syntax, semantics, discourse, reasoning, and morphology. As shown in Figure 3 the Brain Misaligned model consistently underperforms the Brain Preserving model in discourse, morphology, reasoning, semanties, and syntax tasks. This suggests that brain alignment is particularly important for supporting linguistic competence. Statistical tests on individ- ual model and dataset combinations reveal significant differences in the majority of model-dataset combinations, even though the averaged results are not statistically significant. In particular, BERT trained on Harry Potter reveals statistical significance across all linguistic subfields. BERT trained on Moth Radio Hour and Llama trained on Harry Potter on the semantics and syntax subfields, while Llama trained on Moth Radio Hour on the syntax and morphology subfields.
Effeets across Linguistic Phenomena. To gain finer-grained insights, we analyzed results based on specific linguistic phenomena, focusing on those represented by more than five datasets. As shown in Figure we found that for the majority of tasks the Brain Preserving models are better than the Brain Misaligned models, providing further evidence that brain alignment is crucial for these phenomena. Examples for these linguistic phenomena can be found in Table 2
4.3 FURTHER VALIDATION VIA BRAIN-TUNING
We conducted complementary analysis by introducing Brain Tuned model, in which, contrary the Brain Misaligned model, the brain alignment capabilities were intentionally increased during training.
We then compared this new model with the Brain Preserving model. The analysis reveals that in every experimental setting, the Brain Tuned model consistently outperforms the Brain Preserving model with a statistically significant difference. This result suggests that increasing brain alignment translates into a general improvement in linguistic competence. Averaged results across model and dataset combinations are shown in Figure5 Figure 5 shows the averaged results across linguistic subfield, showing statistically significant results for the syntax and semantic subfield, and Figure 6 shows the differences across linguistic phenomena, showing statistically significant results for two tasks. Detailed results for each of these combinations are available in the Appendix D

---

Published as a conference paper at ICLR 2026
A) All tasks
B) Tasks split linguistic subfield
0.30
Brain
0.25
Average win rate
0.20
Auarage ein
0.20
0.15
0.15
0.10
0.05
Brain Misaligned
Brain Preserving
0.00
Figure 3: Average win and standard error across models and dataset combinations of the Brain Misaligned and Brain Preserving models across tasks (Left) and across different linguistic subfields (Right). The average win rate indicates how often each model outperforms its counterpart across model and dataset combinations. The Brain Preserving model significantly outperforms the Brain Misaligned model (p < 0.05, Wilcoxon signed-rank test) (Left). This result suggests that removing brain alignment impairs linguistic competence. The Brain Preserving model shows higher win rate in all the linguistic subfield, in particular for semantics and syntax (Right), even the differences not statistically significant (assessed using Wilcoxon signed-rank test with Nolm-Bonferroni correction), because of unique differences across model-dataset combinations.
0.40
Brein Movllgeed
0.35
Brain Preserving
0.30
Average win rate
0.25
0.20
0.15
0.10
0.05
0.00
argument structure
determinar roan agrenment
binding
event structure
fille gap
genericity
island offerts
regative polarit item Rreesing
regatton
protoroles
rethorical structure theory
quantiflers
Figure 4: Average win rate with standard errоr across model and dataset combinations, across var- ious linguistic phenomena for the Brain Misaligned and Brain Preserving models. Each bar repre- sents the average win rate for specific linguistic phenomenon, with error bars indicating standard error. Brain Preserving models tend to outperform Brain Misaligned models the majority of tasks. Some concrete examples of the linguistic tasks provided in the Table 2
We also compared the Brain Tuned model directly with the original Pretrained model (i.e., the base model before any alignment intervention). While the Brain Tuned model showed advantage the majority of experimental settings, the Pretrained model maintained stronger performance in some settings. Results are reported in the Appendix
DISCUSSION
To investigate the importance of brain alignment in language models, designed two models: the Brain Misaligned model. which is intended to remove brain alignment while preserving language modeling capabilities, and the Brain Preserving model, which accounts for potential confounders.
8

---

Published as a conference paper at ICLR 2026
A) All tasks
B) Tasks split linguistic subfield
0.6
Tueed
Average win rate
0.4
Aparage ein
0.3
0.2
0.1
0.1
Brain Preserving
Brain Tuned
0.0 discourse
semanticcs
Figure 5: Average win rate and standard error across models and dataset combinations of the Brain Preserving and Brain Tuned models across tasks (Left) and across different linguistic subfields (Right). The Brain Tuned model significantly outperforms the Brain Preserving model (p 0.05, Wilcoxon signed-rank test) (Left). This result suggests that improving the brain alignment lead to performance gains in linguistic competence. The Brain Tuned model shows a higher win rate in the discourse, morphology, reasoning, semantics and syntax subfield (Right) and significantly higher in semantics and syntax (p < 0.05, Wilcoxon signed-rank test with Golm-Bonferroni correction), suggesting that improving brain alignment affects semantics and syntax tasks.
1.0
Brain Preserving
Brain Tured
Average win rate
0.6
0.0
argument structure
determiner rosn agreement
binding
event structure
filler gup
generictty
Bland offerts
negattiv polarity Rta Beensing
regaton
protorolas
rethorical structure theory
quantiffers
Figure 6: Average win rate with standard error across model and dataset combinations, across var- ious linguistic phenomena for the Brain Tuned and Brain Preserving models. Each bar represents the average win rate for specific linguistic phenomenon, with error bars indicating standard er- ror. Brain Tuned models tend to outperform Brain Preserving models in the majority of tasks, with statistically significant difference (p 0.05, Wilcoxon signed-rank test with correction) for filler gap and negat vive polarity item 1 licens ng. Some concrete examples of the linguistic tasks provided in the Table 2
We showed that the Brain Misaligned model has weak alignment with brain recordings, while the Brain Preserving model exhibits stronger alignment, particularly in language-related regions of in- terest. The contrast between the two models, across multiple experimental settings, reveals that the difference in alignment is especially pronounced in these areas.
We further evaluated the linguistic competence these models to reveal the functional importance of brain alignment. Our results demonstrate that Brain Misaligned models perform worse than Brain Preserving models on linguistic tasks, across multiple motel-dataset combinations, supporting the hypothesis that brain alignment is crucial for maintaining linguistic competence. Across multiple experimental settings the performance drop is particularly evident in tasks related to the semantic and syntactic subfield, although there are unique differences in every experimental setting.

---

Published as a conference paper at ICLR 2026
We extended this investigation by introducing Brain Tuned model, designed to increase brain alignment. The results of this intervention further strengthen our core argument. We found that the Brain Tuned model systematically outperformed the Brain Preserving model in all experimental set- tings and in particular in semantic and syntax tasks. In many model-dataset combinations the Brain Tuned model outperform the pretrained model on linguistic competence highlighting the relevance of brain-related signal for improving those competences.
These findings, across multiple model-dataset combinations, suggest that brain-aligned information plays a key role in supporting performance on linguistic tasks. is important to note that the absence of statistically significant differences for other linguistic subfields or phenomena does not imply that brain alignment is unimportant for those tasks.
Limitations. Our study has three main limitations. Firstly, the benchmark used to assess linguis- tic competence, while extensive, not exhaustive. There are many additional datasets available that could be included in future evaluations (Wang et al. 2018 2019p. Moreover, some linguistic subfields (e.g., discourse) and specific linguistic phenomena are represented by only a few datasets. As result, the observed behavior of the Brain Misaligned and Brain Preserving models may be influenced by the limited coverage and distribution of tasks in certain categories. Secondly, our results are based limited fMRI datasets. While these widely studied datasets offer extensive per-participant data, findings may still be specific to their characteristies. We designed our exper- iments using cross-validation, testing on held-out data and across multiple participants improve generalizability. However, results might differ with different types of linguistic stimuli. Expanding to more datasets, languages, or cognitive tasks would be an important next step. Thirdly, while the "Brain Misaligned" model does show a clear overall worse performance, there are differences across linguistic subfields depending on the model-dataset combination. Datasets and models can contain different types of information related to linguistic subfields. Nevertheless, our results are informa- tive in demonstrating the effectiveness our methodology and in highlighting the importance of the emergent brain alignment ability of language models.
CONCLUSION
We designed a direct approach to investigate the necessity of brain alignment in pretrained language models. Specifically, we introduced two models: the Brain Misaligned model and the Brain Preserv- ing model. When used together, they allow us to isolate and control for the effect of brain alignment on downstream linguistic competence.
We evaluated these models on more than 200 datasets spanning various linguistic subfields, includ- ing semantics, syntax, morphology, discourse, and reasoning, as well as a broad range of linguistic phenomena. Our results revealed a significant drop in linguistic competence, particularly on se- mantic and syntactic tasks, for the Brain Misaligned model, suggesting that brain alignment plays a critical role in downstream linguistic performance. This conclusion is further supported by our complementary finding that a Brain Tuned model, optimized to increase alignment, consistently outperformed the Brain Preserving model particularly in those tasks.
These findings are highly relevant to the natural language processing literature. Previous studies have explored why brain alignment emerges during pretraining, pointing to possible contributing factors and suggesting that if this alignment emerges, it may reflect shared information acquisition between artificial and biological neural networks.
Our work contributes new dimension to this discussion: we not only ask why brain alignment emerges, but also whether it is important for linguistic competence. Our results provide initial evidence that brain alignment is functionally important, motivating future research in this area.
Moreover, our methodology provides general framework for assessing the causal role of emer- gent properties, as brain alignment, in language models. Future work could apply our methodology to different models, exploring other datasets, extending the approach to assess the necessity of alignment-related capabilities across different modalities (e.g., speech, image) or neural architec- tures.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
Work by Gabriele Merlin was supported by the CS max planck graduate center.
REFERENCES
Mostafa Abdou, Ana Valeria González, Mariya Toneva, Daniel Hershcovich, and Anders Søgaard. Does injecting linguistic structure into language models lead to better alignment with brain record- ings? arXiv preprint arXiv:2101.12600 2021.
Samuel Amouyal, Aya Meltzer-Asscher, and Jonathan Berant. Large language models for psy- cholinguistic plausibility pretesting. Yvette Graham and Matthew Purver (eds.), Findings of the Association for Computational Linguistics: EACL 2024, pp. 181, St. Julian's s. Malta, March 2024. Association for Computational Linguistics. URL https///cclaaatholoooo brg/2024. fi findi naci 12/
Khai Loong Aw and Mariya Toneva. Training language models summarize narratives improves brain alignment. In The Eleventh International Conference on Learning Representations, 2023.
Jeffrey R Binder, Rutvik H Desai, William W Graves, and Lisa L Conant. Where is the semantic system? critical review and meta-analysis of 120 functional neuroimaging studies. Cerebral cortex, 19(12):2767-2766 2009.
Terra Blevins, Hila Gonen, and Luke Zettlemoyer. Prompting language models for linguistic structure. In Anna Rogers, Jordan Goyd-Grabert and Naoaki Okazaki (eds.), Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 6649-6663, Toronto, Canada, July 2023. Association for Computational Linguis- tics. doi: 11865/////22233all-lll/ao Sel-iong.38
URL. https: /aacl antho 1 lotthology.o org/2023.
Charlotte Caucheteux and Jean-Rémi King. Language processing in brains and deep neural net- works: computational convergence and its limits. BioRxiv, 2020.
Charlotte Caucheteux, Alexandre Gramfort, and Jean-Remi King. Decomposing lexical and com- positional syntax and semantics with deep language models. arXiv preprint arXiv:2103.01620, 2021.
Fatma Deniz, Anwar O Junez-Elizalde Alexander G Huth, and Jack L Gallant. The representation of semantic information across human cerebral cortex during listening versus reading is invariant to stimulus modality. Journal of Neuroscience, 39(39):722--7736, 2019.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04880. 2018.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers). pp. 4171-4186, 2019.
Amir Feder, Nadav Oved, Uri Shalit, and Roi Reichart. Causalm: Causal model explanation through counterfactual language models. Computational Linguistics, 37(2))333-386 07 2021. ISSN 0891-2017. doi: 162//6/i 00404. URL 00404
Amir Feder, Katherine A Keith, Emaad Manzoor, Reid Pryzant, Dhanya Sridhar, Zach Wood- Doughty, Jacob Eisenstein, Justin Grimmer, Roi Reichart, Margaret E Roberts, et al. Causal in- ference in natural language processing: Estimation, prediction, interpretation and beyond. Trans- actions of the Association for Computational Linguistics, 10:1 138-11888 1158, 2022.
E. Fedorenko, P.-J Hsieh, A. Cieto-Castanon S. Whitfield-Gabrine and N. Kanwisher. New method for fMRI investigations of language: Defining ROIs functionally in individual subjects. Journal of Nearophysiology, 104(2): 114(2):177-11199 2010.
11

---

Published as a conference paper at ICLR 2026
Evelina Fedorenko and Sharon L Thompson-Schhe Schill. Reworking the language network. Trends in cognitive sciences, 18(3):120-1)) 1260 2014.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario March, and Victor Lempitsky. Domain-adversarial training of neural networks. Journal of machine learning research, 17(59):1-35, 2016.
Jon Gauthier and Roger Levy. Linking artificial and human neural representations of language. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.). Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Processing and the 9th International Joint Con- ference on Natural Language Processing (EMNLP-IJCNLP), pp. 529-539 Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19- 1/D19-10000 URL fttts://aace antho ogy.org/D199110000
Ariel Goldstein, Zaid Zada, Eliav Buchnik, Mariano Schain, Amy Price, Bobbi Aubrey, Samuel A Nastase, Amir Feder, Dotan Emanuel, Alon Cohen, et al. Shared computational principles for language processing in humans and deep language models. Nature neuroscience, 25(3):36--380, 2022.
Eghbal A. Hosseini, Martin Schrimpf, Yian Zhang, Samuel Bowman, Noga Zaslavsky, and Evelina Fedorenko. Artificial Neural Network Language Models Predict Human Brain Responses to Language Even After Developmentally Realistic Amount of Training. Neurobiology of Lan- guage, 5(1):43-63, 63, 04 2024. ISSN 2641-4368. doi: 10.1162/ho URL https:
//doi. 1162/
00137
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con- ference on Learning Representations, 2022. URL htttp:://ppeneewiew http://ppnnriiiee.eeeeeeorroru
id=nZevKeeFYfg
Alexander G Huth, Wendy De Heer, Thomas Griffiths, Frédéric Theunissen, and Jack Gallant. Natural speech reveals the semantic maps that tile human cerebral cortex. Nature, 532 (7600):453 458, 2016.
Shailee Jain and Alexander Huth. Incorporating context into language encoding models for fMRL. In Advances in Neural Information Processing Systems, pp. 6628-6637, 2018.
Antonia Karamolegkou, Mostafa Abdou, and Anders Søgaard. Mapping brains with language mod- cls: A survey. In Anna Rogers, Jordan Boyd-Graben and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 9748-9762, Toronto, Canada, July 2023. Association for Computational Linguisties. doi: 1tt.:/655//11222.mmmm findingg--acl.618. URL https::/acclat httppp///cclamthooooy.ooooo/o/// http://///22.....ggnaao--a-e-000 www.dfiinnnnnin..
6187
Carina Kauf, Greta Tuckute, Roger Levy, Jacob Andreas, and Evelina Fedorenko. Lexical semantic
content, not syntactic structure, the main contributor to ann-brain similarity fmni responses in the language network. bioRxiv, 2023. doi: 10.1101/2023.00 05 05. 53964.. URL /http:: / //www. 2023. 05. 05. 539646
biorxiv. b:o.x/i..rrg/content///ooo http///ottettttte /ar httplll/l2222////
Xiaoyu Liu, Paiheng Xu, Junda Wu, Jiaxin Yuan, Yifan Yang, Yuhang Zhou, Fuxiao Liu, Tian- rui Guan, Haoliang Wang, Tong Yu, Julian McAuley, Wei Ai, and Furong Huang. Large lan- guage models and causal inference in collaboration: A comprehensive survey. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Findings of the Association for Computational Linguistics: NAACL 2025, pp. 7668-7684, Albuquerque, New Mexico, April 2025, Association for Computa- tional Linguistics. ISBN 979-8-89176-195-7, URL httph:ttt///tctt antho logy. bagy.oo. wrg.2022.5555
Findings-nan naacl 427/
Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krish- namoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. Spinquant: Llm quantiza- tion with learned rotations. arXiv preprint arXiv:2405. 0.224005.6666, /6406, 2024.
Gabriele Merlin and Mariya Toneva. Language models and brains align due to more than next-word prediction and word-level information. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen
12

---

Published as a conference paper at ICLR 2026
(eds.), Proceedings of the 2024 Conference Empirical Methods in Natural Language Process- ing. pp. 18431-18454, Miami, Florida, USA, November 2024. Association for Computational Linguisties. doi: 10. http:///10022222.2mmmmmmmm 1024. org/2024. emn 1 pa mam.mnninnn... 1024/
URL bttps://aclanthooooyyy
Patrick Mineault, Niccolò Zanichelli, Joanne Zichen Peng, Anton Arkhipov, Eli Bingham, Julian Jara-Eitinger, Emily Mackevicius, Adam Marblestone, Marcelo Mattar, Andrew Payne, et al. Neuroai for safety. arXiv preprint arXiv:2411.18522 2024.
Omer Moussa, Dietrich Klakow, and Mariya Toneva. Improving semantic understanding in speech language models via brain-tuning. ICI.R, 2025.
Anuja Negi. Subba Reddy Oota, Anwar O Nunez-Eine Elizzzde,,,,,,,, Manish Gupta, and Fatma Deniz. Brain- informed fine-tuning for improved multilingual understanding in language models. bioRxiv, pp- 2025-07, 07, 2025.
Shinji Nishimoto, An T Vu, Thomas Naselaris, Yuval Benjamini, Bin Yu, and Jack L Gallant. Re- constructing visual experiences from brain activity evoked by natural movies. Current biology, 21(19):1641-1646 2011.
Subba Reddy Oota, Emin Çelik, Fatma Deniz, and Mariya Toneva. Speech language models lack important brain-relevant semantics. ACL, 2024a.
Subba Reddy Oota, Manish Gupta, and Mariya Toneva. Joint processing of linguistic properties in brains and language models. Advances in Neural Information Processing Systems, 36, 2024b.
Francesco Ortu, Zhijing Jin, Diego Doimo, Mrinmaya Sachan, Alberto Cazzaniga, and Bernhard Schölkopf. Competition of mechanisms: Tracing how language models handle facts and coun- terfactuals. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pa- pers). pp. 8420-8436, Bangkok, Thailand, August 2024. Association for Computational Linguis- tics. doi: 1e..11665/11122..coo
URL https: //acl://aaaaeee www.aannhhoooy.rr...oo.
ac ong. 4587
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Aniketh Janardhan Reddy and Leila Wehbe. Can fmri reveal the representation of syntactic structure in the brain? Advances in Neural Information Processing Systems, 34:9843-9856, 2021.
J.K. Rowling, M. GrandPre, M. GrandPré, T. Taylor, Arthur A. Levine Books, and Scholastic Inc. Harry Potter and the Sorcerer's 's Stone. Harry Potter. A.A. Levine Books, 1998. ISBN 9780590353403. URL
http://boooooooololooo///// google de det///ookkk7711-=27777000.....
Martin Schrimpf, Idan Asher Blank, Greta Tuckute, Carina Kauf, Eghbal A Hosseini, Nancy Kan- wisher, Joshua B Tenenbaum, and Evelina Fedorenko. The neural architecture of language: In- tegrative modeling converges on predictive processing. Proceedings of the National Academy of Sciences, 118(4)) 2021.
Dan Schwartz, Mariya Toneva, and Leila Wehbe. Inducing brain-relevant bias in natural language processing models. Advances in neural information processing systems, 32, 2019.
Mariya Toneva. Bridging language in machines with language in the brain. PhD thesis, Carnegie Mellon University, 2021.
Mariya Toneva and Leila Wehbe. Interpreting and improving natural- language processing (in ma- chines) with natural wwaguage--rocessiiee (in the brain). Advances in Neural Information Process- ing Systems, 32, 2019.
Mariya Toneva, Tom M Mitchell, and Leila Wehbe. Combining computational controls with natural text reveals aspects of meaning composition. Nature Computational Science, 2(11):745-757, 2022a.
13

---

Published as a conference paper at ICLR 2026
Mariya Toneva, Jennifer Williams, Anand Bollu, Christoph Dann, and Leila Wehbe. Same cause; different effects in the brain. Causal Learning and Reasoning, 2022b.
Nishitha Vattikonda, Aditya R Vaidya, Richard J Antonello, and Alexander G Huth. Brain- wavlm: Fine-tuning speech representations with brain responses to language. arXiv preprint arXiv:2502.08866, 2025.
Andreas Waldis, Yotam Perlitz, Leshem Choshen, Yufang Hou, and Iryna Gurevych. Holmes benchmark to assess the linguistic competence of language models. Transactions of the Asso- ciation for Compulational Linguistics, 12:1616-1647, 2024. doi: 10.1162/tal.a.007155 URL
/tttps://allattholoyy.ooo taw.1111......11..88.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:
2018.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy. and Samuel Bowman. Superglue: A stickier benchmark for understanding systems. Advances in neural information processing systems, 32, 2019.
AmanpreetSingh,
language
Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fi Wang, and Samuel R Bowman. Blimp: The benchmark of linguistic minimal pairs for english. Transactions of the Association for Computational Linguistics, 8:377-392, 2020.
Leila Wehbe, Brian Murphy, Partha Talukdar, Alona Fyshe, Aaditya Ramdas, and Tom Mitchell. Simultaneously uncovering the patterns of brain regions involved in different story reading sub- processes. PloS one, 9(11)xe11257 2014a.
Leila Wehbe, Ashish Vaswani, Kevin Knight, and Tom Mitchell. Aligning context-based statistical models of language with brain activity during reading. In Proceedings of the 2014 Conference on Empirical Methods Natural Language Processing, pp.
2014b.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick Von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-art natural language processing. In Qun Liu and David Schlangen (eds.). Proceedings of the 2020 Confer- ence on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38- 45, Online, October 2020. Association for Computational Linguistics. doi: 10. 10.18653/v1/2020. emnlp-demos.6. URL https: /ttee//aclllaa anthologg.oogg2000000 emn 1-mmop-dmosss 6/
A LINGUISTIC COMPETENCE BENCHMARK
We evaluated the linguistic competence of language models using classifier-based probing more than 200 datasets collected from various sources and included in the Holmes benchmark (Waldis) al. 2024). The benchmark covers datasets spanning a wide range of linguistic phenomena and subfields, including syntax, morphology, semantics, reasoning. and discourse. A comprehensive list of linguistic phenomena and their corresponding subfields is provided Table S For the evaluation, we use the flash-ho lmes version of the benchmark, which is designed to reduce computational cost while maintaining precision in assessing language model performance (see Waldis et al.) (2024) for details). Examples of tasks associated with the linguistic phenomena and linguistic subfield used in our study are reported in Tables I and and illustrated in Figure
B BRAIN MAPPING HEAD
To predict the fMRI recordings corresponding to each TR, we use a linear function, regularized with ridge penalty, that maps model representations to fMRI space, specifically targeting voxels with an estimated noise ceiling > 0.05 located in language-related regions interest (Figure 2P). This function is trained in cross-validated way and evaluated on held-out data. The ridge penalty
14

---

Published as a conference paper at ICLR 2026
Table Examples for linguistic subfields from Waldis al. (2024). The relevant part of the example for the specific label is underlined.
Type
Phenomena
Example
Label
Morphology
Subject- Agreement
And then, the cucumber harled into the air. And then, cucumber were lutled into the dis. And then, the cucumber was harled isto the air. eucumber And then, the cucumber was hurled into the air. And then cucumber hurled into the air.
Correct
Keomy
Syntax
Purt-of-Spp Part-of-Spesch
NN
(Noun Singular)
Semantics
Semantic Roles
Dircction No Negation
Reasoning
Negation
Discourse
Node Type Rhetorical Tree
Satellite
Table 2: Examples for selected linguistic phenomena from Waldis et al. 2024 The asterisk (*)
indicates the correct option when applicable.
Phenomena
Illustrative Example
argument-structuue
Most cashiers disliked*ttirted..
binding
Carlos said that Lori helped himm/fi.self.
determiner noun agreement
Craig explored that grocery store"/store
event structure
Give them to a library or burn them. => Distributive
filler-gap
Brett knew what many waiters find. *Brrttt knew that many waiters find.
genericity
assume you mcan the crazy horse memorial. Not Dynamic
island-effects
Which bikes John ixing?*/Whice is John fixing bikes?
antonym negution
It was not nreally hot, it was cold.
negative polarity item licensing
Only/Even Bill would ever complain.
semantic proto-roles
These look fine to mc = Exists as physical
quantifiers
There aren't many lights darkening.
rhetorical structure theory
The statistics quoted by the new Census Bureau report Elaboration
subject-verb agreement
A sketch lights does not"/do not appear.
selected via nested vross-validation. For each participant, we train four functions, each using three of the four IMRI subsets for training and the remaining one for testing. To generate model representations, we average the token embeddings corresponding to each TR, and construct the input by concatenating the embeddings from the eurrent TR with those from the previous five TRs. The features of the words presented the previous TRs are included to account for the lag in the hemodynamic response that fMRI records. Because the response measured by fMRI an indirect consequence of brain activity that peaks about 6 seconds after stimulus onset, predictive methods commonly include preceding time points (Nishimoto al., 2011 Wehbe al.] 2014a] Huth al. 2016). This allows for a data-driven estimation of the hemodynamic response functions (HRFs) for each voxel, which is preferable to assuming one because different voxels may exhibit different HRFs.
15

---

Published as a conference paper at ICLR 2026
Table 3: List of linguistic phenomena and their corresponding subfields in the Holmes benchmark.
linguistic phenomena
subfield
linguistic phenomena
subfield
next sentence prediction rhetorical structure theory sentence order
discourse
semantic odd man out word sense
semantics
discourse
semantics
discourse
word content
semantics
discourse connective
discourse
coordination inversion
semantics
coreference resolution
discourse
object animacy
semantics
bridging
discourse
event structure
semantics
irregular forms
morphology
factuality
semantics
subject-verb agreement
morphology
complex words
semanties
determiner noun agreement
morphology
genericity
semantics
anaphor agreement age comparison
morphology
metaphor
semantics
reasoning
named entity labeling
semantics
negation
reasoning
negative polarity item licensing
semanties
speculation
reasoning
argument structure
syntax
multi-hop composition
reasoning
bigram-shift
syntax
property conjunction
reasoning
binding
syntax
object comparison
reasoning
tree-depth
syntax
antonym negation
reasoning
syntax
encyclopedic composition taxonomy conjunction
reasoning
subject-verb agreement
syntax
reasoning
anaphor agreement
syntax
always never
reasoning
top-constituentttask
syntax
object gender passive
semanties
subject number
syntax
semantics
deoncausativ-iincoattiv alternation
syntax
protoroles
semanties
control raising
syntax
quantifiers
semantics
ellipsis
syntax
synonym-/antonynn -tt--aaannooomdetettttonn
detecction
semantics
sentence length
syntax
verb dynamic
semantics
filler gap readability
syntax
semantic role labeling
semantics
syntax
sentiment analysis
semantics
island effects
syntax
time
semantics
local attractor
syntax
subject animacy
semantics
part-of-speech
syntax
subject gender
semantics
object number
syntax
tense
semantics
constituent parsing
syntax
relation classification
semantics
negative polarity item licensing
syntax
C NOISE CEILING ESTIMATION
To assess the signal quality of the fMRI data, we estimated noise ceiling values, which quantify the proportion of variance that could be explained by ideal data-generating model. This method involves predicting the fMRI activity of target participant using linear models trained on data from another participant. For more detailed explanation, refer to Schrimpp et al. (202T). Estimating the noise ceiling is particularly useful given the inherently high level of noise in [MRI data.
16

---

Published as a conference paper at ICLR 2026
00,05
Figure 7: Voxel-wise estimated noise ceiling values for participants included in Harry Potter dataset (Webbe al. 2014 To exclude noisy voxels, selected, for each participant, those with noise ceiling estimates above 0.05.
-0.05
Noise ceiling values
Figure 8: Voxel-wise estimated noise ceiling values for participants included in Moth Radio Hour dataset Deniz et al. 2019). exclude noisy voxels, we selected, for each participant, those with noise ceiling estimates above 0.05.
17

---

Published as a conference paper at ICLR 2026
D BERT MISALIGNMENT ON HARRY POTTER DATASET
We report the brain alignment results for Brain Misaligned and Brain Preserving trained with data from each participant in Figure as well as a quantitative summary in Figure TO Figure report the quantitative summary for brain alignment for the Brain Tuned model compared to Brain Pre- serving model. Results for the Holmes benchmark for all the comparisons are reported in Figure [2
15 16 17
18

---

Published as a conference paper at ICLR 2026
Brain Preserving
Brain Misaligned
Difference
0.3
0.3 -0.1
0.1
Figure 9: Performances of BERT-based Brain Misaligned and Brain Preserving models on the Harry Potter dataset the brain alignment task. Brain plots show voxel-wise Pearson correlations between model activations and brain responses for each subject. The left column displays results for the Brain Preserving model, the center column for the Brain Misaligned model, and the right column shows their difference (Preserving minus Misaligned). Warmer colors indicate stronger alignment with brain activity. These results illustrate the digtribution of brain alignment across subjects and highlight areas where brain misalignment has effects.

---

Published as a conference paper at ICLR 2026
Median
0.14
Brain iisaigred Brain Preserving
0.12
0.30
0.00
0.08
0.04
0.02
0.02
Figure 10: Median Pearson correlation for BERT-based models on the Harry Potter dataset for each participant. Brain Misaligned models perform significantly worse than Brain Preserving models for seven subjects (p < 0.05, indicated by and assessed using the Wilcoxon agmed-rank. test).
Preserving
0.175
(hanaa Canaatirr
0.150
0.125
0.100
0.005
0.000
Figure 11: Median Pearson correlation for BERT-based models on the Harry Potter dataset for each participant. Brain Preserving models perform significantly worse than Brain Tuned models for six subjects (p 0.05, indicated by and assessed using the Wilcoxon signed-ran rank test).
20

---

Published as a conference paper at ICLR 2026
A) All tasks
B) Tasks split by linguistic subfield
1.0
Bra Nical gred Srain massiving
0.8
Average win rate
Average win rata
0.6
0.4
0.4
0.2
Brain Misaligned
Brain Preserving
discourse
marphology
reasoning
sermamtics
Figure 12: Average win rate and standard error of the BERT-based Brain Misaligned and Brain Pre- serving models on the Harry Potter dataset across participants and tasks (Left) and across different linguistic subfields (Right). The win rate indicates how often each model outperforms its counterpart across tasks and participants. The Brain Preserving model significantly outperforms the Brain Mis- aligned model (p < 0.001, indicated by ***), as assessed using Wilcoxon signed-rank test (Left). This result suggests that removing brain alignment negatively influences linguistic competence. The Brain Preserving model shows significantly higher win rate in all the linguistic subfield (Right) (p < 0.05, Wilcoxon signed-rank test with Holm- Bonn--Borrrroii correction), suggesting that improving brain alignment affects all linguistic subfields.
Brain Misaligned Preserving
0.8
Average win rate
0.6
0.4
0.0
orgument structure
determiner noun agreement
binding
event structure
filler gap
genericity
island effects
negative polarity item licensing
negation
protorales
rethorical structure theory
quantifiers
Figure 13: Average win rate with standard error various linguistic phenomena for the BERT- based Brain Misaligned and Brain Preserving models on the Harry Potter dataset. Each bar repre- sents the average win rate for specific linguistic phenomenon, with error bars indicating standard emror. Some concrete examples of the linguistic tasks are provided the Table 2
21

---

Published as a conference paper at ICLR 2026
A) All tasks
B) Tasks split linguistic subfield
1.0
deesendrn
1.0
0.8
Average win rate
0.6
Average
0.6
0.2
0.2
Brain Preserving
Brain Tuned
0.0
discourse
Figure 14: Average win rate and standard error of the BERT-based Brain Preserving and Brain Tuned models on the Harry Potter dataset across participants and tasks (Left) and across different linguistic subfields (Right). The win rate indicates how often each model outperforms its counterpart across tasks and participants. The Brain Tuned model significantly outperforms the Brain Preserving model (p < 0.001, indicated by ***), as assessed using a Wilcoxon signed-rank test (Lefit.. This result suggests that improving brain alignment positively influences linguistic competence. The Brain Tuned model shows a higher win rate in the syntax, semantics, reasoning, morphology and discourse subfield (Right) and significantly higher for all linguistic subfields (p 0.05, Wilcoxon signed- rank test with Holm-Bonferroni correction), suggesting that improving brain alignment affects all linguistic subfields.
Preserving Brain Tuned
1.2
1.0
Average win rate
0.8
0.4
0.2
0.0
orgument structure
determiner noun agreement
binding
event structure
filler gap
genericity
island effects
negative polarity Item licensing
negation
protoroles
rethorical structure theory
quantifiers
Figure 15: Average win rate with standard error across various linguistic phenomena for the BERT- based Brain Preserving and Brain Tuned models the Harry Potter dataset. Each bar represents the average win rate for a specific linguistic phenomenon, with error bars indicating standard error. Some concrete examples of the linguistic tasks are provided the Table 2
22

---

Published as a conference paper at ICLR 2026
A) All tasks
B) Tasks split linguistic subfield
1.0
Rntrained
1.0
0.8
Average win rate
0.6
Average
0.6
0.2
0.2
Pretrained
Brain Tuned
0.0
discourse
Figure 16: Average win rate and standard error of the BERT-based Brain Tuned and Pretrained models on the Harry Potter dataset across participants and tasks (Left) and across different linguistic subfields (Right). The win rate indicates how often each model outperforms its counterpart across tasks and participants. The Brain Tuned model shows higher win rate in the morphology subfield (Right) (although Wilcoxon signed-rank test with Holm-Bonferee correction reveal no signifi- cance), suggesting that improving brain alignment affects this linguistic subfield.
Pretrained
Brain Tuned
Average win rate
0.6
0.4
0.2
0.0
orgument structure
determiner noun agreement
binding
event structure
filler gap
genericity
Island effects
negative polarity Item licensing
negation
protoroles
rethorical structure theory
quantifiers
Figure 17: Average win rate with standard error across various linguistic phenomena for the BERT- based Brain Tuned and Pretrained models the Harry Potter dataset. Each bar represents the average win rate for a specific linguistic phenomenon, with error bars indicating standard error. Some concrete examples of the linguistic tasks provided in the Table 2
23

---

Published as a conference paper at ICLR 2026
E BERT MISALIGNMENT ON MOTH RADIO HOUR DATASET
We report the brain alignment results for Brain Misaligned and Brain Preserving trained with data from each participant in Figure 18 as well as quantitative summary in Figure 19 Figure 20 report the quantitative summary for brain alignment for the Brain Tuned model compared to Brain Preserving model. Results for the Holmes benchmark for all the comparisons are reported in Figure
21
23
25
26
24

---

Published as a conference paper at ICLR 2026
Brain Preserving
Brain Misaligned
Difference
0.3 0
0.3 -0.1
0.1
Figure 18: Performances of BERT-based Brain Misaligned and Brain Preserving models the Moth Radio Hour dataset at the brain alignment task. Brain plots show voxel-wise Pearson correlations between model activations and brain responses for each subject. The left column displays results for the Brain Preserving model, the center column for the Brain Misaligned model, and the right column shows their difference (Preserving minus Misaligned). Warmer colors indicate stronger alignment with brain activity. These results illustrate the distribution of brain alignment across subjects and highlight areas where brain misalignment has effects.
25

---

Published as a conference paper at ICLR 2026
Median Pearson Correlations
0.045
Drain Misa igned Brain Preserving
0.340
Hodsan
0.030
0.025
0.020
Figure 19: Median Pearson correlation for BERT-based models on the Moth Radio Hour dataset for each participant. Brain Misaligned models perform significantly worse than Brain Preserving models for six subjects (p < 0.05, indicated by * and assessed using the Wilcoxon signed-rank test).
Preserving Tuned
0.345
0.040
(hanaaCCraaal Cormatiis
0.035
0.030
0.005
Figure 20: Median Pearson correlation for BERT-based models on the Moth Radio Hour dataset for each participant. Brain Preserving models perform significantly worse than Brain Tuned models for five subjects (p < 0.05, indicated by * and assessed using the Wilcoxon signed-rank test).
26

---

Published as a conference paper at ICLR 2026
A) All tasks
B) Tasks split by linguistic subfield
0.40
Stachert tor Lingent
0.35
Average win rate
C4
0.30
0.25
0.20
0.15
Brain Misaligned
Brain Preserving
Figure 21: Average win rate and standard error of the BERT- based Brain Misaligned and Brain Preserving models on the Moth Radio Hour dataset across participants and tasks (Left) and across different linguistic subfields (Right). The win rate indicates how often each model outperforms its counterpart across tasks and participants. The Brain Preserving model significantly outperforms the Brain Misaligned model (p 0.01, indicated by **), as assessed using a Wilcoxon signed-rank test (Left). This result suggests that removing brain alignment negatively influences linguistic compe- tence. The Brain Preserving model shows a higher win rate in the syntax, semantics, reasoning and discourse subfield (Right) and significantly higher for syntax and semantics subfields (p < 0.05, Wilcoxon signed-rank test with Holm-Bonferroni correction), suggesting that removing brain align- ment particularly affects syntax and semantic tasks.
0.6
Misaligned
Brain Preserving
0.5
Average win rate
0.3
0.2
argument structure
determiner noun agreement
binding
event structure
filler gap
genericity
island effects
negative polarity item licensing
negation
protoroles
rethorical structure theory
quantifiers
Figure 22: Average win rate with standard error across various linguistic phenomena for the BERT- based Brain Misaligned and Brain Preserving models the Moth Radio Hour dataset. Each bar represents the average win rate for a specific linguistic phenomenon, with error bars indicating standard error. Brain Preserving models tend outperform Brain Misaligned models, particularly in categories such as genericity and quantifiers. Some concrete examples of the linguistic tasks are provided in the Table
27

---

Published as a conference paper at ICLR 2026
A) All tasks
B) Tasks split linguistic subfield
1.0
deesendrn
1.0
0.8
Average win rate
0.6
Average
0.6
0.2
0.2
Brain Preserving
Brain Tuned
0.0 discourse
seranttccs
Figure 23: Average win rate and standard error of the BERT-based Brain Preserving and Brain Tuned models on the Moth Radio Hour dataset across participants and tasks (Left) and across dif- ferent linguistic subfields (Right). The win rate indicates how often each model outperforms its counterpart across tasks and participants. The Brain Tuned model significantly outperforms the Brain Preserving model (p < 0.001, indicated by ***), as assessed using a Wilcoxon signed-rank test (Left). This result suggests that improving brain alignment positively influences linguistic com- petence. The Brain Tuned model shows a higher win rate in the syntax, semantics, morphology and discourse subfield (Right) and significantly higher for syntax, semantics and morphology subfields correction), suggesting that improving
(p < 0.05, Wilcoxon signed-rank test with
brain alignment affects those linguistic subfields.
Preserving
Brain Tuned
#:0
Average win rate
0.0
orgument structure
determiner noun egreement
binding
event structure
filler gap
genericity
island effects
negative polarity Item licensing
negation
protoroles
rethorical structure theory
quantiflers
Figure 24: Average win rate with standard error across various linguistic phenomena for the BERT- based Brain Preserving and Brain Tuned models the Moth Radio Hour dataset. Each bar repre- sents the average win rate for a specific linguistic phenomenon, with error bars indicating standard error Some concrete examples the linguistic tasks are provided in the Table
28

---

Published as a conference paper at ICLR 2026
A) All tasks
B) Tasks split linguistic subfield
0.50
Rntrained
0.45
1.0
Average win rate
0.40
0.35
Average
0.6
0.30
0.2
Pretrained
Brain Tuned
0.0 discourse
seraantics
Figure 25: Average win rate and standard error of the BERT-based Brain Tuned and Pretrained models the Moth Radio Hour dataset across participants and tasks (Left)) and across different linguistic subfields (Right). The win rate indicates how often each model outperforms counterpart across tasks and participants. The Brain Tuned model significantly outperforms the Pretrained model (p < 0.05, indicated by *), as assessed using Wilcoxon signed-rank test (Left). This result suggests that improving brain alignment positively influences linguistic competence. The Brain Tuned model shows a higher win rate in the syntax subfield (Right) and significantly higher for syntax subfield correction), suggesting that improving
(p 0.05, Wilcoxon signed-rank test with brain alignment affects that linguistic subfield.
Pretrained
Brain Tuned
1.0
Average win rate
0.8
0.6
0.4
0.2
orgument structure
0.0
determiner noun egreement
binding
event structure
filler gap
genericity
island effects
negative polarity item licensing
negation
protoroles
rethorical structure theory
quantifiers
Figure 26: Average win rate with standard error across various linguistic phenomena for the BERT- based Brain Tuned and Pretrained models on the Moth Radio Hour dataset. Each bar represents the average win rate for specific linguistic phenomenon, with error bars indicating standard error. Some concrete examples of the linguistic tasks are provided the Table 2
29

---

Published as a conference paper at ICLR 2026
F GPT2 MISALIGNMENT ON HARRY POTTER BRAIN DATA RESULTS
We report the brain alignment results for Brain Misaligned and Brain Preserving trained with data from each participant in Figure 27 as well as quantitative summary in Figure 28 Figure 29 report the quantitative summary for brain alignment for the Brain Tuned model compared to Brain Preserving model. Results for the Holmes benchmark for all the comparisons are reported in Figure
30
32
33
34
35
30

---

Published as a conference paper at ICLR 2026
Brain Preserving
Brain Misaligned
Difference
0.3
0.3 -0.1
0.1
Figure 27: Performances of GPT2-based Brain Misaligned and Brain Preserving models on the Harry Potter dataset the brain alignment task. Brain plots show voxel-wise Pearson correlations between model activations and brain responses for each subject. The left column displays results for the Brain Preserving model, the center column for the Brain Misaligned model, and the right column shows their difference (Preserving minus Misaligned). Warmer colors indicate stronger alignment with brain activity. These results illustrate the distribution of brain alignment across subjects and highlight areas where brain misalignment has effects.

---

Published as a conference paper at ICLR 2026
Median Correlations
0.200
Broin Misa ig ng Brain Preserving
Median Narssn Conalations
0.150
0.100
0.000
Figure 28: Median Pearson correlation for GPT2-based models on the Harry Potter dataset for each participant. Brain Misaligned models perform significantly worse than Brain Preserving models for seven subjects (p < 0.05, indicated by and assessed using the Wilcoxon signed-rant test).
Preserving Tuned
0.175
0.150
Median Paanan Cormatiis
0.125
0.100
0.675
0.005
0.000
Figure 29: Median Pearson correlation for GPT2-based models on the Harry Potter dataset for each participant. Brain Preserving models perform significantly worse than Brain Tuned models for two subjects (p 0.05, indicated by and assessed using the Wilcoxon signed-rank test).
32

---

Published as a conference paper at ICLR 2026
A) All tasks
B) Tasks split by linguistic subfield
0.25
Brain Nical gred
0.11
Average Win rate
0.10
Average win rate
0.08
0.30
0.07
0.05
Brain Misaligned
Brain Preserving
discurse
morphology
reasoning
sementics
syntex
Figure 30: Average win rate and standard errоr of the GPT2-based Brain Misaligned and Brain Pre- serving models on the Harry Potter dataset across participants and tasks (Left) and across different linguistic subfields (Right). The win rate indicates how often each model outperforms its counter- part across tasks and participants. The Brain Preserving model outperforms the Brain Misaligned model (significance assessed using Wilcoxon signed-rank test reveal 0.055) (Left). This re- sult suggests that removing brain alignment negatively influences linguistic competence. The Brain Preserving model shows higher win rate in particular in the semantics and syntax subfield (Right) (although Wilcoxon signed-rant test with Bonf-ooorrrooi correction reveal no significance), sug- gesting that removing brain alignment particularly affects semantics and syntax processing tasks.
0.30
Brain Misaligned
Preserving
0.25
Average win rate
0.20
0.15
0.10
0.05
0.00
orgument structure
determiner noun agreement
binding
event structure
filler gap
genericity
island effects
negative polarity item licensing
negation
protorales
rethorical structure theory
quantifiers
Figure 31: Average win rate with standard error various linguistic phenomena for the GPT2- based Brain Misaligned and Brain Preserving models on the Harry Potter dataset. Each bar repre- sents the average win rate for specific linguistic phenomenon, with error bars indicating standard emror. Some concrete examples of the linguistic tasks are provided the Table 2
33

---

Published as a conference paper at ICLR 2026
A) All tasks
B) Tasks split linguistic subfield
0.30
0.25
0.4
Average win rate
0.20
Average ein aate
0.15
0.10
0.2
Brain Preserving
Brain Tuned
0.0 discourse
Average win rate and standard error of the GPT2-based Brain Preserving and Brain Tuned models the Harry Potter dataset across participants and tasks (Left) and across different linguistic subfields (Right). The win rate indicates how often each model outperforms counterpart across tasks and participants. The Brain Tuned model significantly outperforms the Brain Preserving model (p < 0.001, indicated by ***), as assessed using a Wilcoxon signed-rank test (Left). This re- sult suggests that improving brain alignment positively influences linguistic competence. The Brain Tuned model shows a higher win rate in the syntax, semantics, reasoning, morphology and discourse subfield (Right) and significantly higher for syntax and morphology subfields (p 0.05, Wilcoxon correction), suggesting that improving brain alignment af-
Figure 32:
signed-rank test with fects those linguistic subfields.
0.5
Brain Preserving
Brain Tuned
0.4
Average win rate
0.2
0.1
0.0
orgument structure
determiner noun egreement
binding
event structure
filler gap
genericity
island effects
negative polarity Item licensing
negation
protoroles
rethorical structure theory
quantifiers
Figure 33: Average win rate with standard error across various linguistic phenomena for the GPT2- based Brain Preserving and Brain Tuned models the Harry Potter dataset. Each bar represents the average win rate for specific linguistic phenomenon, with error bars indicating standard error. Some concrete examples of the linguistic tasks are provided the Table 2
34

---

Published as a conference paper at ICLR 2026
A) All tasks
B) Tasks split linguistic subfield
0.5
0.4
0.5
Average win rate
0.4
0.3
Average ein
0.2
0.1
Pretrained
Brain Tuned
0.0 discourse
seraantics
Figure 34: Average win rate and standard error of the GPT2- Brain Tuned and Pretrained mod- els on the Harry Potter dataset across participants and tasks (Left) and across different linguistic sub- fields (Right). The win rate indicates how often each model outperforms and participants. The Brain Tuned model shows higher win rate in the reasoning and discourse subfield (Right) and significantly higher for discourse subfield (p < 0.05, Wilcoxon signed-ram rank test with nonf-Bonferroni correction), suggesting that improving brain alignment affects that linguistic subfield.
counterpart across tasks
Pretrained
Average win rate
0.3
0.2
argument structure
determiner noun agreement
binding
event structure
filler gap
genericity
Island effects
negative polarity item licensing
negation
protoroles
rethorical structure theory
quantiflers
Figure 35: Average win rate with standard eπror across various linguistic phenomena for the GPT2- based Brain Tuned and Pretrained models on the Harry Potter dataset. Each bar represents the average win rate for specific linguistic phenomenon, with error bars indicating standard enror. Some concrete examples the linguistic tasks are provided the Table 2
35

---

Published as a conference paper at ICLR 2026
G GPT2 MISALIGNMENT ON MOTH RADIO HOUR DATASET
We report the brain alignment results for Brain Misaligned and Brain Preserving trained with data from each participant in Figure 36 as well as quantitative summary in Figure 37 Figure 38 report the quantitative summary for brain alignment for the Brain Tuned model compared to Brain Preserving model. Results for the Holmes benchmark for all the comparisons are reported in Figure
39
40
43
36

---

Published as a conference paper at ICLR 2026
Brain Preserving
Brain Misaligned
Difference
0.3 0
0.3 -0.1
0.1
Figure 36: Performances of GPT2-based Brain Misaligned and Brain Preserving models the Moth Radio Hour dataset at the brain alignment task. Brain plots show voxel-wise Pearson correlations between model activations and brain responses for each subject. The left column displays results for the Brain Preserving model, the center column for the Brain Misaligned model, and the right column shows their difference (Preserving minus Misaligned). Warmer colors indicate stronger alignment with brain activity. These results illustrate the distribution of brain alignment across subjects and highlight areas where brain misalignment has effects.
37

---

Published as a conference paper at ICLR 2026
Median Pearson Correlations
Brain Preserving
0.0375
0.0350
Medeen Neeron Corneetees
0.0325
0.0300
0.0275
Figure 37: Median Pearson correlation for GPT2-based models on Moth Radio Hour dataset for each participant. Brain Misaligned models perform significantly worse than Brain Preserving models for 3 subjects
0.05, indicated * and assessed using the Wilcoxon signed-rank test).
0.050
Preserving
Brain Tured
Comeletions
0.035
0.030
0.025
0.020
0.015
0010
Figure 38: Median Pearson correlation for GPT2-based models on Moth Radio Hour dataset for each participant. Brain Preserving models perform significantly worse than Brain Tuned models for six subjects (p < 0.05, indicated by # and assessed using the Wilcoxon signed-rank test).
A) All tasks
B) Tasks split by linguistic subfield
0.30
Ninal gred Thoseriigg
Average win rate
Average With rate
0.14
0.35
0.12
0.30
0.10
0.08
Brain Misaligned
Brain Preserving
discourse
marphology
reasoming
sermrrtics
syntax
Figure 39: Average win rate and standard error of the GPT2-based Brain Misaligned and Brain Preserving models on the Moth Radio Hour dataset across participants and tasks (Left) and across different linguistic subfields (Right). The win rate indicates how often each model outperforms its counterpart across tasks and participants. The Brain Preserving model shows a higher win rate (Right) in the semantics and morphology subfield (although Wilcoxon signed-rank test with Holm- Bonferroni correction reveal no significance), suggesting that removing brain alignment particularly affects semanties and morphology tasks.
38

---

Published as a conference paper at ICLR 2026
0.30
Misaligned
Brain Preserving
0.25
Average win rate
0.20
0.15
0.10
0.05
0.00
argument structure
determiner noun agreement
binding
event structure
filler gap
genericity
island effects
negative polarity item licensing
negation
protoroles
rethorical structure theory
quantifiers
Figure 40: Average win rate with standard error across various linguistic phenomena for the GPT2- based Brain Misaligned and Brain Preserving models on the Moth Radio Hour dataset. Each bar represents the average win rate for specific linguistic phenomenon, with error bars indicating standard error. Brain Preserving models tend to outperform Brain Misaligned models, particularly in categories such as genericity, event structure and binding. Some concrete examples of the linguistic tasks are provided in the Table
39

---

Published as a conference paper at ICLR 2026
A) All tasks
B) Tasks split linguistic subfield
0.6
Tueed
0.5
Average win rate
0.4-
Asarage ein rite
0.3
0.2
0.1
Brain Preserving
Brain Tuned
0.0 discourse
Figure 41: Average win rate and standard error of the GPT2-based Brain Preserving and Brain Tuned models the Moth Radio Hour dataset across participants and tasks (Left) and across different linguistic subfields (Right). The win rate indicates how often each model outperforms counterpart across tasks and participants. The Brain Tuned model significantly outperforms the Brain Preserving model (p < 0.001, indicated by ***), as assessed using a Wilcoxon signed-rank test (Left). This result suggests that improving brain alignment positively influences linguistic competence. The Brain Tuned model shows higher win rate the syntax, semantics, reasoning, morphology and discourse subfield (Right) and significantly higher for syntax, semantics, reasoning and discourse subfields (p < 0.05, Wilcoxon signed-rank test with Holm-Bonferroni correction), suggesting that improving brain alignment affects those linguistic subfields.
Preserving Brain Tuned
Average win rate
orgument structure
0.0
determiner noun agreement
binding
event structure
filler gap
genericity
island effects
negative polarity Item licensing
negation
protoroles
rethorical structure theory
quantiflers
Figure 42: Average win rate with standard error across various linguistic phenomena for the GPT2- based Brain Preserving and Brain Tuned models the Moth Radio Hour dataset. Each bar repre- sents the average win rate for a specific linguistic phenomenon, with error bars indicating standard error Some concrete examples the linguistic tasks are provided in the Table
40

---

Published as a conference paper at ICLR 2026
A) All tasks
B) Tasks split linguistic subfield
0.5
0.4
0.5
Average win rate
0.4
0.3
Average ein
0.2
0.1
Pretrained
Brain Tuned
0.0 discourse
seranttcs
Figure 43: Average win rate and standard error of the GPT2-based Brain Tuned and Pretrained models the Moth Radio Hour dataset across participants and tasks (Left) and across different linguistic subfields (Right). The win rate indicates how often each model outperforms its counter- part across tasks and participants. The Brain Tuned model significantly outperforms the Pretrained model (p < 0.001, indicated by ***), as assessed using a Wilcoxon signed-rank test (Left). This re- sult suggests that improving brain alignment positively influences linguistic competence. The Brain Tuned model shows a higher win rate in the syntax, semantics, reasoning, morphology and discourse subfield (Right) and significantly higher for reasoning and discourse subfields (p 0.05, Wilcoxon correction), suggesting that improving brain alignment af-
signed-rank test with fects those linguistic subfields.
0.8
Pretrained
0.7
Brain Tuned
Average win rate
0.4
0.3
0.2
0.0
orgument structure
determiner noun agreement
binding
event structure
filler gap
genericity
island effects
negative polarity Item licensing
negation
protoroles
rethorical structure theory
quantifiers
Figure 44: Average win rate with standard error across various linguistic phenomena for the GPT2- based Brain Tuned and Pretrained models the Moth Radio Hour dataset. Each bar represents the average win rate for specific linguistic phenomenon, with error bars indicating standard error. Some concrete examples of the linguistic tasks are provided in the Table 2
41

---

Published as a conference paper at ICLR 2026
H LLAMA MISALIGNMENT ON HARRY POTTER DATASET
We report the brain alignment results for Brain Misaligned and Brain Preserving trained with data from each participant in Figure 45 as well as quantitative summary in Figure 46 Figure 47 report the quantitative summary for brain alignment for the Brain Tuned model compared to Brain Preserving model. Results for the Holmes benchmark for all the comparisons are reported in Figure
48
50
52 53
42

---

Published as a conference paper at ICLR 2026
Brain Preserving
Brain Misaligned
Difference
0.3
0.3 -0.1
0.1
Figure 45: Performances of Llama-based Brain Misaligned and Brain Preserving models on the Harry Potter dataset the brain alignment task. Brain plots show voxel-wise Pearson correlations between model activations and brain responses for each subject. The left column displays results for the Brain Preserving model, the center column for the Brain Misaligned model, and the right column shows their difference (Preserving minus Misaligned). Warmer colors indicate stronger alignment with brain activity. These results illustrate the distribution of brain alignment across subjects and highlight areas where brain misalignment has effects.

---

Published as a conference paper at ICLR 2026
Median Correlations
0.20
Brain Wsaligned Brain Preserving
0.15
Conslalion
0.00
Figure 46: Median Pearson correlation for Llama-based models the Harry Potter dataset for each participant. Brain Misaligned models perform significantly worse than Brain Preserving models for seven subjects (p 0.05, indicated by and assessed using the Wilcoxon signed-rank test).
Median Pearson Correlations
0.20
Brain Preserving Tuned
0.15
0.30
Figure 47: Median Pearson correlation for Llama-based models on the Harry Potter dataset for each participant. Brain Preserving models perform significantly worse than Brain Tuned models for seven subjects (p 0.05, indicated by * and assessed using the Wilcoxon signed-rank test).
44

---

Published as a conference paper at ICLR 2026
A) All tasks
B) Tasks split by linguistic subfield
0.30
Srain Mical gnzd fia Preserving
0.175
Average win rate
0.150
Average win raee
0.20
0.125
0.15
0.100
0.10
0.075
0.050
Misaligned
Preserving
discourse
morphology
reasoning
semantiss
syntak
Figure 48: Average win rate and standard error of the Llama-based Brain Misaligned and Brain Pre- serving models on the Harry Potter dataset across participants and tasks (Left) and across different linguistic subfields (Right). The win rate indicates how often each model outperforms its counterpart across tasks and participants. The Brain Preserving model significantly outperforms the Brain Mis- aligned model (p 0.001, indicated by ***), assessed using Wilcoxon signed-rank test (Left). This result suggests that removing brain alignment negatively influences linguistic competence. The Brain Preserving model shows higher win rate in the syntax, semantics, reasoning, morphology and discourse subfield (Right) and significantly higher for syntax and semantics subfields (p < 0.05, Wilcoxon signed-rank test with
correction), suggesting that removing brain align- ment particularly affects syntax and semantic tasks.
0.40
Brain Misaligned
0.35
Brain Preserving
0.30
Average win rate
0.25
0.20
0.15
0.10
0.05
0.00
argument structure
determiner noun agreement
binding
event structure
filler gap
genericity
island effects
negative polarity item licensing
negation
protoroles
rethorical structure theory
quantifiers
Figure 49: Average win rate with standard enror across various linguistic phenomena for the Llama- based Brain Misaligned and Brain Preserving models on the Harry Potter dataset. Each bar repre- sents the average win rate for specific linguistic phenomenon, with error bars indicating standard emror. Some concrete examples the linguistic tasks are provided the Table 2
45

---

Published as a conference paper at ICLR 2026
A) All tasks
B) Tasks split linguistic subfield
0.4
Tuned
Average win rate
0.3
Asarage ein
0.2
0.1
0.1
Brain Preserving
Brain Tuned
0.0
discourse
semantics
Figure 50: Average win rate and standard error of the Llama-basse Brain Preserving and Brain Tuned models the Harry Potter dataset across participants and tasks (Left) and across different linguistic subfields (Right). The win rate indicates how often each model outperforms counterpart across tasks and participants. The Brain Tuned model significantly outperforms the Brain Preserving model (p < 0.001, indicated by ***), as assessed using a Wilcoxon signed-rank test (Left). This result suggests that improving brain alignment positively influences linguistic competence. The Brain Tuned model shows higher win rate the syntax, semantics, reasoning, morphology and discourse subfield (Right) and significantly higher for syntax, semantics and morphology subfields correction), suggesting that improving
(p < 0.05, Wilcoxon signed-rank test with
brain alignment affects all linguistic subfields.
1.0
Brain Preserving
Brain Tuned
0.8
Average win rate
0.4
0.2
orgument structure
0.0
determiner noun egreement
binding
event structure
filler gap
genericity
island effects
negative polarity Item licensing
negation
protoroles
rethorical structure theory
quantiflers
Figure 51: Average win rate with standard error across various linguistic phenomena for the Llama- based Brain Preserving and Brain Tuned models the Harry Potter dataset. Each bar represents the average win rate for a specific linguistic phenomenon, with error bars indicating standard error. Some concrete examples of the linguistic tasks are provided the Table 2
46

---

Published as a conference paper at ICLR 2026
A) All tasks
B) Tasks split linguistic subfield
0.4
Orais
0.5
Average win rate
0.3
0.4
Average ein
0.2
0.1
0.1
Pretrained
Brain Tuned
0.0 discourse
seranntcss
Figure 52: Average win rate and standard error of the Llama-bases Brain Tuned and Pretrained mod- els on the Harry Potter dataset across participants and tasks (Left) and across different linguistic sub- fields (Right). The win rate indicates how often each model outperforms and participants. The Brain Tuned model significantly outperforms the Pretrained model (p 0.001, indicated by ***), as assessed using a Wilcoxon signed-rank test (Left). This result suggests that im- proving brain alignment positively influences linguistic competence. The Brain Tuned model shows a higher win rate in the syntax, semantics and reasoning subfields (Right) and significantly higher for semantics and syntax subfields < 0.05, Wilcoxon signed-rank test with correction), suggesting that improving brain alignment affects those linguistic subfields.
counterpart across tasks
Pretrained
0.8
Average win rate
0.6
0.4
orgument structure
0.0
determiner noun egreement
binding
event structure
filler gap
genericity
island effects
negative polarity item licensing
negation
protoroles
rethorical structure theory
quantifiers
Figure 53: Average win rate with standard error across various linguistic phenomena for the Llama- based Brain Tuned and Pretrained models on the Harry Potter dataset. Each bar represents the average win rate for specific linguistic phenomenon, with error bars indicating standard error. Some concrete examples of the linguistic tasks are provided in the Table
47

---

Published as a conference paper at ICLR 2026
I LLAMA MISALIGNMENT ON MOTH RADIO HOUR DATASET
We report the brain alignment results for Brain Misaligned and Brain Preserving trained with data from each participant in Figure 54 as well as quantitative summary in Figure 53 Figure [66] report the quantitative summary for brain alignment for the Brain Tuned model compared to Brain Preserving model. Results for the Holmes benchmark for all the comparisons reported in Figure
57 58
60 61 62
48

---

Published as a conference paper at ICLR 2026
Brain Preserving
Brain Misaligned
Difference
0.3 0
0.3 -0.1
0.1
Figure 54: Performances of Llama-base Brain Misaligned and Brain Preserving models the Moth Radio Hour dataset at the brain alignment task. Brain plots show voxel-wise Pearson corre- lations between model activations and brain responses for each subject. The left column displays results for the Brain Preserving model, the center column for the Brain Misaligned model, and the right column shows their difference (Preserving minus Misaligned). Warmer colors indicate stronger alignment with brain activity. These results illustrate the distribution of brain alignment across sub- jects and highlight areas where brain misalignment has effects.
49

---

Published as a conference paper at ICLR 2026
Median Pearson Correlations
Brain Preserving
Modian Consaation
0.03
0.01
Figure 55: Median Pearson correlation for Llama-based models on the Moth Radio Hour dataset for each participant. Brain Misaligned models perform significantly worse than Brain Preserving models for eight subjects (p 0.05, indicated by and assessed using the Wilcoxon signed-rank test).
0.050
Brain Preserving Tuned
0.045
Coreaatinn
0.040
0.035
0.030
0.000
0.15
0010
Figure 56: Median Pearson correlation for (luma-based models on the Harry Potter dataset for each participant. Brain Preserving models perform significantly worse than Brain Tuned models for three subjects (p < 0.05, indicated by and assessed using the Wilcoxon signed-rant -ank test).
50

---

Published as a conference paper at ICLR 2026
A) All tasks
B) Tasks split by linguistic subfield
0.30
Srain Mical grzd lin Preserving
0.175
Average win rate
0.150
Avarage win raee
0.20
0.125
0.15
0.100
0.10
0.075
0.050
Brain Misaligned
Brain Preserving
discourse
morphology
reasoning
semantics
syntax
Figure 57: Average win rate and standard error of the Llama-based Brain Misaligned and Brain Preserving models on the Moth Radio Hour dataset across participants and tasks (Left) and across different linguistic subfields (Right). The win rate indicates how often each model outperforms its counterpart across tasks and participants. The Brain Preserving model significantly outperforms the Brain Misaligned model (p < 0.001, indicated by ***), assessed using Wilcoxon signed- rank test (Left). This result suggests that removing brain alignment negatively influences linguistic competence. The Brain Preserving model shows higher win rate in the syntax, reasoning, mor- phology and discourse subfield (Right) and significantly higher for syntax and morphology subfields (p 0.05, Wilcoxon signed-rank test with Boof-Borferroni correction). suggesting that removing brain alignment particularly affects syntax and morphology tasks.
0.40
Brain Misaligned
0.35
Brain Preserving
0.30
Average win rate
0.25
0.20
0.15
0.10
0.05
0.00
argument structure
determiner noun agreement
bindlng
event structure
filler gap
genericity
island effects
negative polarity item licensing
negation
protoroles
rethorical structure theory
quantifiers
Figure 58: Average win rate with standard enror across various linguistic phenomena for the Llama- based Brain Misaligned and Brain Preserving models on the Moth Radio Hour dataset. Each bar represents the average win rate for specific linguistic phenomenon, with error bars indicating standard error. Some concrete examples of the linguistic tasks are provided in the Table
51

---

Published as a conference paper at ICLR 2026
A) All tasks
B) Tasks split linguistic subfield
0.200
Tuned
0.175
0.4
Average win rate
0.150
Average ein caee
0.125
0.100
0.1
0.050
Brain Preserving
Brain Tuned
0.0 discourse
seranttccs
Figure 59: Average win rate and standard eπror of the Alama-based Brain Preserving and Brain Tuned models on the Moth Radio Hour dataset across participants and tasks (Left) and across dif- ferent linguistic subfields (Right). The win rate indicates how often each model outperforms its counterpart across tasks and participants. The Brain Tuned model significantly outperforms the Brain Preserving model (p < 0.001, indicated by ***), as assessed using a Wilcoxon signed-rank test (Left). This result suggests that improving brain alignment positively influences linguistic com- petence. The Brain Tuned model shows a higher win rate in the syntax, semantics and discourse subfields (Right) and significantly higher for syntax and semantics subfields (p < 0.05, Wilcoxon signed-rank test with fects those linguistic subfields.
correction), suggesting that improving brain alignment af-
0.8
Brain Preserving
0.7
Brain Tuned
Average win rate
0.4
0.3
0.2
0.0
orgument structure
determiner noun egreement
binding
event structure
filler gap
genericity
island effects
negative polarity Item licensing
negation
protoroles
rethorical structure theory
quantiflers
Figure 60: Average win rate with standard error across various linguistic phenomena for the Llama- based Brain Preserving and Brain Tuned models the Moth Radio Hour dataset. Each bar repre- sents the average win rate for a specific linguistic phenomenon, with error bars indicating standard error Some concrete examples the linguistic tasks are provided in the Table
52

---

Published as a conference paper at ICLR 2026
A) All tasks
B) Tasks split linguistic subfield
0.30
Orais
0.25
0.5
Average win rate
0.20
0.4
0.15
Average
0.10
0.1
Pretrained
Brain Tuned
0.0 discourse
seraantccs
Figure 61: Average win rate and standard error of the Llama-based Brain Tuned and Pretrained models on the Moth Radio Hour dataset across participants and tasks (Left) and across different lin- guistic subfields (Right). The win rate indicates how often each model outperforms its counterpart across tasks and participants. The Brain Tuned model significantly outperforms the Pretrained model (p < 0.01, indicated by #*), as assessed using a Wilcoxon signed-rank test (Left). This result sug- gests that improving brain alignment positively influences linguistic competence. The Brain Tuned model shows a higher win rate in the syntax, semantics, reasoning and discourse subfield (Right) and significantly higher for syntax correction), suggesting that improving brain alignment affects this linguistic subfield.
0.05, Wilcoxon signed-rank test with
Pretrained
0.8
Average win rate
0.6
0.4
orgument structure
0.0
determiner noun egreement
binding
event structure
filler gap
genericity
island effects
negative polarity item licensing
negation
protoroles
rethorical structure theory
quantifiers
Figure 62: Average win rate with standard error across various linguistic phenomena for the Llama- based Brain Tuned and Pretrained models on the Moth Radio Hour dataset. Each bar represents the average win rate for specific linguistic phenomenon, with enror bars indicating standard error. Some concrete examples of the linguistic tasks are provided the Table 2
53

---

Published as a conference paper at ICLR 2026
J
AVERAGED COMPARISONS: ADDITIONAL RESULTS
Preserving
Tured
0.8
Average win rete
0.4
argument structure
determinar noun egreement
binding
event structure
fiter gap
gemericity
ssland effeets
negative polarity item licensing
negation
protoroles
rethonical structure theary
quantiflars
Figure 63: Averaged win rate with standard error for all model and dataset combinations across various linguistic phenomena of the Brain Tuned and Brain Preserving models. Each bar represents the average win rate for a specific linguistic phenomenon, with error bars indicating standard error. Brain Tuned models tend to outperform Brain Preserving models, particularly in categories such as filler gap and negative polarity item licensing, Some concrete examples of the linguistic tasks are provided in the Table 2
54

---

Published as a conference paper at ICLR 2026
A) All tasks
B) Tasks split by linguistic subfield
0.30
0.6
0.25
0.5
Average win rale
0.20
Average ein
0.2
0.10
0.1
0.05
Pretrained
Brain Tuned
morphology
reasoning
semantics
ntax
Figure 64: Averaged win rate and standard error for all the model and dataset combinations of the Brain Tuned and Pretrained models across tasks (Left) and across different linguistic subfields (Right). The win rate indicates how often each model outperforms its counterpart across tasks and participants. The Brain Tuned models outperform the Pretrained models (Left). This result suggests that train to align with brain recording leads to improvement in linguistic competence. The Brain Tuned model shows higher win rate the syntax and semantics subfields (Right) (although Wilcoxon signed-rank test with Holm-Bonnferron correction reveal no significance), suggesting that removing brain alignment particularly affects those tasks.
0.6
Pretrained Brain Tunad
0.5
Average win rete
0.1
argumtent structure
0.0
deterniner noun agreement
binding
event structure
genericity
island offects
reggative polarty item licensing
negation
protorotes
rethorical structure theary
auantifiers
filler
Figure 65: Averaged win rate and standard eттor for all the model and dataset combinations of the Brain Tuned and Pretrained models across different linguistic phenomena (Right). Each bar represents the average win rate for a specific linguistic phenomenon, with error bars indicating standard error. Brain Tuned models tend to outperform Pretrained models. Some concrete examples of the linguistic tasks are provided in the Table 2
55


Reviews:
[{"review_id": "BYarPhzW7K", "reviewer": "Reviewer_SLR2", "summary": "This paper studies a simple but important question. Does the alignment between large language models (LLMs) and brain activity matter, or is it just a coincidence? To answer this, the authors use adversarial training to erase the information in the model that can predict brain activity, without affecting the model’s language ability.\nThey use BERT-base and GPT-small models on two fMRI datasets. As a control, they train other models in which the brain data is shuffled, to rule out effects from the training itself.\nThey then evaluate these models on over 200 linguistic tasks. Models where \"brain alignment\" was erased perform worse on semantics, syntax, and reasoning tasks. The authors conclude that brain-aligned representations are important for high-level language ability in LLMs.", "questions": "see weakness.", "limitations": "", "rating": 4, "confidence": 1, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The main contribution is proposing a new paradigm beyond simple correlation studies. The paper creates “brain de-aligned” models through adversarial training, with “brain retention” models as controls. This is a well-designed intervention experiment, providing a new tool for studying the function of representations.\n2. Using the Holmes benchmark to evaluate over 200 fine-grained linguistic tasks is a major strength. It allows the authors to see how de-alignment affects various aspects of language, not just a single task.", "weaknesses": "1. The key findings are only based on bert-base and gpt-small. In 2025, using these models is not enough to study emergent properties like brain alignment in LLMs. Many key language abilities and representations only show up in much larger models (for example, over 7B parameters). Do these findings hold for modern models like Llama or Qwen?\n2.  On the Harry Potter dataset, the difference in GPT-2 model performance is not statistically significant (p=0.055). On the Moth Radio Hour dataset, the authors say the results are inconsistent because the “brain removal” method was weak. Appendix Figure 18 shows that the method only works for 3 out of 6 subjects. This inconsistency (good results for BERT, but not for GPT-2) weakens the main message. The paper does not explore why such architecture differences (encoder vs decoder) produce such different results.\n\nIn addition, there is a '??' at line 337, which looks unprofessional in a formal submission.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ghWzy0Ku0C", "reviewer": "Reviewer_J54j", "summary": "The paper trains LLMs to *poorly* predict brain activity (Wehbe et al. 2014 HP and Deniz et al. 2019 MRH fMRI datasets), in order to test the relevance of brain alignment for language task performance.\nThe main claim is that LLMs misaligned to the brain exhibit poor downstream performance, with which the authors argue that brain alignment is critical for linguistic competence.", "questions": "Please see Weaknesses 1-3 in particular. I am willing to increase my score with the evaluation of larger models (e.g., Qwen2.5-72B, Pythia, OLMo), and a quantification of performance deltas (rather than win rate). \n\nI like the misalignment training method! Please describe related work of training models with brain data.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Explicitly training models for brain misalignment is novel as far as I am aware. This is a cool method for causally testing the effect of neural data regularization in both directions.\n\n2. Great control: models trained for misalignment are compared with models under the same training pipeline but with permuted stimulus-response pairs.\n\n3. Methods and results are presented clearly, making the paper easy to follow.", "weaknesses": "### 1. Small scale of tested models\nThe models under consideration (BERT-base and GPT-small) are very small by modern standards, limiting the applicability of claims to the state of the art. In particular, smaller models might lack capacity to retain high linguistic performance despite reduced brain alignment, whereas larger models' performance might be affected by brain-misalignment less strongly or not at all.\n\n### 2. Small effect size\nThe average win rate between misaligned model and control differs by 2 percent points (0.19 vs 0.21). I further wonder what the actual difference in performance is. Even small performance differences can result in different win rates, but the effect size nonetheless can well be very small.\n\n### 3. Increased alignment of misaligned model in some regions\nThere is something weird with the training: why are the models trained to be misaligned actually more aligned for a substantial number of voxels (Fig. 2C)? It seems the authors attribute this to such regions not being language-related, but since the model is trained with whole-brain data this shouldn't happen.\n\n\n### Minor\n* L247 typo \"more tha**t** 200 datasets\"\n* L337 missing reference \"??\"\n* Figure 3A: shouldn't the win rate between two models sum up to 1?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7jTZi7HjV0", "reviewer": "Reviewer_7k7B", "summary": "The authors introduce Brain Misaligned models that are adversarially fine-tuned, with a gradient-reversal head trained to anti-predict fMRI, to decrease brain predictivity while preserving standard LM loss, and Brain Preserving controls trained identically except that fMRI targets are permuted so that adversarial updates cannot meaningfully target the true mapping. The setup uses BERT-base and GPT-2-small, fMRI from Harry Potter reading and Moth Radio Hour stories, and evaluates language modeling, brain predictivity, and downstream linguistic competence via Holmes/FlashHolmes (>200 probing datasets spanning semantics, syntax, morphology, discourse, reasoning). Core findings: (i) the misaligned models reliably reduce voxel-wise brain predictivity in language ROIs without degrading LM loss; (ii) across participants and seeds, misalignment lowers win-rates on the Holmes benchmark overall, with especially robust drops in semantics (and often syntax/reasoning). The pattern replicates across model/dataset pairs, with clearest effects for BERT; GPT-2 shows similar trends with weaker significance in one setting. The paper argues this is causal evidence that brain-aligned information supports linguistic competence.", "questions": "- How layer-localized is the removal? Did you inspect per-layer brain predictivity to see whether misalignment concentrates in earlier vs later layers, and whether the competence drop tracks specific layers/heads?\n- If you compute brain-alignment with RSA or CKA (and perhaps ROI-wise CKA), do you still see the same manipulation check and downstream deficits? Any reason to expect metric dependence? A nice work comparing representation similarity frameworks is by [Williams, 2024](https://openreview.net/forum?id=zMdnnFasgC).\n- Could you repeat the intervention with INLP or RLACE on frozen representations (or hybrid: INLP on a projection bottleneck during fine-tuning) to test whether the competence drop is method-agnostic?\n- Have you tried Narratives by [Nastase et al., 2021](https://www.nature.com/articles/s41597-021-01033-3) (multiple stories, many subjects) or the sentence-level fMRI dataset by [Pereira et al. (2018)](https://www.nature.com/articles/s41467-018-03068-4) to probe genre and task sensitivity of the effect? If not, what are the expected obstacles (timing alignment, TR windows, etc.)?\n- Beyond probing, do misaligned models underperform on GLUE/SuperGLUE, coreference, and NLI when controlling for instruction-tuning? If tested, please report; if not, justify relying solely on probing.\n- Your permutation control is strong, but adversarial training can still regularize or distort representation geometry in ways unrelated to brain signals. Can you add a no-brain adversarial control (e.g., adversary trained on shuffled text features or random vectors) to bound generic adversarial side-effects?\n- Is there a specific reason competence evaluation relies on probing? Holmes is excellent for linguistic phenomena, but do you think adding task benchmarks (e.g., GLUE/SuperGLUE, coreference, QA) or behavioral psycholinguistic tests would help bridge representational competence to end-task behavior?\n\n**Comments:**\n- Lines 31–32 lack a reference to a comprehensive prior survey: [Karamolegkou et al., 2023](https://aclanthology.org/2023.findings-acl.618/).\n- Lines 076-090: The contributions paragraph reads partly like a summary of results (\"We find that the competence drop is especially pronounced…\"). Conceptually, contributions should describe what is contributed (e.g., a causal framework for testing brain alignment), not what was found. I suggest rephrasing to highlight the methodological and theoretical advances, moving empirical claims to the Results section.\n- Lines 101-103: I would recommend tightening causal language. For example, I am not sure whether you study if brain alignment *is necessary for* maintaining linguistic competence in language models, maybe I would rephrase it as *is implicated in*. \n- By Line 101, the reader still does not know that there are two model \"types\" used: brain misaligned and brain preserving. In the abstract you mention the brain preserving models as \"well-matched brain aligned counterparts\". It would be helpful to clarify earlier that both are adversarially fine-tuned variants of the base model, differing only in whether the fMRI data are permuted. This distinction is crucial for interpreting the experimental design and for understanding how the \"alignment\" being tested is manipulated. Brain-preserving models should be added to your contributions. \n- Section 3.3.2 (“Brain Preserving Model”) is confusing to me. If the fMRI responses are permuted during training, how can this model be described as “preserving” brain alignment? You mention that you \"permute the order of the fMRI images to disrupt the correspondence between stimuli and brain activity\", this to me does not suggests brain preserving. Maybe the name or the explanation needs refinement... perhaps \"Brain-Control\" or \"Permutation-Control\" would be more transparent? In addition, a zero-shot baseline and a standard brain-fine-tuned model (finetuned to predict brain signal given language stimuli, or vice versa) would provide valuable reference points, as I have noted above.\n- Lines 224–228: Please specify the number of test examples. The current phrasing, \"For each test example, it is measured the average cross entropy across the randomly masked tokens\", leaves unclear the dataset size and sampling process. Given that the Harry Potter fMRI dataset contains only around 1,211 brain images, it would be important to clarify (i) how many text–fMRI pairs were used for testing, (ii) whether multiple tokens per TR were aggregated, and (iii) how the averaging procedure interacts with the limited sample size. This information is essential for assessing the robustness and generalizability of the reported results.\n- Lines 336–337: Please correct the cross-references \"Appendix C and ??\".\n- Figures 3 and 4 would benefit from including a bar with baseline model that is neither misaligned nor brain-preserving (e.g., the original pretrained and/or a fine-tuned version). This would allow readers to disentangle whether observed differences arise from misalignment, permutation control, or general pretraining/fine-tuning effects.\n- The statistical significance markers (asterisks) in Figures 3 and 4 are difficult to interpret—please ensure consistent notation and a clear legend.\n- In Figures 3 and 11, the brain-misaligned models seem to outperform baselines on morphological phenomena. This is an intriguing and unexpected finding that deserves discussion, as it may indicate selective effects of the adversarial objective across linguistic domains.\n- Although the abstract claims experiments on two fMRI datasets and two models, the main paper only visualizes results for BERT + Harry Potter. Including GPT-2 and Moth results (even as additional bars in Figures 3 and 4) would greatly strengthen the consistency and interpretability of the findings. The divergence between BERT and GPT-2 patterns otherwise leaves conclusions somewhat underdetermined.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper is conceptually original in treating brain–LM alignment as an intervenable property rather than a purely correlational observation. Constructing “brain-misaligned” variants via adversarial fine-tuning and contrasting them with a permutation-based control is a creative way to probe causal relevance.\n\n- Methodologically, the study is careful about a manipulation check (reduced brain predictivity in language ROIs) and about holding general LM performance roughly fixed during selection, which increases confidence that downstream effects are tied to the intervention rather than to obvious capacity loss. Empirically, the evaluation spans a broad set of linguistic competencies (Holmes/FlashHolmes).\n\n- The paper is generally clear and well organized, with helpful schematics and ROI visualizations. It is a pity that most of the experiments/results are added to the appendix and are not integrated into Figures 3 and 4.", "weaknesses": "- The framing somewhat overstates what the intervention establishes. Studying the effects of removing alignment does not automatically prove that alignment, per se, is beneficial. Removing brain–LM similarity and observing performance drops suggests that aligned information may correlate with linguistic competence, but it does not directly prove that alignment itself is necessary or causally beneficial. How do we ensure that the suggested manipulation does not act as structured noise or remove semantically relevant information? It would help to clarify whether this intervention reflects targeted “de-alignment” or a broader perturbation of representational geometry. Similarly, the construct of alignment could be made more robust: the paper currently relies on a single linear voxel-prediction metric, but conclusions might vary across similarity frameworks (e.g., RSA, CKA, or Procrustes) and layers. Reporting whether results hold across metrics, or demonstrating metric invariance, would strengthen causal claims and continue the discussion by recent comparisons ([Bo et al., 2025](https://openreview.net/forum?id=JGANEMteY8)). Finally, the naming of the Brain Preserving model is potentially misleading, since permuting fMRI targets does not intuitively preserve alignment; a label such as Permutation-Control would be clearer, and the distinction between misaligned and control models should appear earlier and be included among the stated contributions.\n\n- The experimental design would be stronger with two additional reference points: a zero-shot base model and a classic brain-tuned model (trained to predict brain activity given language stimuli or vice versa) to help disentangle generic fine-tuning effects from alignment-specific ones. Reporting and visualization also need tightening: please specify the exact number of test examples and how test samples are derived for cross-entropy estimation (especially given the ~1,211 images in Harry Potter), clarify whether multiple tokens per TR are aggregated, and whether averaging is per token or per example. \n\n- Some results are fragile and not adequately discussed. GPT-2/Harry Potter shows only a trend (p≈0.055), and Moth/GPT-2 is inconsistent due to weaker removal; discussing why adversarial removal is less effective for autoregressive models or the Moth stimuli would help.\n\nOverall, to broaden context and strengthen claims, I would recommend adding citations and, where feasible, experiments around: (a) Narratives by [Nastase et al., 2021](https://www.nature.com/articles/s41597-021-01033-3) as a widely used naturalistic fMRI resource, which would diversify stimuli and subjects; (b) [Pereira et al. (2018)](https://www.nature.com/articles/s41467-018-03068-4) as a complementary sentence-level dataset; (c) alternative representational similarity metrics (RSA/CKA/Procrustes) given active debate on metric choice; (d) concept-erasure baselines (INLP, RLACE, IGBP/single-projection) to test method-robustness of the causal effect; (e) Report exact effect sizes and confidence intervals for win-rate differences, not only p-values; (f) Clarify whether LM losses truly matched (distributions, not just means) and include masked-LM vs autoregressive comparability caveats; (g) Finally, consider a small positive-control experiment: brain-tuning the same base models (without adversarial reversal) to increase predictivity and checking for competence gains, to bracket the causal relationship in both directions (cf. recent brain-tuning gains in speech LMs). This would nicely support your core result.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pliY8SDkgH", "reviewer": "Reviewer_Fhx9", "summary": "The authors introduce a novel methodology to create \"brain-misaligned\" models by fine-tuning pretrained LLMs (BERT and GPT-2) with a dual objective: maintaining language modeling performance while actively training the model's representations to be poor predictors of human fMRI activity recorded during reading. This is achieved using an adversarial setup with a gradient reversal layer. As a control, \"brain-preserving\" models are trained using the same procedure but with permuted fMRI data, isolating the effect of genuine brain-stimulus alignment. The linguistic competence of these misaligned and preserving models is then comprehensively evaluated on over 200 downstream tasks. The central finding is that deliberately reducing brain alignment significantly impairs a model's performance on these linguistic tasks, particularly in the domains of semantics, syntax, and reasoning.", "questions": "na", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The creation of \"brain-misaligned\" models is a clever and powerful technique. Using a gradient reversal layer to explicitly penalize brain predictability, while simultaneously preserving language modeling ability, allows the authors to isolate the variable of interest in a way that correlational studies cannot. The \"brain preserving\" model serves as a well-thought-out control, accounting for potential confounding effects of the fine-tuning procedure itself.", "weaknesses": "1. The performance drop is clear and significant for BERT models, but the effect is weaker and only marginally significant for GPT-2 on the Harry Potter dataset. Furthermore, the authors note that for GPT-2 on the Moth Radio Hour dataset, \"results are not consistent due to the weaker effect of brain removal.\" This variability undermines the universality of the core claim. The paper would be much stronger if it could offer a hypothesis or analysis explaining why this discrepancy exists.\n\n2. The authors fail to conduct experiments on the Narratives dataset, which is much larger, contains more subjects, and has been widely adopted in many previous works. Additional experiments on this dataset will provide more reliable insights.\n\n3. The authors only investigate bert and gpt2, which are out-of-date models. Many recent studies in brain-language model alignment use llama models. I think choosing llama models are especially important when you want to investigate the performance of Brain misaligned models.\n\n4. The paper would be far more transparent and impactful if it also reported absolute performance metrics across task categories instead of \"win rate\". Without this, it is difficult to judge the practical importance of the observed effect.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)