Divergences between Language Models and Human Brains
Authors: Yuchen Zhou
Emmy Liu
Graham Neubig
Michael J. Tarr
Leila Wehbe
Venue: NeurIPS 2024 poster (2024)
Keywords: Natural Language Processing
NLP
Brain Imaging
Neuroimaging
Magnetoencephalography
MEG
Neuroscience
Cognitive Science
Interpretability
Deep Learning
Primary area: neuroscience_and_cognitive_science
TL;DR: Language models differ from human brains in social/emotional intelligence and physical commonsense. Fine-tuning language models on these domains improves their alignment with human understanding.

Abstract: Do machines and humans process language in similar ways? Recent research has hinted at the affirmative, showing that human neural activity can be effectively predicted using the internal representations of language models (LMs). Although such results are thought to reflect shared computational principles between LMs and human brains, there are also clear differences in how LMs and humans represent and use language. In this work, we systematically explore the divergences between human and machine language processing by examining the differences between LM representations and human brain responses to language as measured by Magnetoencephalography (MEG) across two datasets in which subjects read and listened to narrative stories. Using an LLM-based data-driven approach, we identify two domains that LMs do not capture well: social/emotional intelligence and physical commonsense. We validate these findings with human behavioral experiments and hypothesize that the gap is due to insufficient representations of social/emotional and physical knowledge in LMs. Our results show that fine-tuning LMs on these domains can improve their alignment with human brain responses.

Divergences between Language Models and Human Brains
Yuchen Zhou
Emmy Liu
Graham Neubig Michael J. Tarr
Leila Wehbe
Carnegie Mellon University (zhouyuchen, e gneubig, mi chaeltarz 1wehbe) edu
Abstract
Do machines and humans process language in similar ways? Recent research has hinted at the affirmative, showing that human neural activity can be effectively predicted using the internal representations of language models (LMs). Although such results are thought to reflect shared computational principles between LMs and human brains, there also clear differences in how LMs and humans rep- resent and use language. In this work, we systematically explore the divergences between human and machine language processing by examining the differences between LM representations and human brain responses to language as measured by Magnetoenccehhalorrpphy (MEG) across two datasets in which subjects read and listened to narrative stories. Using an LLM-based data-driven approach, we identify two domains that LMs do not capture well: social/emotional intelligence and physical commonsense. We validate these findings with human behavioral experiments and hypothesize that the gap due to insufficient representations social/emotional and physical knowledge in LMs. Our results show that fine-tuning LMS on these domains can improve their alignment with human brain responses)
1 Introduction
Language models (LMs) now demonstrate proficiency that may equal or even surpass human-level performance tasks including generating text [Boonn al. 2020a answering questions [Lewis) ct al. 2019]. translating languages (Costa-jussà et al. 20022. and even tasks that necessitate reasoning and inference DDaguupta et al.) 2022]. This has inspired researchers leverage LM representations to investigate and model the human brain's 's language system, positing that LMs may serve as reliable proxies for human linguistic processes [Abdou] 2022 Prior studies have found that human neural activity, as measured by neuroimaging techniques such as fMRI [Jain and Huth 2018 Toneva and Wehbe 2019]. EEG (Hale et al. 2018), MEG (Wehbe et al.) 2014a). and ECoG [Goldstein al. 2022 can effectively be predicted using representations from language models such as BERT [Devlin al. 2018 or GPT-2 [Radford et al., 2019). Robust neural prediction is hypothesized stem from the shared computational objective of both LMs and the human brain: predicting subsequent words based on prior context [Yamins and DiCarlo 2016 Schrimpf et 2021].
Despite the evident behavioral similarities, the extent to which LMs and human brains align func- tionally for language processing remains an open question. Essentially, the methods that LMs and humans use to acquire language are very different. LMs learn statistical regularities across massive sets of linguistic symbols, whereas humans rely on applying structured linguistic principles across relatively little input. Additionally, LMs that are confined to linguistic data are likely to fail to ground linguistic symbols in real-world contexts [Harnad] 1990 Bender and Koller 2020 Bisk et al.) 2020a]. Furthermore, the learning environments and goals of LMs and humans are markedly different. While humans communicate through active inquiry, expressing needs, directed communication, and
Data and code are available at: https://gitthu/ http::/hww.....aaaaaaaeaaaaaa/ee
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Language Model
LM Embeddings
Time Points
Channels
0.1 0.6 0.4
Ridge Regression
0.3 0.7
0.2
0.9
Hypothesis
Proposer
0.4
0.8
0.1
0.3
Texts
Per Word Error (MSE)
Responses
Time
Channels
0.2 0.6 0.3 0.4
Discretization
0,4 0.7 0.3 0.9
0.1 0.5 0.3
Figure 1: Schematic of our experimental approach. The LM takes as input the current word along with its preceding context produce the current word's LM embedding. This embedding is then used as input to ridge regression model to predict the human brain responses associated with the word. The Mean Squared Error (MSE) between the predicted and actual MEG responses calculated. Finally, an LLM-based hypothesis proposer is employed to formulate natural language hypotheses explaining the divergence between the predicted and actual MEG responses.
scaffolding conversations [Kuhl 2011], LMs are predominantly trained as passive recipients of raw text data. Consequently, LMs may struggle with comprehending social pragmatics and the nuances of words whose meanings fluctuate across different social contexts [Mahowald et al.
2023
Previous research exploring the relationship between human and LM language processing has typically focused on the types of linguistic features [Oota et al.) 2022a Sun et al. 2023]. neural 2021], or training and fine-tunin tine-tuning methods [Sun and Moens 2023] that may yield better predictions of brain responses. Diverging from this approach, Aw and Toneva 2023 proposed that the divergence between human and LM language processing might stem from LMs' inadequate understanding of texts. They supported this hypothesis by demonstrating that LMs fine-tuned on summarization tasks align more closely with human brain responses. Yet, this hypothesis is only one of many potential explanations. In this work, we adopt bottom-up, data-driven methodology to systematically investigate the differences between human and machine language processing, Our main contributions are follows:
network architectures [Schrimpf
1. In contrast to prior studies focusing the similarities between LMs and human brains, our research emphasizes their differences. We monitor the temporal progression of errors LM predictions ((2).
word-by-word basis two datasets with distinct language input modalities
2. Explaining the prediction errors for every word challenging due to the vast amount of text. Instead of manually formulating hypotheses, we adopt LLM-based method that automatically proposes natural language hypotheses to explain the divergent responses between human brains and language models (c3). The top candidate explanations are related to social/emotional intelligence and physical commonsense (44). We validate these hypotheses with human behavioral experiments.
3. We present evidence that fine-tuning LMs on tasks related to the two identified phenomena can align them more closely with human brain responses. This implies that the observed divergences between LMs and human brains may stem from LMs' inadequate representation these specific types of knowledge (55)
2 Predictive MEG Model
2.1 Data Preparation and Preprocessing
While many studies investigating the correlation between brain responses and language models utilize fMRI recordings (e.g., (Caucheteux al., 2023 Jain et al., 2020)), ), the comparatively low temporal resolution of fMRI hinders its ability to accurately capture the processing of individual words. To

---

to 25 ms
23 to
50t 73 mr
73 to 200 TM
100 125
323 153 ms
150 177 mO
179 ITS
200 to 223 TM
225 250 ma
0.5
0.4
230 100lo21
275 30.00
200 215
525 ta 350
350
375 402
420 4223
423 450
450
475 500
0.1
0.0
Figure 2: Pearson correlation of actual MEG responses with predicted responses using embeddings from layer 7 of GPT-2 XI on the Harry Potter dataset. The displayed layout is a flattened repre- sentation of the helmet-shaped sensor array. Deeper reds indicate more accurate LM predictions. Language regions are well predicted in language processing time windows (refer to 2.4 for more details).
address this limitation, our research employed MEG data. We strategically used two different MEG datasets, each with distinct input modalities, to assess potential variations in the brain's 's response patterns under these conditions.
The first dataset [Wehbe et al. 2014a] has eight participants reading Chapter 9 of Harry Potter and the Sorcerer's Stone (5,176 16 words) and four participants reading Chapter 10 of the same book (4,475 words) Each word was exposed on a screen for a fixed duration of 500ms. MEG data were collected on an Elekta NeuroMag MEG with 306 channels at 102 cranial points, and sampled at a rate of 1 kHz. The acquired data underwent preprocessing procedures using the Signal Space Separation (SSS) method [Taulu al., 22044 and its temporal extension, tSSS (Taulu and Simola (0066. The signal was then time-locked with individual words and down-sampled into non-overlapping 25ms time bins. Given the typical low Signal-to-Noise Ratio (SNR) of MEG, we adopted denoising technique (Ravishankar et al. 2021] that takes advantage of cross-subject correspondences to get an aggregated, denoised version of MEG responses (refer to Appppedii Affor more details).
To enhance reproducibility and generalizability of our study, we additionally collected MEG data from one participant who listened six narratives (1 (11,626 words) from The Moth, a platform featuring personal storytelling. These stories were chosen from the stimuli used a published story listening fMRI dataset [LeBel et al. 2023]. Five of these stories were repeated twice, while story was repeated five times. The data acquisition was performed using a MEGIN scanner equipped with 306 channels at 102 cranial points. The preprocessing pipeline
similar to that applied to the first dataset. Given that all story repetitions were from the same participant, we averaged the MEG responses for each story's ' repetitions to enhance SNR without using an alignment method.
2.2 Predicting MEG Responses from LM Embeddings
substantial number of recent studies exploring the correlation between brain responses and LMs have employed GPT-2 Pasquiou et al. 2022 Caucheteux et al. 2022 2023 Toneva et al., 2022]. To ensure consistency and comparability with these studies, we utilized the pre-trained GPT-2 XL model with 1.5B parameters, sourced from HuggingFace' 's transformers library [Wolf et al. 2020a as the backbone language model. Following previous work [Toneva and Wehbe |ehbe|[019], for every word w, we provided the model with context consisting of the preceding 19 words. We used the output of hidden layers of the LM, subsequently referred to as LM embeddings, to predict the MEG responses associated with each word (Figure T). For comparison, we also replicated some analyses on Llama-2
7B [oovron et al. 2023a (refer to Appendix D for more details).
Building upon established research that demonstrates the capability LM embeddings to linearly predict MEG responses [Wehbe et al. 2014a| Jain and Huth 2018, Caucheteux and King 2022a] we utilized a linear ridge regression model as the encoding model. Considering the time-co correlated nature of MEG data, it was essential to maintain the temporal structure when partitioning the data for training and testing purposes (Yang et al., 2019]. Therefore, we implemented a 10-fold cross- validation procedure that splits the MBG data into MEG responses as M and LM embeddings as L. For split i, we set aside one fold as the test set (Mi,test T,i.test) and fitted a ridge regression model with weight matrix W and bias D' using the remaining folds, denoted as (M (Minn (.tran)). The regularization parameters were chosen via
continuous chunks. We denote the actual
3

---

Ulama-2
Tinslin
40
300
Figure Pearson correlation between actual MEG responses and predicted responses from (A) GPT-2 XI. and (B) Llama-2 across LM layers and time after word onset on the Harry Potter dataset. Both models exhibit high correlations in early and intermediate layers at around 200ms. Correlation is computed across words and averaged across MEG channels.
nested cross-validatinn Following model training, we applied the trained weight matrix and bias to predict the brain responses from the LM outputs for the test set:
Mitest L "'ttstt" +
Finally, the test predictions from all folds concatenated to form the comprehensive prediction of MEG responses from the LM:
M concat (oncat((fiittet)
This process is performed for each of the different time windows relative to word onset.
2.3 Best Language Model Layer for Predicting MEG Responses
Prior research has shown that intermediate layers of language models often best predict human brain responses [Toneva and Wehbe 2019 Jain and Huth 2018 Oota et al. 2022b). Therefore, we selected the layer that best predicts brain responses. FFiuuee illustrates the Pearson correlation between actual MEG responses and those predicted by LM embeddings across layers and time points relative to word onset. We used the average correlation across all words and time windows as the metric to select the best layer. Echoing previous findings, confirmed that intermediate layers exhibit higher correlations, with layer being the best at predicting brain responses in GPT-2 XL. Similarly, for Llama-2, layer 3 was identified the most predictive.
2.4 Spatio-temporal Pattern of Predictions
As sanity check, we examined if the predictive model can effectively predict the brain areas and time course language processing. These areas include the inferior frontal gyrus, superior temporal gyrus, certain sections of the middle temporal gyrus, and angular gyrus [Blank et al. [016 Rogalsky
al. 2015 Sahin et 2009 Brennan and Pylkkänen 2012] Friederici 2002 Visser al. 2010
Rogalsky and Hickok
2009].
As shown in Figure we observe temporal progression of accurately predicted areas after word onset. The prediction performance peaks first in the occipital lobe between 75-100ms. Given that LM embeddings encode information (c.g.. word frequency) correlated to the number of letters in a word and MEG is sensitive to abrupt changes in visual inputs, we attribute this early peak to the initial visual perception of a word. This is followed by heightened prediction performance in the bilateral temporal lobe between 175-250ms, when we expect semantic processing to start. This observation aligns with previous research indicating that most language experiments with naturalistic stimuli reveal bilateral language representations [Wehbe et al. 2014b Huth et al. 2016 Deniz et al.) 2019 Toneva et al. 2022]. Finally, between 250-375ms, the anterior temporal lobe and frontal lobe show increased prediction performance, which is likely related to further semantic processing. This sequential pattern of prediction performance replicates the spatio-temporal dynamics of language
processing found in previous literature [Wehbe et al. 2014a Toneva et al. 2022

---

Table 1:
Top 10 hypotheses generated from the best layer of GPT-2 XL on the Harry Potter dataset
Hypothesis
Validity
p-value
have high level of emotional intensity
0.250
0.010
involve complex sentence structures or grammar include emotional language descriptions
0.250
0.015
0.238
0.008
have high level of tension conflict
0.237
0,023
have characters using body language or non-verbal cues
0.225
0,032
are emotionally charged, making it challenging for language
0.213
0.020
models to accurately interpret the intended tone or sentiment include conflicts between characters
0,200
0.035
have characters interacting with their environment
0.188
0,059
have complex sentence structures have dialogue between characters with varying emotions
0.175
0.081
0.175
0.022
Table 2: Top 10 hypotheses generated from the best layer of GPT-2 XL the Moth dataset
Hypothesis
Validity
p-value
contain elements of fiction or exaggeration
0,212
0.012
feature emotional or dramatic language refer to cultural societal norms
0.150
0.090
0.138
0.107
include sensory details imagery have strong emotional dramatic content
0.137
0.107
0.100
0.173
show a lack of coherence or logical flow contain elements of surprise and unpredictability
0.100
0.111
0.094
0.201
contain emotional, personal narratives idiomatic expressions figurative language refer to specific events incidents
0.088
0,201
0.088
0.178
0.087
0.237
3 Identifying Phenomena of Interest
Our objective is to investigate the elements of MEG responses that cannot be well explained by the LM. We work with an average of cleaned MEG responses from group of subjects and multiple trials, which illustrate the common elements of language processing across individuals. Therefore, for words where MEG responses are well predicted, it is likely that this marks a genuine divergence between human brains and the LM. It is important to clarify that our approach trains an encoding model to predict human brain responses based on language model outputs, rather than the reverse. This means our methodology identifies information that is captured by MEG but is not present in the language model, rather than information captured by the language model but is not present in MEG responses.
Leveraging the high temporal resolution of MEG, computed the Mean Squared Errors (MSEs) between actual and predicted MEG responses for each individual word on channels that demonstrated statistically significant correlations (one-sided, p=0.001). For word w,
MSE(w)
(M)(w) - M(w)))²
(I)
S
iES
where S is the set of significant channels.
3.1 Automatically Discovering Differences between Brain Responses and LM Predictions
Given the vast amount of text, manual pattern discovery becomes challenging. Figure 7 presents sample sentences color-coded based prediction error, illustrating the challenges in formulating hypotheses from observations.
To discover subtle differences between MEG responses and LM predictions, used method that automatically describes differences between text corpora using proposer and verifier LMs [Zhong
al. 2023
This system consists of first prompting LLM (GPT-3; Brown et al. 2020bb with
5

---

have alogus butween characters with varying amorions
Divergent Senterse
Choca
Daual
Consergett Serietce
enchonally to socurbtely interpret the intonded hone sontiment
challnging language model
is level amotionna
neve high leve of
confiice
indldd ensconal anguuge of deac pulens
includee
have compleer tenteence thructures that are diftc ar for languege
grammar to anatice
Inveive complex diffcult for languagg mosds comprchend
using brryy language
0.000
0.323
3,507
1.000
Peroelage of Resposses
have locus on action of wvents
indlude mements danger pent
Equal
convergerc sercence
reseengs
contain deccript ens phystcal arpeorance characterictes
langiege mideles to delect
inckude slnng or colloquae sore that not commony frund
kinguage
involve elemeents of meg or fantasy
outzids of bing medels trannee vicbbubry
Includo diologue or do dblect thit ard not eapil rspresenczd
have high hequeecy proper nouns and namen
percentage
Figure 4: Distribution of human responses for (A) the top 10 and (B) the bottom 10 hypotheses, ranked by the percentage of 'Divergent Sentence' responses.
number of samples from two corpora (Do, D2) to generate many hypotheses on how the first corpus differs from the second, and then using fine-tuned validator model (FLAN-T5-XXL: Chung et al. (2022)) to validate how often each proposed hypothesis is true based on pairs from each corpus sampled from held-out set. Specifically, the verifier presented with prompt containing two sentences from Do and D1, and asked whether or not the hypothesis is true, and this is repeated across the development set for each hypothesis. For the exact prompts used in the proposer and verifier, please refer to Appendix E, Sentences were then ranked based on their mean MSE. The top 100 ceast-predicted sentences constituted the DO set, while the 100 well predicedd sentences constituted the D1 set. This process of hypothesis proposal and verification was repeated across cross-validation folds.
3.2 Proposed Hypotheses
The top ten hypotheses from the Harry Potter dataset ranked by validity are listed Table We identified two primary differences between the language model and the human brain: firstly, the processing of social and emotional information, and secondly, the capacity for interaction with the surrounding environment. These are henceforth referred to as social/emotional intelligence and physical commonsense, respectively. Importantly, these hypotheses resonate with conclusions drawn in prior research, as detailed in 444 Similarly, we ran the hypothesis proposer on the Moth dataset. This replication produced slightly varied but fundamentally similar topics to those discovered the Harry Potter dataset (Table 2). This congruence across datasets with different input modalities aligns with previous research showing that after initial sensory processing, the brain's language processing is consistent across reading and listening [Deniz al. 2019 Chen et al. 2023
VValidity measures the difference in certainty that the hypothesis is true between the two corpora, see Zhong 2023 for more details.

---

3.3 Manual Hypothesis Verification
We conducted an experiment involving human participants for additional validation of our hypotheses.
We gathered data from 10 participants using Qualtrics, resulting in a collection of 1,400 trials. In each trial, participants were presented with hypothesis selected either from the top 10 or bottom 10 hypotheses generated from the Harry Potter dataset, along with pair of sentences - one from DO and the other from D1 - in a randomized order. The task for participants was to determine which sentence aligned more closely with the given hypothesis, choosing between "More True for Sentence A", "More True for Sentence B", or "Equally true".
The response distribution for each hypothesis is shown in Figure 4
Note that a given hypothesis is not expected apply all divergent sentences (e.g., it might suffice for sentence be emotionally intense or grammatically complex to be divergent). If hypothesis does not align with sentence from the divergent set. participants should show no preference between the two sentences presented. Chi-square analysis revealed statistically significant differences in the distribution of responses between the top and bottom hypotheses (p= 0.024). preference towards divergent sentences was observed in the top hypotheses condition while preference towards "Fqually True" was observed in the bottom hypothesis condition. This pattern can be attributed to the role of the proposer, which was instructed to generate hypotheses that effectively distinguish Do (comprising divergent sentences) from Dj. As a result, the bottom hypotheses tend to be those that fail to differentiate between Do and D rather than those that are more explanatory of D compared to Do (refer to Appendiix F for more details).
4 Selected Phenomena
Comprehending social/emotional and physical commonsense requires humans use a broad spectrum of contextual knowledge. We briefly discuss the insights and challenges highlighted in the existing neuropsychological and NLP literature regarding these domains.
Human social and emotional intelligence requires both introspection and predicting the feelings of others (Salovey and Mayer 1990]. Neuropsychological research on social cognition has identified a network of brain regions that support understanding other people's ' intentions, actions, and emotions [Saxe et al.] 2006]. Crucially, emotions are intrinsic to the human experience and pervasively interact with other mental facilities, including language (Satpute and Lindquist, 2021]. I. As such, creating agents with social and emotional intelligence has been a longstanding goal of NLP [Gunning] 2018
Paiva al. 202T]. However, LLMs still fall short of human abilities for inferring the mental states and emotions of others ("theory-of-mi minddd" tasks) [Sap et al. 2022].
Physical commonsense refers to knowledge about the physical properties of everyday objects and physical phenomena [Forbes et al. 2019 Bisk et al. 20200b. From a neuropsychological perspective, language is not the primary means through which humans acquire commonsense physical knowledge. Instead, humans rely on sensory inputs and interactions with their environment [Baillargeon] [994]. Notably, the category of a physical object affects which brain regions are recruited when interacting with that object. For example, interacting with people activates the theory of mind areas [Saxc et al. 2006), the visual face areas [Sergent al. 1992 Kanwisher al. 1997]. and body areas [Downing et al. 2001], interacting with corridors while navigating recruits the visual place [Epstein and Kanwisher 1998 and spatial navigation areas, and interacting with tools recruits the dorsal object-processing stream [Almeida et al., 2010). Interestingly, reading about these domains has also been found to recruit these same visual regions [Wehbe al. 2014bb Huth al. 2016). Given how physical commonsense knowledge is acquired, it is not surprising that, within NLP, this domain poses challenge for language models. While these models can potentially learn representations capturing specific physical properties of the world, such as an object's ''s color or game board's 's state [Abdou] ct al., | Li et al. 2023). remains unclear whether purely text-based representations can capture the richness and complexity of physical commonsense as exhibited by humans [Fobee et al. 2019 Bisk et al. 2020b
5 Improving Brain Alignment via Fine-tuning
We hypothesize that the inability of language models to accurately predict associated brain responses stems from their inadequate representations of social/emotional understanding and physical world
7

---

A
1.0
0.6
004
30.2
150
275
300
350
425
475
B
Time (ms)
1.0
0.8
0.6
0.4
non significant
0.2
0.0
25
75
100
150
200
225
250 275 Time [ms)]
300
350
375
400
425
475
Figure 5: Performance comparison of the base model with models fine-tuned (A) social and (B) physical datasets. Each panel's y-axis shows the percentage channels the fine-tuned model with better, worse, non-significantly different performance (measured by Pearson correlation) compared to the base model. Fine-tuned models outperform the base model during language processing time windows, Refer to Appendix for detailed view of each MEG channel plotted.
knowledge. Drawing inspiration from Aw and Toneva 2023], we fine-tuned the GPT-2 XL model on datasets specific to the two phenomena examine if targeted fine-tuning could enhance the model's alignment with brain activity.
Furthermore, examined whether domain-specine fine-tuning would specifically bolster the model's capability in predicting MEG responses associated with words from that domain, as compared to words outside that domain. To this end, recruited three raters annotate Chapter 9 of Harry Potter across the two domains. We release these annotations as a resource for the dataset to facilitate further analysis. Details on the annotation process be found in An/ppeddix H| Examples of each phenomenon within the Harry Potter text can be found in Appendix
5.1 Datasets
Social/Emotional Intelligence We study social and emotional intelligence using the Social IQa dataset (Sap al. 2019]. This dataset contains questions about people' 's feelings and their social implications.
Physical Commonsense We study physical commonsense using the PiQA dataset [Bisk et al. 2020b This dataset contains goal-driven questions based on everyday situations. These questions were taken from the website instructables.com com, where people share DIY project instructions.
We also provide examples from each dataset in Table 3
Table 3: Datasets for Fine-
Fine-Tuning Tuming with Sample Questions and Answers (Correct Answer in Bold)
Dataset
Type
Num train
Options
Sample question
Sample answers
Social IQa Social/Emotion
33.4k
3
Sydney had so much pent up emotion, they burst into at work. How would Sydney feel afterwards?
1. affected
2. like they released their tension
worse
PiQA
Physical
16.1k
When boiling butter. when it's ready, you can
1. Pour it onto a plate
2. Pour into jar

---

5.2 Fine-tuning Setup
In order to keep the architecture of fine-tuned models consistent with the base model, we format the multiple choice task as N language modeling tasks, where N is the number of options. Specifically, for the combined context and question z, we directly concatenate each possible multiple-choiee answer (31:-.. ..N to to form N different sentences. After passing the concatenated sequences through the model, we sum the logits of all tokens corresponding to each multiple-choice option to obtain score proportional to its oggliikllhood. These scores are then gathered into size (1, N) tensor, and cross-entropy loss relative to the correct multiple choice answer is used to train the model. Further details on the fine-tuning setup can be found in Appendix G
5.3 Comparing Fine-tuned Models with the Base Model
We evaluated the fine-tuned GPT-2 XL model on the Harry Potter dataset. To identify channels with statistically significant differences between the base and fine-tuned model, we calculated empirical p-values by comparing the true correlation value with 10,000 simulated ones obtained by permuting the brain data. Details of the algorithm can be found in Appendixx J
Fine-tuned models are better aligned with the brain on both tasks. As illustrated in Figure 5/A, the model fine-tuned on the social dataset exceeds the base model in performance across the majority of channels within the 50ms to 300ms time interval post word onset. Notably, this interval corresponds to the language processing time windows, as identified in $2.4 In a similar vein, the fine-tuned physical model exceeds the base model's performance in almost all channels during the 50-275ms interval post word onset (Figure 5).. However, interestingly, almost all channels are worse than the base model outside this time window. This time selectivity may indicate that the improvements of the fine-tuned model are tailored towards linguistic comprehension rather than broader brain functionalities.
Fine-tuning improves alignment more for words annotated with that category. We compared the reduction in prediction error for words annotated within each category and words outside each category by computing the difference in MSE between the model fine-tuned on the corresponding task and the base model. As demonstrated in Figure 6/A prediction errors for social words exhibit significant reduction compared to non-social words 200-275ms post word onset. Additionally, there significant improvement in MSE for physical words over non-physical words 150-225ms post word onset (Figure 6)3). We also ran additional control experiments to check if MSE improvement is specific to words that match the category of the dataset on which the model was fine-tuned. Specifically, we evaluated the prediction improvement of physical words on the model fine-tuned on the social dataset, and vice versa (Appendix L).
Improvements are not related to increased language-modeling ability. Prior work has found that LMs with lower perplexity can better predict brain activity (Schrimpf et al.) 2021). Therefore, additional finettuning may have improved the language model's ability to perform the LM task in general, leading to improved alignment. To rule out this possibility, performed 3-fold cross- validation on Harry Potter and the Sorcerer's Stone, excluding Chapters 9 and 10, which were used as data in this study. We trained the base model, well the fine-tuned models, the train set in each fold with the language modeling objective, and found that the final average losses on the test sets were similar (See Appendix Mor details).
6 Related Work
Numerous studies have found that LM hidden states can linearly map onto human brain responses to
speech and text measured by MEG, EEG, and fMRI (Wehbe et al. 2014a Hale et al. 2018 Jain and
Huth 2018 Abnar et al. 2019 Jat et al.
2019 Gauthier and Levy 2019 Toneva and Wehbe 2019
Caucheteux and King 2022a Jain al.
2020 Toneva al. 2022
Aw and Toneva 2022
At a more foundational level, studies have identified shared computational principles between LMs and human brains. Evidence suggests that both human brains and LMs are perpetually engaged in predicting the subsequent word [Schrimpf et al. 2021]. LM surprisal is found to be positively corre- lated with brain activation, reaching its peak approximately 400 ms post word onset [Goldstein et al.) 2022]. This aligns well with N400, which denotes a decline in brain activation upon encountering
unexpected words around 400 ms after word onset (Lau et al.] 2009 Parviz et al. 2011 Halgren et al.

---

B
dhys non-phyyccal
0,002
VSS intrrveeeen
2.005
0.001
0.004
0.003
MSE
0.002
social
0.007
-0
100
200
son
600
100
200
300
100
Time (mma)
Timme [mss]
Figure Comparison of improved MSE between (A) social and (B) physical words and those outside each category evaluated on models fine-tuned on corresponding datasets. Positive values denote lower MSEs in the fine-tuned model. Shaded region indicates standard error. Asterisks denote time points with significant differences between the two groups (Student's ' t-test with FDR correction, (orrection.-00005) p=0.55..
2002 Moreover, LM representations can predict the hierarchy of brain responses [Caucheteux and
King 2022b Caucheteux al. 2023]. Despite this, Antonello and Huth [022] have pointed out that
high correlation between brain activity and LMs does not necessarily imply that they operate under similar computational principles.
We not only observe this LM-brain alignment but can also actively intervene in it. Research has demonstrated that the alignment between LMs and human brains can be improved by task-specific fine-tuning. A notable instance the study by [Schwartz et al. (2019). where the fine-tuning BERT using both IMRI and MEG signals enhanced its ability to predict fMRI responses. Importantly, this improvement was not participant-specific and could be transferred to hold-out individuals. Another study [Aw and Toneva rization, also facilitated better alignment with brain activity. Furthermore, altering the architecture of BERT such that it aligns better with the brain improves performance on downstream NLP tasks [Toneva and Wehbe 2019]. These findings suggest potentially symbiotic relationship between enhancing task performance in LMs and boosting their alignment with brain responses.
2023 showed that task-oriented fine-tuning, particularly for narrative summa-
7 Conclusions, Limitations, and Future Work
We explore a critical question connecting language models with human neural activity: How do LMs differ from human brains processing language? We employed an LLM-based approach to automatically propose hypotheses explaining the elements of human brain responses that cannot be well explained by language models. Social/emotional intelligence and physical commonsense emerged as the two dominant themes. After fine-tuning GPT-2 XI. on datasets related to these themes, we observed an improved alignment between LM predictions and human brain responses.
Limited by the availability of datasets with aligned brain data, our study was conducted on a relatively narrow range of texts. While we observed consistent patterns across two language modalities, is important to note that both datasets utilized were exclusively narrative stories. This limited scope raises the possibility that additional, undetected divergences exist, potentially obscured by the quantity of text and the methodology employed for hypothesis generation from the sentences. By developing more robust tools for pattern discovery and incorporating wider array of textual materials, our approach can be adapted to more comprehensively address the question in future studies.
Acknowledgments and Disclosure of Funding
This work supported by the James S. McDonnell Foundation. We also acknowledge the support of the National Sciences and Engineering Research Council of Canada (NSERC) through post- graduate fellowship given to EL (PGSD). This work was also partially supported by the National Institute On Deafness And Other Communication Disorders of the National Institutes of Health under Award Number RO1 DC020088. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
10

---

References
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 2020a
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence- ceriicett--qquueee to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13611, 2019.
Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. No language left behind: Scaling human-centered machine translation. arXiv preprint NKiv:2207.04672, 2022.
Ishita Dasgupta, Andrew K Lampinen, Stephanie CY Chan, Antonia Creswell, Dharshan Kumaran, James L McClelland, and Felix Hill. Language models show human-like content effeets on reasoning. arXiv preprint arXiv:2207.0705/1 2022.
Mostafa Abdou. Connecting neural response measurements & computational models language: a non-comprehensive guide. arXiv preprint arXiv:2203.05300 2022.
Shailee Jain and Alexander Huth. Incorporating context into language encoding models for fmri. Advances in neural information processing systems, 31, 2018.
Mariya Toneva and Leila Wehbe. Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain). Advances in neural information processing systems, 32, 2019.
John Hale, Chris Dyer, Adhiguna Kuncoro, and Jonathan Brennan. Finding syntax in human encephalography with beam search. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages
2018
Leila Wehbe, Ashish Vaswani, Kevin Knight, and Tom Mitchell. Aligning context- based statistical models of language with brain activity during reading. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2014a.
Ariel Goldstein, Zaid Zada, Eliav Buchnik, Mariano Schain, Amy Price, Bobbi Aubrey, Samuel Nastase, Amir Feder, Dotan Emanuel, Alon Cohen, et al. Shared computational principles for language processing in humans and deep language models. Nature neuroscience, 25(3)336--380 2022.
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9 2019.
Daniel L K Yamins and James J DiCarlo. Using goal-driven deep learning models to understand sensory cortex. Nat Neurosci, 39633356-335, 2016.
Martin Schrimpf, Idan Asher Blank, Greta Tuckute, Carina Kauf, Eghbal A Hosseini, Nancy Kan- wisher, Joshua Tenenbaum, and Evelina Fedorenko. The neural architecture of language: Integrative modeling converges on predictive processing. Proceedings of the National Academy of Sciences, 1 :18(45):c210664611 2021.
Stevan Hamadd. The symbol grounding problem. Physica D: Nonlinear Phenomena, 42(13)::333 1990.
Emily M Bender and Alexander Koller. Climbing towards nlu: On meaning, form, and understanding in the age of data. In Proceedings of the 58th annual meeting of the asxociation for computational linguistics, pages 5185-5198, 2020.
11

---

Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, Nicolas Pinto, and Joseph Turian. Experience grounds language, 2020a.
Patricia K Kuhl. Early language learning and literacy: Neuroscience implications for education. Mind, brain, and education, 5(3):128-144 142, 2011.
Kyle Mahowald, Anna A. Ivanova, Idan A. Blank, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. Dissociating language and thought in large language models: cognitive perspective, 2023.
Subba Reddy Oota, Jashn Arora, Veeral Agarwal, Mounika Marreddy, Manish Gupta, and Bapi Raju Surampudi. Neural language taskonomy: Which nlp tasks are the most predictive of fmni brain activity? arXiv preprint arXiv:2005.01404 2022a.
Jingyuan Sun, Xiaohan Zhang, and human brain and artificial supervised representations of language. arXiv preprint arXiv:2310.04460, 2023.
Moens. Tuning in neural encoding: Linking
Jingyuan Sun and Marie-Francine Moens. Fine-tuned vs. prompt-tuned supervised representations: Which better account for brain language representations? arXiv preprint arXiv:2310.01854 2023.
Khai Loong Aw and Mariya Toneva. Training language models to summarize narratives improves brain alignment. In The Eleventh International Conference on Learning Representations, 2023.
Charlotte Caucheteux, Alexandre Gramfort, and Jean-Rémi King. Evidence of predictive coding hierarchy in the human brain listening to speech. Nature Human Behaviour, pages 1-12, 2023.
Shailee Jain, Vy Vo, Shivangi Mahto, Amanda LeBel, Javier S Turek, and Alexander Huth. In- terpretable multi-timescaee models for predicting fmri responses continuous natural speech. Advances in Neural Information Processing Systems, 33:13738-13744 2020.
Samu Taulu, Matti Kajola, and Juha Simola. Suppression of interference and artifacts by the signal space separation method. Brain topography, 16(4):269-27) 2004.
Samu Taulu and Juha Simola. Spatiotemporal signal space separation method for rejecting nearby interference in MEG measurements. Physics in medicine and biology, 51(7):1759, 2006.
Srinivas Ravishankar, Mariya Toneva, and Leila Wehbe. Single-trial meg data can be denoised through cross-subject predictive modeling. Frontiers in Computational Neuroscience, 15:737324, 2021.
Amanda LeBel, Lauren Wagner, Shailee Jain, Aneesh Adhikari-Desai, Bhavin Gupta, Allyson Morgenthal, Jerry Tang, Lixiang Xu, and Alexander G Huth. A natural language fmri dataset for voxelwise encoding models. Scientific Data, 10(1):555, 2023.
Alexandre Pasquiou, Yair Lakretz, John Hale, Bertrand Thirion, and Christophe Pallier. Neu- ral language models are not born equal to fit brain data, but training helps. arXiv preprint arXiv:2207.03380, 2022.
Charlotte Caucheteux, Alexandre Gramfort, and Jean-Rémi King. Deep language algorithms predict semantic comprehension from brain activity. Scientific reports, 12(1):16327, 2022.
Mariya Toneva, Tom M Mitchell, and Leila Wehbe. Combining computational controls with natural text reveals aspects of meaning composition. Nature Computational Science, 2(11):745-757, 2022.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State-of-the-aa natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pages 38 45, 2020a.
12

---

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cris- tian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, lliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023a.
Charlotte Caucheteux and Jean-Rémi King. Brains and algorithms partially converge in natural language processing. Communications biology, 5(1):1-10, 2022a.
Yuxiao Yang, Omid G Sani, Edward F Chang, and Maryam M Shanechi. Dynamic network modeling and dimensionality reduction for human ecog activity. Journal of neural engineering, 16(5); 056014, 2019.
Subba Reddy Oota, Manish Gupta, and Mariya Toneva. Joint processing of linguistic properties in brains and language models. arXiv preprint arXiv:2212.08094, 2022b.
Idan Blank, Zuzanna Balewski, Kyle Mahowald, and Evelina Fedorenko. Syntactic processing is distributed across the language system. Neuroimage, 127:307-323, 2016.
Corianne Rogalsky, Diogo Almeida, Jon Sprouse, and Gregory Hickok. Sentence processing selectiv- ity in broca's s area: evident for structure but not syntactic movement. Language, cognition and neuroscience, 30(10): - 2015.
Ned T Sahin, Steven Pinker, Sydney Cash, Donald Schomer, and Eric Halgren. Sequential processing of lexical, grammatical, and phonological information within broca's area. Science, 326(5951):4455 449, 2009.
Jonathan Brennan and Liina Pylkkänen. The time-course and spatial distribution of brain activity associated with sentence processing. Neuroimage, 139-111481 2012.
Angela D Friederici. Towards a neural basis of auditory sentence processing. Trends in cognitive sciences, 6(2):78-84, 2002.
Maya Visser, Elizabeth Jefferies, and MA Lambon Ralph. Semantic processing in the anterior temporal lobes: a meta-analysis of the functional neuroimaging literature. Journal of cognitive neuroscience, 22(6): 2010.
Corianne Rogalsky and Gregory Hickok. Selective attention to semantic and syntactic features modulates sentence processing networks in anterior temporal cortex. Cerebral Cortex, 19(4): 786-796, 2009.
Leila Wehbe, Brian Murphy, Partha Talukdar, Alona Fyshe, Aaditya Ramdas, and Tom Mitchell. Simultaneously uncovering the patterss of brain regions involved in different story reading subpro- cesses. PLOS ONE, ((11):e112555, 2014b.
Alexander G Huth, Wendy A de Heer, Thomas L Griffiths, Frédéric E Theunissen, Jack L Gallant, Wendy a De Heer, Thomas L Griffiths, and Jack L Gallant. Natural speech reveals the semantic maps that tile human cerebral cortex. Nature, 532(7600):453-44 458, 2016. doi: 10.1038/nature17633
Natural.
Fatma Deniz, Anwar O Nunez-Elizalde, Alexander G Huth, and Jack L Gallant. The representation of semantic information across human cerebral cortex during listening versus reading is invariant to stimulus modality. Journal of Neuroscience, 39(39);772--7336, 2019.
Ruiqi Zhong, Peter Zhang, Steve Li, Jinwoo Ahn, Dan Klein, and Jacob Steinhardt. Goal driven discovery of distributional differences via language descriptions, 2023.
13

---

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhimi Agarwal, Ariel WerberrtVosss.. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020b.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. Scaling cerruucttoonfinetuee language models, 2022.
Catherine Chen, Tom Dupré la Tour, Jack Gallant, Dan Klein, and Fatma Deniz. The cortical representation of language timescales is shared between reading and listening. bioRxiv, pages 2023-01, 2023.
Peter Salovey and John D. Mayer. Emotional intelligence. Imagination, Cognition and Personality, (3):185-211, 1990.
R. Saxe et al. Uniquely human social cognition. Current opinion neurobiology, 16(2)233522) 2006.
Ajay B. Satpute and Kristen A. Lindquist. At the Neural Intersection Between Language and Emotion. Affective Science, 2(2):207-220, 2021.
David Gunning. Machine common sense concept paper, 201 8.
Ana Paiva, Filipa Correia, Raquel Oliveira, Fernando Santos, and Patrícia Arriaga. Empathy and Prosociality in Social Agents, page 385-432. Association for Computing Machinery, New York, NY, USA, edition, 2021.
Maarten Sap, Ronan Le Bras, Daniel Fried, and Yejin Choi. Neural theory-of-mind? on the limits of social intelligence in large LMs. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3762-3788 Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
Maxwell Forbes, Ari Holtzman, and Yejin Choi. Do neural language representations learn physical commonsense?, 2019.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020b.
Renée Baillargeon. How do infants learn about the physical world? Current Directions in Psycholog- ical Science, 3(5): 140, 1994.
J Sergent, S Ohta, and MacDonald. Functional neuroanatomy of face and object processing:
positron emission tomography study. Brain, 11::15-36, 1992.
Kanwisher, McDermott, and M Chun. The fusiform face area: a module in human extrastriate cortex specialized for face perception. Jourmal of Neuroscience, 17(11) 17(11):302-4311 1997.
E. Downing, Jiang, M Shuman, and N Kanwisher. cortical area selective for visual processing of the human body. Science, 293(5539):247--244 2001.
Epstein and Kanwisher. cortical representation the local visual environment. Nature, 392 (6676):558-66)) -611 1998.
Jorge Almeida, Bradford Mahon, and Alfonso Caramazza. The role of the dorsal visual processing stream in tool identification. Psychological science, 2166)777--778 2010.
14

---

Mostafa Abdou, Artur Kulmizev, Daniel Hershcovich, Stella Frank, Ellie Pavlick, and Anders Søgaard. Can language models encode perceptual structure without grounding? a case study in color, 2021.
Kenneth Li, Aspen K. Hopkins, David Bau, Fernanda Viégas, Hanspeter Pfister, and Martin Watten- berg. Emergent world representations: Exploring a sequence model trained on synthetic task, 2023.
Maarten Sap. Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Common- sense reasoning about social interactions. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Hong Kong, China, November 2019.
Language Processing (EMNLP-IJCNLP)) pages Association for Computational Linguistics.
Samira Abnar, Lisa Beinborn, Rochelle Choenni, and Willem Zuidema. Blackbox meets blackbox: Representational similarity and stability analysis of neural language models and brains. arXiv preprint wrXiv:1906.015323 2019.
Sharmistha Jat, Hao Tang, Partha Talukdar, and Tom Mitchell. Relating simple sentence represen- tations in deep neural networks and the brain. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5137-5154, 2019.
Jon Gauthier and Roger Levy. Linking artificial and human neural representations of language. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 529-539, Hong Kong, China, November 2019, Association for Computational Linguistics.
Khai Loong Aw and Mariya Toneva. Training language models for deeper understanding improves brain alignment. arXiv preprini arXiv:2212. 2022.
Ellen Lau, Diogo Almeida, Paul C Hines, and David Poeppel. A lexical basis for n400 context effects: Evidence from meg. Brain and language, 111(3):161-17, 2009.
Mehdi Parviz, Mark Johnson, Blake Johnson, and Jon Brock. Using language models and latent semantic analysis to characterise the n400m neural response. In Proceedings of the australasian lunguage technology asxociation workshop 201 1, pages 38-46, 2011.
Eric Halgren, Rupali P Dhond, Natalie Christensen, Cyma Van Petten, Ksenija Marinkovic, Jeffrey D Lewine, and Anders M Dale. N400-like magnetoenceepallogaphy responses modulated by se- mantic context, word frequency, and lexical class in sentences. Neuroimage, (7(3):1001111) 2002.
Charlotte Caucheteux and Jean-Rémi King. Brains and algorithms partially converge in natural language processing. Communications biology, 5(1):134, 2022b.
Richard Antonello and Alexander Huth. Predictive coding just feature discovery? alternative account of why language models fit brain data. Neurobiology of Language, pages 1-16, 2022.
Dan Schwartz, Mariya Toneva, and Leila Wehbe. Inducing brain-relevant bias in natural language processing models. In Advances Neural Information Processing Systems, pages 14123-14133, 2019.
Tuhin Chakrabarty, Arkadiy Saakyan, Debanjan Ghosh, and Smaranda Muresan. Flute: Figurative language understanding through textual explanations, 2022.
Emmy Liu, Chenxuan Cui, Kenneth Zheng, and Graham Neubig. Testing the ability of language models to interpret figurative language. In Proceedings of the 2022 Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4437-4452, Seattle, United States, July 2022. Association for Computational Linguistics.
Verna Dankers, Christopher G Lucas, and Ivan Titov. Can Transformer be Too Compositional? Analysing Idiom Processing in Neural Machine Translation. Technical report.
15

---

Emmy Liu and Graham Neubig. Are representations built from the ground up? an empirical examination of local composition language models. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9053-9077 Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
Hugo Touyron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09280 2023b.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface's transformers: State-of-the-art natural language processing, 2020b.
Jeff Rasley, Samyam Rajbhandari. Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimiza- tions enable training deep learning models with over 100 billion parameters. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD '20, page 3505-3506, New York, NY, USA, 2020. Association for Computing Machinery.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
Yoav Benjamini and Yosef Hochberg. Controlling the false discovery rate: a practical and powerful approach multiple testing. Journal of the Royal statistical society: series (Methodological), 57(1):289-300, 1995.
16

---

A MEG Denoising
Because of the typical low Signal-to-None Ratio (SNR) associated with MEG, we adopted denoising technique Raaiihankar et al. 202T] that takes advantage of cross-subject correspondences to get an aggregated, denoised version of MEG responses. Specifically, this process involves modeling the MEG responses Mt of subject as linear function of the MEG responses M, from source subject
Mr-s Wte, bre-s
We estimated the target subject's MEG responses from all other subjects:
M,
Σ
Mies
8ES,oft
where is the set of subjects and N is the number of subjects. These individual estimates are then aggregated generate denoised version of MEG responses:
M
Σ Me
N
BES
B
Sample Sentences
Given the vast amount of text in the datasets, manually discovering patterns becomes challenging.
Figure 7 provides an illustrative example by presenting set of sample sentences that are color-coded based on the magnitude of their prediction error. This visualization demonstrates the complexity involved in formulating hypotheses from observations.
1. He had been looking forward to learning to fly more than anything else.
"Of course he has," said Ron, wheeling around.
3. But Neville, nervous and jumpy and frightened of being left on the ground, pushed off hard before the whistle had touched Madam Hooch's lips.
most divergent
least divergent
Figure 7: Sample sentences from the Harry Potter dataset, with colors indicating prediction error levels. Each of the five colors corresponds to 20- 20-percentile range of words from the entire dataset.
C Additional Results on GPT-2 XI
C.1 Results on Last Layer
addition to the best layer, we also performed analyses on the last layer the language model.
C.1.1 Spatial-Temporal Patterns of Predictions
The spatial-tempora pattern of predictions observed in the last layer Fiiure 8 is similar to that of the best layer, However, there is notable difference in the magnitude the values. Specifically, the maximum correlation in the last layer is lower, decreasing from 0.53 in the best layer to 0.43.
C.1.2 Proposed Hypotheses
We also generate hypotheses from the predictions of the last layer Table 4p. These hypotheses exhibit similarities with those derived from the best-performing layer, notably in their inclusion of emotions
17

---

to 25 ms
23 to
50t
73 200
200 125
323 mS
150 1017 mS
175 to ITS
200 to
22D
225 250
0.5
0.4
230 10
275
200 215
525
25012
375 i 420 13450
450
475 300
0.1
0.0
Figure 8: Pearson correlation of actual MEG responses with those predicted by LMM embedding from the last layer of GPT-2 XI. (evaluated on the test set). The displayed layout is flattened representation the helmet-shaped sensor array. Deeper reds indicate more accurate LM predictions. Language regions are effectively predicted in language processing time windows (refer to 22.4 for more details).
and social interactions. However, distinctive aspect of these hypotheses is their association with supernatural and magical elements. Additionally, we observe the emergence of figurative language, aligning with previous research that indicates language models underperform humans in both the
interpretation and generation of figurative language [Chakrabarty et al., 2022 Liu et al. 2022 and
the correct representation of idiomatic phrases [Dankers al. Liu and Neubig 2022
Table 4: Top 10 hypotheses generated from the last layer of GPT-2 XL on the Harry Potter dataset
Hypothesis
Validity
p-value
contain descriptions unusual settings or creatures
0.1750
0.0754
has a lot of dialogue, with characters speaking to each other contain figurative language
0.1719
0.0855
0.1367
0.1409
contain rhetorical questions exclamations
0.1125
0.1685
contains references to obscure facts or trivia, such as the longest game of Quidditch mentions the unknown or unexpected, such as an unknown creature or surprise announcement
0,1094
0.0627
0.1094
0.1856
contain references the emotions of characters
0.1062
0.1996
contain references the supernatural
0.0875
0.1827
mentions dangerous creatures events, such trolls and duels contain references to magic
0.0813
0.2383
0.0688
0.2550
D Replication on Llama-2 7B
Although GPT-2 is widely used language model in brain research, it's not the latest model in the field. Models with more parameters and advanced training methods could show different results. Therefore, we replicated some analyses on Llama-2 TTouvro et al., 2023b]. We used the implementation in the HuggingFace library [Wolf et 2020b] with 7B parameters. As Figure 3B shows, early layers exhibit high correlations, with layer 3 identified as having the highest correlation.
D.1 Spatio-Tempont Patterns of Predictions
The spatial-temporal pattern of predictions in layer of Llama-2 (Figure 9) closely resembles those found GPT-2 XI.. This similarity implies that both language models effectively capture the representations of words.
D.2 Proposed Hypotheses
Hypotheses from the predictions of layer of Llama-2 can be found in Table Interestingly, the focus of these hypotheses is primarily on physical objects and events. In comparison to the hypotheses produced by GPT-2 XI., there is notable absence of social and emotional aspects, suggesting that Llama-2 7B could have more advanced comprehension of social and emotional contexts.
18

---

to 25 ma
23 to
50t
73 200
200 125
323 133 ms
150 mS
179 12 ITS
200 to
223
225 250
0.5
0.4
230 1000
273
200 225
525 250
25012
375 ia
420 455
450
475 300
0.1
0.0
Figure 9: Pearson correlation of actual MEG responses with those predicted by LM embedding from the best layer (layer 3) of Llama-2 (evaluated on the test set). The displayed layout is flattened representation the helmet-shaped sensor array. Deeper reds indicate more accurate LM predictions. Language regions are effectively predicted in language processing time windows (refer to 2 for more details).
Table 5: Top 10 hypotheses generated from the best layer of Llama-2 on the Harry Potter dataset
Hypothesis
Validity
p-value
involve action or movement, such as running or tiptoeing
0.300
0.005
refer to specific events or actions, such as flying lesson or spell not working refer specific objects locations, such the front steps the trophy room describe physical actions or movements
0.237
0,029
0.237
0.013
0.175
0.081
discuss or descrihe dangerous or frightening situations include actions or physical movements
0.150
0.056
0.150
0.117
contain words phrases that specific to the wizarding world have of chaos disorder
0.140
0.130
0.137
0.040
have a high level of tension or suspense
0.137
0.128
include words or phrases that are specific to the wizarding world
0.127
0.152
E Proposer and Verifier Prompts
The prompt for the proposer is:
(A_ble b block]
The dataset includes two chapters from "Harry Potter and the Sorcerer's Stone". The two groups are generated based the difference between language model and human responses to these sentences. The Group A snippets sentences where language models and humans show divergent responses, while the Group B snippets sentences where language models and humans show similar responses.
am a literary analyst investigating the characteristics of words. My goal is to figure out which sentences induce different responses for language models and human responses.
Please write a list of hypotheses about the datapoints from Group A (listed by bullet points "-"). Each hypothesis should be formatted as a sentence fragment. Here are three examples.
"/example_hyyothesiis.
sexaampe_hhppoheesi 2)
"(example_hye "aanuple_hypohhess__33
Based on the two sentence groups (A and B) from the above, more sentences in Group A...
The prompt for the validator is:
Check whether the TEXT satisfies PROPERTY. Respond with Yes No. When uncertain, output No.
Now complete the following example-
input: PROPERTY: (hypothesis)
19

---

TEXT: [text]
output:
F Manual Hypothesis Verification
E1
Experiment Setup
We recruited 10 participants through Qualtrics. Of these, participants completed 100 trials each, while participant completed 500 trials. In each trial, participants were presented with hypothesis selected either from the top or bottom 10 hypotheses generated from the Harry Potter dataset, along with a pair of sentences (the convergent sentence set) - in a randomized order. The task for participants was to determine which sentence aligned more closely with the given hypothesis, choosing between "More True for Sentence A", "More True for Sentence B", or "Equally true". Note that given hypothesis is not expected to apply all divergent sentences (e.g., might suffice for sentence be emotionally intense or grammatically complex to be divergent) so it is expected that some of the responses will be "Equally true". screenshot the experiment can be found in Figure 10
from DO (the divergent sentence set) and the other from D1
tosk, asked validote hypotheses about differences between collections sentences. You will be asked whether or not the hypothesis is more true for Sentence than Sentence You don't need to think too much about individual true for the two sentences presented. As an example, you will see tricl example below. it's not possible to evaluate seems equally true, click "Equaily true', only select this option there's enough information distingulsh between the two sentences.
possible
Progress: of 100
Hypothesls: contains movement, flying running
Hypothesis: contain sarcasm irony may language models detect
Sentence Harry ran onto broom, then flew to the top of the
Sentence A: "Excuse me, Professor Flltwick, could borrow Wood moment?" Sentence 'Wood told We're the team Beaters."
Sentence B: The flowers were blooming the garden.
More true for Sentence
More tlue for Sentonce
Equally
Equally
Figure 10: Screenshots of the experiment.
F.2 Results
We constructed a contingency table with dimensions (Top Hypothesis, Bottom Hypothesis) by (Prefer Divergent, Equal, Prefer Convergent) (Table 6). binomial test conducted on the contingency table (looking only at the "Prefer Divergent" and "Prefer Convergent" responses) showed that divergent sentences were more likely to be chosen over convergent sentences on average (p < (0-11). A Chi-square test revealed statistically significant differences in the distribution of responses between the top and bottom hypotheses (p 0.024). Additionally, we utilized the Chi-square test compare the frequency of "Prefer Divergent,' "Equal." and "Prefer Convergent" responses in two conditions. Notably, preference towards divergent sentences was observed in the top hypotheses condition compared to the bottom hypotheses (p- 0.093). In contrast, in the bottom hypothesis condition, there was a marked preference for "equally true" (p 0.008). significant difference was observed in the preference for convergent sentences between the conditions (p 0.676).
20

---

Table 6: Contingency Table for Human Responses
Divergent
Equal
Convergent
Top
377
122
Bottom
336
159
G Fine-tuning details
G.1 Computational Details
GPT-2 XI. was trained separately each of the two datasets in subsection 5.1 on 4 A6000 GPUs with 16-bit quantization and batch size of I per GPU. Deepspeed with ZeRo stage optimization was used in order parallelize training [Rasley et al.) 2020]. The Adam optimizer a learning rate of le-5, betas of (0.9, 0.999), epsilon of 1e-8, and no weight decay. Models were trained with early stopping with patience of [Kingma and Ba 2017
used with
G.2 Multiple-choice training
Let 2; represent the concatenation of the context, if applicable, and the question. Then for each answer choice Yi, we concatenate it with the question and context, and feed it to the model to obtain a sequence of logits.
4 Model(x;; @ yi)
Then we sum the logits corresponding to the sequence, where [1, 7) represents the total length of I, @ yr-
score;
Finally, we take the cross-entrope loss of these values relative to a one-hot encoding of the correct option, where t; if option correct, or else
exp elpgit)
P;
N
exp(logit,)
L Σ t (P.)
G.2.1 Performance on Multiple-Choice Datasets
We note that the performance of the final model may not approach that of GPT-2 XL fine-tuned with an output size of N denoting each option, as we keep the output dimension the same as the size of the vocabulary. However, we report the final accuracy achieved by each model on the original datasets here.
Table 7: Summary of model performance
Dataset
Best epoch
Accuracy (%)
Baseline (random) accuracy
Social IQa
4
54.86%
33.33%
PiQA
1
73.88%
50.00%
21

---

H
Annotations
To decide which category word belongs to, we employed three raters who used binary coding to indicate if word belonged the target category. The consistency among raters was evaluated using Krippendorff's ' alpha. Their consistency was 0.54 for social/emotion and 0.87 for physical. Finally, if least out of the three people annotated a word as fitting category, we counted belonging to that category.
H.1 Annotation Guidelines
H.1.1 Social/Emotional Intelligence
Include words that depict the emotions of characters and/or social interactions.
Exclude words that suggest emotions social interactions indirectly. For instance, "slam the door" shouldn't be annotated.
H.1.2 Physical commonsense
Annotate words referring to tangible entities, such as characters (people) and physical objects.
Do not annotate words that represent concrete ideas but lack physical substance, like "laughter""
Pronouns should also be excluded.
I Examples of phenomena in Harry Potter
We give some examples of the phenomena in the dataset according the annotations. Words of that category are marked in bold.
L1 Social/Emotional
Harry had never believed he would meet boy he hated more than Dudley.
Hermione Granger almost nervous about flying as Neville was.
But Neville, nervous and jumpy and frightened of being left on the ground, pushed off hard before the whistle had touched Madam Hooch's lips.
1.2 Physical Commonsense
Up the front steps, up the marble staircase inside, and still Professor McGonagall didn' t say a word to him.
Ron had piece of steak and kidney pie halfway to his mouth, but he'd forgotten all about
They pulled on their bathrobes, picked up their wands, and crept across the tower room, down the spiral staircase, and into the Gryffindor common room.
Algorithm for Permutation Test
To identify channels on which the performance of the fine-tuned model and the base model has statistically significant differences, we calculated empirical p-values by comparing the true correlation value with 10,000 simulated ones obtained by permuting the brain data as shown in Algorithm I Given that we are assessing multiple hypotheses simultaneously, we also used the Benjamini- Hochberg False Discovery Rate (FDR) [Benjamini and Hochberg [995] to correct for multiple comparisons, level 0.05.
22

---

Algorithm Permutation test (for one channel, one time window)
Input: Brain data D. Prediction from base model Pt, Prediction from fine-tuned model Pa
D, P1, and P2 are all of size (1, .), where N the number of words in the dataset.
Output: pralue
X = corr( D, P1) - corr(D, P2)
Counter = 0
for i = to 10, 000 do
D; permute (D)
X; = corr( Di, P1) - corr(Di,, P2)
if' X, X then
Counter = Counter +
end if end for
pualue Counter+1 10,000+1
K Comparison between Fine-tuned models and the Base Model
We provide a detailed view of the comparison between the base language model and the models fine-tuned social (Figure 11) and physical (Figure 12) datasets with each channel plotted.
L Additional Control Experiments on MSE Improvement
We conducted additional control experiments evaluate whether MSE improvement is specific to words that match the category of the dataset on which the model was fine-tuned. Specifically, we evaluated the improvement of physical words the model fine-tuned on the social dataset, and vice versa.
This analysis reveals that the performance of the model fine-tuned on the social dataset does not significantly differ when assessed with physical and non-physical words (Figure 13B). This finding implies that the enhancements observed are specifically tied to social words. On the other hand, in the model fine-tuned on the physical dataset, we observed a marginal, though not statistically significant, boost in performance with social words (Figure 13C). We propose that this marginal improvement could be attributed to the presence of social and emotional knowledge embedded within the physical dataset. To substantiate this hypothesis, conducted thorough review of the physical dataset and identified items that indeed pertain to social or emotional scenarios.
Examples of such items include:
how do you give surprise party?
To help your child feel less afraid when they're re going to sleep
How can you get child to smile in a photo?
To help a friend feel better when they are sad
how to avoid danger?
To determine if someone has romantic feelings for you
These findings led us to conclude that the marginal improvement processing social words by the model fine-tuned on the physical dataset may result from exposure to social and emotional content.
M Cross-validatee on language modelling task
We perform 3-fold cross-validation on the remaining chapters of the Harry Potter book (excluding chapters 9 and 10), where we randomly shuffle paragraphs and assign to train:validationttes sets respectively 77%, 16.5%, and 16.5% of the data. Paragraphs that exceeded the context length were excluded. Both the base GPT-2 model as well as each model fine-tuned on the three domains were trained to predict the next word for 3 epochs, with the same hyperparameters used in Anpenniix G
23

---

0 to 25 ms
25 to 50 ms
50 to 75 ms
75 to 100 ms
0.5
0.0%
85.9%
100.0%
100.. 0%
1.3%
0.0%
0.0%
100 to 125 ms
125 to 150 ms
150 to 175 ms
175 to 200 ms
0.5
100.0%
100.0%
100.0%
100. 0%
0.0%
0.0%
0.0%
-0.1
Correlation on Finetured Model
200 to 225 ms
225 to 250 ms
250 to 275 ms
275 to 300 ms
0.5
100.0%
100.0%
100.0%
100. 0%
0.0%
0.0%
0.0%
300 to 325 ms
325 to 350 ms
350 to 375 ms
375 to 400 ms
0.5
100.0%
99.0%
0.0%
100. 0%
0.0%
0.0%
13.1%
0.0%
400 to 425 ms
425 to 450 ms
450 to 475 ms
475 to 500 ms
0.5
100.0%
0.0%
0.0%
0.0%
0.0%
2.6%
2.0%
2.3%
0.3 0.5
0.5
0.1
0.3
0.5
Correlation on
Figure 11: Performance evaluation of the model fine-tuned on the Social IQa (social and emotional) dataset versus the base model using Pearson correlation. Each dot represents MEG channel. Red channels indicate better predictions by the fine-tuned model, blue channels indicate better predictions by the base model, and gray dots denote non-significant differences. The fine-tuned model outperforms the base model in predicting most channels during language processing time windows.
Results on the test set for each fold are listed below. The average negative-log-likelhhoo loss per token the end of training is reported in Table
Table 8: Summary of language-modeling loss across cross-validation folds for models on the remaining chapters of Harry Poller.
Model
Avg. Loss (%) + St.dev
Fold Loss
Fold 2 Loss
Fold Loss
Base
0.08795 + 0.01707
0.09794
0.06391
0.1020
Social
0.1148 + 0.00286
0.1119
0.1187
0.1
Physical
0.1001 + 0.00184
0.1019
0.1009
0.0976

---

0 to 25 ms
25 to 50 ms
50 to 75 ms
75 to 100 ms
0.5
0.0%
0.0%
100.0%
100.. 0%
100.0%
8%
0.0%
0.0%
100 to 125 m5
125 to 150 ms
150 to 175 ms
175 to 200 ms
0.5
100.0%
100.0%
100.0%
100.. 0%
0.0%
0.0%
0.0%
0.0%
Correlation on Finetured Model
200 to 225 ms
225 to 250 ms
250 to 275 ms
275 to 300 ms
0.5
95.8%
100.0%
100.0%
100.. 0%%
0.0%
0.0%
0.0%
300 to 325 ms
325 to 350 ms
350 to 375 ms
375 to 400 ms
0.5
0.0%
0.0%
0.0%
0.0%
96.7%
100.0%
100.0%
100.0%
400 to 425 ms
425 to 450 ms
450 to 475 ms
475 to 500 ms
0.5
0.0%
0.0%
0.0%
0.0%
91.5%
100.00 00.00%
100.0%
100.0%
0.3
0.5
0.5
0.1
0.3 0.5
on
Figure 12: Performance evaluation of the model fine-tuned on the PiQA (physical) dataset versus base model using Pearson correlation. Each dot represents MEG channel. Red channels indicate better predictions by the fine-tunes model, blue channels indicate better predictions by the base model, and gray dots denote non-significant differences. The fine-tuned model outperforms the base model predicting most channels during language processing time windows.
N Societal Impacts
This study represents significant intersection between Neuroscience and Machine Learning, striving to push the boundaries of machine learning models while deepening our understanding of how the human brain functions. In broader context, this research lays the groundwork for future breakthroughs in the field of neuroscience and for making human-computer interfaces more efficient and intuitive.
However, the development of human-computer interfaces may cause problems in privacy and security, as these interfaces often require the collection and processing of personal data, increasing the risk of data breaches and unauthorized access. There are also ethical concerns regarding the potential for surveillance and the impact on employment, as advanced interfaces might automate tasks currently performed by humans, potentially displacing workers.
25

---

B
0.002
MSE imprvement
0,002
caysical
Fine-turing Datasel Social
0.001
0.001
non phys
0.000
Canagory social
MSE
-0.001
non-social
[ms]
0.006
Word social
Word Category physiccll
NSE imorovement
Physical
0.004
0.004
0.002
0.000
0.000
200
300
Physical
Categorization
Figure 13: Comparison of improved MSE for A) social vs. non-social words the social model, B) physical vs. non-physical words on the social model, C) social vs. non-social words on the physical model, and D) physical vs. non-physical words on the physical model. Positive values denote lower MSEs the fine-tuned model. Shaded region indicates standard error. Asterisks denote time points with significant differences between the two groups (Student's 's t-test with FDR correction, p=0.55). 000.05)
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope. The abstract provides a concise summary of the key contributions, while the introduction elaborates on the assumptions and contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Limitations the work are discussed in section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
27

---

Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
The paper provides comprehensive disclosure of all the necessary information required to reproduce the main experimental results. The methodologies are described in sufficient detail, allowing for accurate replication of the experiments. Additionally, the code implementation is accessible via an online repository.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
28

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification: The code implementation for all experiments is accessible via an online reposi- tory. The Harry Potter dataset is open-sourced. The Moth dataset, which is component larger research initiative, will be publicly released once additional data collection is complete. Instructions command and environment to reproduce experimental results can be found the online repository.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
mtapps.cc/ubllcc/uu..eee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if' applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The paper specifies all the training and test details necessary to understand the results. The training of the encoding model is detailed in (sbbsecttion 2.2] and the fine-tuning details of the language models are specified in Appendix G] Additionally, the code for training these models available in the online repository provided.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The paper appropriately reports error bars and p-values, providing clear indications of the statistical significance of the experiments. Additionally, p- p-values are corrected for multiple comparisons.
29

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
The paper provides detailed information on the computational resources required to reproduce the experiments, including the type and number of GPUs used, as well the specific hyperparameters employed for fine-tuning the language model.
Justification:
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics dtttps://neuriie cttpppbccc/ttiiicuuueeiieiiiiiee
Answer: [Yes]
Justification: The research fully conforms the NeurIPS Code of Ethics. ensures fair compensation and adherence protocols for human research participants. Data privacy and consent are prioritized, with fair use of datasets. The paper transparently addresses potential societal impacts, including safety, security, discrimination, and environmental concerns, providing mitigation strategies. It avoids facilitating illegal activities and ensures fairness and human rights protection. Documentation of data and models is thorough, with necessary licenses and privacy protocols. Responsible release of models and accessibility of research artifacts are ensured, with all essential elements for reproducibility disclosed.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
30

---

If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The societal impacts of the work are discussed in detail in Appendix N covering both the potential positive and negative effects on society.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Not applicable. The paper does not involve data models that carry a high risk of misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
31

---

Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
The assets used in the paper are properly cited, with the creators or original owners credited. The versions we used are explicitly stated, and the license and terms of use are respected.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the satiaastsspppesswiii. paperswi thcode. com/datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Yes, the new assets introduced in the paper are well documented. Detailed instructions on how to use the code and data are provided in the online repository.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: The paper includes the full text of instructions given to participants and screenshot, which can be found Appendix F
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
32

---

According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [Yes]
Justification:
The experiments conducted in the study posed risk to participants. Addi- tionally, all experiments were thoroughly reviewed and approved by the Institutional Review Board (IRB).
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
33


Reviews:
[{"review_id": "gIKvEtSHG8", "reviewer": "Reviewer_hoiu", "summary": "The authors use a data-driven method to generate hypotheses about particular words/linguistic contexts in which a brain encoding model does not accurately predict the brain response to language. they find that social/emotional/physical complexity, along with linguistic complexity, are all associated with worse brain encoding performance. They fine-tune the base GPT model to perform better at social/emotional reasoning tasks and find that this yields an improvement in brain encoding for these particular words within the relevant language processing window (~75-400 ms).\n\n\nUpdate: I have read the other reviews and author rebuttals and will keep my score as-is.", "questions": "It seems like there is a lot more going on in the example sentences (Appendix C) than the hypotheses cover. For example, many function words are colored as \"most divergent\" despite not fitting the leading hypotheses. Can you provide a broader sample that might convince the reader of the coverage of the model-derived hypotheses?", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This is an interesting and novel approach using LLM proposer-verifier along with a behavioral experiment to validate the labeling generated by models.\n- The thorough experimental pipeline shows that a fine-tuning intervention designed along these hypotheses improves brain encoding performance as expected.\n- The paper opens up a new mysterious result to be explored -- it looks like there's a tradeoff between fitting these temporally intermediate responses and very early and very late responses to words.", "weaknesses": "- The fine-tuning intervention needs to be appropriately baselined, for example by fine-tuning on other tasks which don't match the hypotheses about what is driving brain encoding performance. If fine-tuning on an appropriate counterfactual sample doesn't improve brain encoding performance, this will strengthen the positive finding. (I'm not totally clear on the discussion in Appendix M -- are you saying that fine-tuning through language modeling on Harry Potter doesn't yield brain encoding improvements? This is a step in the right direction, I think.)\n- The effect sizes of improvements here are very small; the results would be much stronger if you could reproduce this effect on a second dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you Reviewer hoiu for the positive and valuable comments! Please find our answers to your comments below.\n\n## Fine-tuning\n\n> **The fine-tuning intervention needs to be appropriately baselined, for example by fine-tuning on other tasks which don't match the hypotheses about what is driving brain encoding performance. If fine-tuning on an appropriate counterfactual sample doesn't improve brain encoding performance, this will strengthen the positive finding.**\n\nThank you for highlighting the need to baseline the fine-tuning results on unrelated tasks. While we didn't conduct the exact experiment you suggested, we did run two related experiments as control checks. First, we confirmed that the model's improved brain alignment is not due to increased language modeling ability (Appendix M). Second, we verified the domain specificity of fine-tuning by evaluating the performance of physical words in the model fine-tuned on the social dataset (Appendix L). Our experiments show that the model's performance on physical versus non-physical words does not differ significantly, indicating that the improvement is specific to social knowledge (Figure 13B). Similar results were found when evaluating the model fine-tuned on the physical dataset with social versus non-social words (Figure 13C).\n\n>**The effect sizes of improvements here are very small; the results would be much stronger if you could reproduce this effect on a second dataset.**\n\nWe appreciate your suggestion to replicate the fine-tuning results on a different dataset. However, due to the limited rebuttal period, we are unable to conduct this experiment as it requires recruiting human annotators to annotate each word in the second dataset. We greatly appreciate your input and look forward to incorporating datasets from a broader range of contexts in future studies to further validate and expand our findings.\n\n\n## Example Sentences\n\n> **It seems like there is a lot more going on in the example sentences (Appendix C) than the hypotheses cover. For example, many function words are colored as \"most divergent\" despite not fitting the leading hypotheses. Can you provide a broader sample that might convince the reader of the coverage of the model-derived hypotheses?**\n\nPlease refer to the PDF in the global rebuttal for 10 additional sentences selected from the top 20% most divergent sentences. We note that identifying hypotheses by examining individual words in sentences is challenging, which motivates our use of an automatic hypothesis proposing method. However, it is worth noting that words related to emotions (e.g., pleased, nervous, afraid, madly) are often among the most divergent.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jgzms6uL92", "reviewer": "Reviewer_jRnr", "summary": "This paper explores the differences between LMs and the human brain in processing language. The authors conduct experiments using MEG data from reading and listening tasks to investigate elements of MEG responses that LMs cannot adequately explain. LLMs are used to automatically generate hypotheses, identifying the domains where LMs lack knowledge compared to the human brain. The authors then fine-tune LMs on these specific domains to improve the LMs-brain alignment.", "questions": "The relatively narrow datasets used in the paper, focusing on human dialogue and stories, which intuitively consist of emotion representation, could easily introduce bias.", "limitations": "See Weakness and Questions", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The idea of assessing and interpreting the biological plausibility of LMs is a feasible way to enhance both their interpretability and model design. Focusing on the divergences between LMs and the brain is currently lacking in the field.\n\nThe paper is well-written, includes good visualizations, and the idea is easy to follow with good reproducibility.", "weaknesses": "Only two LMs are used in the experiment. Additionally, GPT-2 is somewhat outdated compared to recently released open-source LLMs, as it shows limited language understanding and reasoning capability, making it far from achieving human-level AGI. The focus should be more on LLMs such as Mixtral and Gema.\n\nMore details about the prompts and API usage should be discussed for the proposer and verifier LLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you Reviewer jRnr for the valuable and constructive feedback! Please find our answers to your comments below.\n\n## GPT-2 vs Modern LLMs\n\nWe agree that more modern models may demonstrate enhanced capabilities in areas like language understanding and social reasoning. To explore this, we replicated our analyses on Llama-2 7B, a larger and more recent model compared to GPT-2 XL (see Appendix E). Our results show that Llama-2 7B’s hypotheses predominantly focus on physical knowledge, with the social and emotional dimension no longer a theme in the generated hypotheses. This suggests that Llama-2 7B might possess a more sophisticated understanding of social and emotional contexts. We look forward to replicating the analyses on other models with different parameter counts, pre-training datasets, and methodologies for training and fine-tuning in future work.\n\n## Proposer and Verifier\n\nThanks for bringing up the importance of discussing more details about prompts and API usage for the proposer and verifier LLMs. We have included the prompt we used and will incorporate these details into the next version of the paper.\n\nWe used the gpt-4-turbo-instruct as the proposer model. The prompt for the proposer is:\n\n> {A_block}\n> \n> {B_block}\n> \n> The dataset includes two chapters from \"Harry Potter and the Sorcerer's Stone\". The two groups are generated based on the difference between language model and human responses to these sentences. The Group A snippets sentences where language models and humans show divergent responses, while the Group B snippets sentences where language models and humans show similar responses. \n> \n> I am a literary analyst investigating the characteristics of words. My goal is to figure out which sentences induce different responses for language models and human responses. \n> \n> Please write a list of hypotheses about the datapoints from Group A (listed by bullet points \"-\"). Each hypothesis should be formatted as a sentence fragment. Here are three examples.\n> \n> \\- \"{example_hypothesis_1}\"\n> \n> \\- \"{example_hypothesis_2}\"\n> \n> \\- \"{example_hypothesis_3}\"\n> \n> Based on the two sentence groups (A and B) from the above, more sentences in Group A ...\n\n\nWe used FLAN-T5-XXL as the validator. The prompt for the validator is:\n\n> Check whether the TEXT satisfies a PROPERTY. Respond with Yes or No. When uncertain, output No. \n> \n> Now complete the following example -\n> \n> input: PROPERTY: {hypothesis}\n> \n> TEXT: {text}\n> \n> output:\n\n\n## Datasets\n\nThank you for pointing out the concern about the relatively narrow datasets used in the paper. While we did discuss it in the final paragraph of the Conclusions, Limitations, and Future Work section, it is a limitation that we plan to address in future work. Due to the high cost of recording human brain responses with neural imaging techniques, there is a very limited selection of publicly available datasets. We agree that incorporating datasets from a broader range of contexts in future studies would be valuable for further validating and expanding our findings. Indeed, to our knowledge, several new relevant datasets are currently in progress, and we also plan to collect our own data using more diverse texts.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y8xQb0EXTD", "reviewer": "Reviewer_b2dQ", "summary": "Language models are known to predict MEG signals in humans during reading. In this work, the authors explored for what \"kinds\" of texts are language models bad at predicting MEG signals. The authors used a novel method to propose possible hypotheses, and found multiple possible categorizations of weak prediction texts. Focusing on social knowledge and spatial commonsense, the authors find that finetuning the language model on texts from these respective domains improves the prediction accuracy.", "questions": "1. Temporal signal prediction is convincing, another aspect that is interesting is the spectral structure of the predictions.\n2. Do layers other than 7 display the same pattern of lower prediction accuracy? Do you observe this consistently across layers, or just at layer 7?", "limitations": "Properly addressed.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The experiment design is clean and straightforward.\n2. The detailed evaluation of error pattern during brain signal prediction is very valuable.\n3. The automated method of hypothesis proposal might be generalizable to other forms of experiments in cognitive sciences as well.", "weaknesses": "Many aspects mentioned in the work where the language model are not performant on, for example common sense and social reasoning, have large leaps forward in more modern models. A future work could focus on using more modern language models with stronger capacity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you Reviewer b2dQ for the positive and constructive review! Please find our answers to your comments below.\n\n>**Many aspects mentioned in the work where the language model are not performant on, for example common sense and social reasoning, have large leaps forward in more modern models. A future work could focus on using more modern language models with stronger capacity.**\n\n\nThank you for your comment, we agree that more modern models may demonstrate enhanced capabilities in areas like common sense and social reasoning. To explore this, we replicated our analyses on Llama-2 7B, a larger and more recent model compared to GPT-2 XL (see Appendix E). Our results show that Llama-2 7B’s hypotheses predominantly focus on physical knowledge, with the social and emotional dimension no longer a theme in the generated hypotheses. This suggests that Llama-2 7B might possess a more sophisticated understanding of social and emotional contexts. We look forward to replicating the analyses on other models with different parameter counts, pre-training datasets, and methodologies for training and fine-tuning in future work.\n\n>**Temporal signal prediction is convincing, another aspect that is interesting is the spectral structure of the predictions.**\n\n\nYes, we think exploring spectral dimensions would be an intriguing direction for future research. For example, alpha waves are often associated with passive attention, whereas higher frequency waves, such as gamma waves, are linked to concentration and information processing. We’re excited about the possibility of examining how brain signals from different frequency bands might correlate with language model embeddings.\n\n> **Do layers other than 7 display the same pattern of lower prediction accuracy? Do you observe this consistently across layers, or just at layer 7?**\n\n\nWe also ran the analyses on the last layer of GPT-2 XL (Appendix D.1). The generated hypotheses included those observed from the best performing layer, but also encompassed themes such as magical elements and figurative language. This suggests a layer-dependent diversity in encoded information. Nonetheless, it appears that across layers, there is a consistent divergence of physical and social knowledge between the LM and human brains.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o6nr8ys1aX", "reviewer": "Reviewer_fRry", "summary": "The paper studies the differences in language representations in LLMs (GPT-2 XL) and the human brain (MEG activity), by using LLM representations to predict MEG activity. They use an LLM-based approach to identify two domains where LLM representations do not capture brain activity well: social/emotional intelligence and physical commonsense. They also show that fine-tuning GPT-2 XL on these domains can improve their alignment with human brain responses.", "questions": "1. Why is layer 7 of GPT-2 XL the best at predicting brain responses for this dataset? Layer 7 of 48 seems relatively low as opposed to intermediate, whereas prior research showed that intermediate layers are best (lines 115-116 in your paper).", "limitations": "The limitations I identified have been mentioned in the paper.", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "From most to least significant:\n1. The paper studies the highly interesting, important, and relevant topics of social/emotional intelligence and physical commonsense. Their results highlight the importance of improving social/emotional intelligence and physical commonsense to build more human-brain-aligned LLMs.\n2. Interesting methodology using proposer and verifier LLMs. LLMs are growing increasingly capable, and it is great that the paper takes advantage of this trend, to advance the study of the brain.\n3. After their analyses on brain alignment, I appreciate that they performed further investigations: human behavioral experiments (Section 3.3), and releasing annotations as a further resource to the dataset (Section 5).", "weaknesses": "From most to least significant:\n1. They used a relatively low layer (layer 7) of GPT2-XL when making the claim that LMs have poor brain alignment for high-level properties (social/emotional intelligence, physical commonsense). They should have studied a higher layer instead/too.\n- The paper argues they identified two domains that LMs do not capture well: social/emotional intelligence and physical commonsense.\n- However, they used layer 7 of GPT-2 XL, a relatively low layer since GPT-2 XL has 48 layers. They say they use this because it is the best at predicting brain activity, but this seems much lower than prior research: Layer 8 of 12 in BERT and Layer 12 of 19 in T-XL [1]\n- Prior work has shown that lower layers of LLMs capture lower-level linguistic properties, while higher layers capture higher-level properties. Prior work also showed that \"LM representations can predict the hierarchy of brain responses\" (lines 290-291 in their paper).\n- In this case, is it possible that the higher layers (30+) of the same GPT-2 XL model will achieve high brain alignment for social/emotional intelligence and physical commonsense? \n- However, re-evaluating more layers is expensive, and I am not requesting them to do this if they can address the comment in writing. \n2. Results in Table 1 are not convincing in supporting their overall claims. The LLM-based system of proposers and verifiers generated hypotheses on how the top 100 least-predicted sentences differ from the top 100 well-predicted sentences. However, it seems that 331 of 708 human participants did not agree with the top hypotheses suggested by the LLM system (Table 1), i.e., either found the sentences to be \"Equal\" or \"Convergent\". This casts doubt that the two sets of sentences actually differ in terms of social/emotional intelligence and physical commonsense. This, in turn, casts doubt on their overall claim that LLMs do not capture social/emotional intelligence and physical commonsense well.\n3. Additional training data as a potential confound\n- The paper showed that training LLMs on a dataset for emotional intelligence improves brain alignment. I agree that by itself this is interesting. However, is this because of: (A) statistical similarities between the text in this training dataset and the MEG stimuli, especially for words related to emotional intelligence, or (B) LLM gains better emotional intelligence understanding, as labels were provided and it learns to predict emotions correctly?\n- Alternative reasons that finetuning on social dataset improved brain alignment on social words: (a) because the model was provided additional training data? (b) because the additional dataset was higher quality?\n- Possible control experiment: training the model (using language modeling) on the same dataset but without associating each question with the correct label. Basically, do not tell the model which is the correct answer (no supervision).\n- It is possible that the authors can answer these questions using their further experiments in Appendix I and M. Still, I hope they explicitly address, within the paper, the issue of additional training data as a potential confound.\n\n[1] M. Toneva and L. Wehbe, “Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain).” 2019", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you Reviewer fRry for the valuable and constructive feedback! Please find our answers to your comments below.\n\n\n## Best Performing Layer\n\nRegarding the question about why layer 7 (0-indexed) is the best layer at predicting brain responses in our study, which differs from prior literature, we would like to clarify two main differences between the prior research and our study.\n\nFirst, the conclusion from [1] is based on fMRI data, whereas our experiments are based on MEG data. Previous study [2] has demonstrated that MEG detects activity primarily in temporal areas, whereas fMRI reveals activity in both temporal and prefrontal regions during language-related tasks. This suggests that the discrepancy in the best-performing layer may arise from the different aspects of brain activity captured by these imaging techniques. However, prior research has consistently shown MEG's effectiveness in detecting the processing of emotions [3-6], supporting its value for studying social and emotional processing.\n\n\nSecond, the models used in our study have significantly more parameters than those used in prior research. [1] used models like BERT (110M parameters) and Transformer-XL (128M parameters), while our study employed larger models: GPT-2 XL (1.5B parameters) and Llama-2 7B (7B parameters). One possibility is that as model size increases, the best performing layer tends to be relatively earlier. For instance, in GPT-2 XL, layer 7 is the best (Figure 5A), while in Llama-2 7B, it is layer 3 (Figure 5B).\n\n\nWe also want to highlight that we ran our analyses on the **last layer of GPT-2 XL** (Appendix D.1). The generated hypotheses included those observed from the best performing layer, but also encompassed themes such as magical elements and figurative language. This suggests a layer-dependent diversity in encoded information. Nonetheless, it appears that across layers, there is a consistent divergence of physical and social knowledge between the LM and human brains.\n\n\n## Human Study\n\nRegarding the question about the human study, we would like to clarify the experimental setting. On each trial, participants were presented with one hypothesis and asked to select between one sentence from the divergent set and one sentence from the convergent set. There are a total of ten hypotheses, and each sentence may only satisfy one or two of them. When the hypothesis does not match the sentence from the divergent set, participants should have no preference for either sentence. Thus, the absolute values are not meaningful; the key point is the comparison of the percentage of responses preferrring \"divergent\" between the top and bottom hypotheses conditions, represented by the blue areas in Figure 10A and Figure 10B. Thank you for bringing this to our attention, we'll will ensure that we clarify these points about the study in the reversion of the paper.\n\n## Fine-Tuning\n\nRegarding the question about additional training data as a potential confound in fine-tuning, we appreciate your insights. To address this, we conducted two control checks. First, we confirmed that the model's improved brain alignment is not due to increased language modeling ability (Appendix M). Second, we verified the domain specificity of fine-tuning by evaluating the performance of physical words in the model fine-tuned on the social dataset (Appendix L). Our experiments show that the model's performance on physical versus non-physical words does not differ significantly, indicating that the improvement is specific to social knowledge (Figure 13B). Similar results were found when evaluating the model fine-tuned on the physical dataset with social versus non-social words (Figure 13C). We appreciate your raising these points and will explicitly address the potential confound of additional training data in the revised version of the paper.\n\n\n\n[**1**] Toneva, Mariya, and Leila Wehbe. \"Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain).\" Advances in neural information processing systems 32 (2019).\n\n[**2**] Billingsley-Marshall, Rebecca L., et al. \"A comparison of functional MRI and magnetoencephalography for receptive language mapping.\" Journal of Neuroscience Methods 161.2 (2007): 306-313.\n\n[**3**] Giorgetta, Cinzia, et al. \"Waves of regret: A meg study of emotion and decision-making.\" Neuropsychologia 51.1 (2013): 38-51.\n\n[**4**] Peyk, Peter, et al. \"Emotion processing in the visual brain: a MEG analysis.\" Brain topography 20 (2008): 205-215.\n\n[**5**] Dumas, Thibaud, et al. \"MEG evidence for dynamic amygdala modulations by gaze and facial emotions.\" PloS one 8.9 (2013): e74145.\n\n[**6**] Hagan, Cindy C., et al. \"MEG demonstrates a supra-additive response to facial and vocal emotion in the right superior temporal sulcus.\" Proceedings of the National Academy of Sciences 106.47 (2009): 20010-20015.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Please refer to the attached PDF for 10 additional sentences colored based on prediction error. These sentences are selected from the top 20% most divergent sentences in the Harry Potter dataset. Each of the five colors corresponds to a 20-percentile range of words from the entire dataset.

Meta-review: The authors employ a data-driven approach to identify specific words and linguistic contexts where a brain encoding model fails to accurately predict neural responses to language. They discover that factors such as social, emotional, physical, and linguistic complexity contribute to poorer performance of the brain encoding model. To address this, they fine-tune the base GPT model to enhance its capabilities in social and emotional reasoning tasks. This adjustment leads to improved brain encoding predictions for these specific words during the relevant language processing time frame, approximately 75 to 400 milliseconds.

The reviewers raised many technical issues, including experiments with only two models, the possibility that newer models would demonstrate different performance and various technical questions about the experiments. Some of these issues have been solved in the discussion period and all four reviewers are leaning towards acceptance. Overall, this is an empirical paper on an important topic and I believe it makes a nice contribution that justifies publication. 

As a side note - the paper is missing a very relevant citation that also shows that fine tuning to a specific domain can substantially improve alignment with the human brain, although with fMRI data:

Tikochinski, R., Goldstein, A., Yeshurun, Y., Hasson, U., & Reichart, R. (2023). Perspective changes in human listeners are aligned with the contextual transformation of the word embedding space. Cerebral Cortex, 33(12), 7830-7842.‏

I strongly encourage the authors to cite and compare to this paper.
Decision: Accept (poster)