Divergences between Language Models and Human Brains

Authors: yuchen  zhou
emmy  liu
graham  neubig
michael j tarr
leila  wehbe
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 04:51:02

Abstract: Do machines and humans process language in similar ways? Recent research has hinted at the affirmative, showing that human neural activity can be effectively predicted using the internal representations of language models (LMs). Although such results are thought to reflect shared computational principles between LMs and human brains, there are also clear differences in how LMs and humans represent and use language. In this work, we systematically explore the divergences between human and machine language processing by examining the differences between LM representations and human brain responses to language as measured by Magnetoencephalography (MEG) across two datasets in which subjects read and listened to narrative stories. Using an LLM-based data-driven approach, we identify two domains that LMs do not capture well: social/emotional intelligence and physical commonsense. We validate these findings with human behavioral experiments and hypothesize that the gap is due to insufficient representations of social/emotional and physical knowledge in LMs. Our results show that fine-tuning LMs on these domains can improve their alignment with human brain responses.

Main paper: 
 introduction: 
 Language models (LMs) now demonstrate proficiency that may equal or even surpass human-level performance on tasks including generating text [Brown et al., 2020a] , answering questions [Lewis et al., 2019] , translating languages [Costa-jussà et al., 2022] , and even tasks that necessitate reasoning and inference [Dasgupta et al., 2022] . This has inspired researchers to leverage LM representations to investigate and model the human brain's language system, positing that LMs may serve as reliable proxies for human linguistic processes [Abdou, 2022] . Prior studies have found that human neural activity, as measured by neuroimaging techniques such as fMRI [Jain and Huth, 2018, Toneva and Wehbe, 2019] , EEG [Hale et al., 2018] , MEG [Wehbe et al., 2014a] , and ECoG [Goldstein et al., 2022] , can effectively be predicted using representations from language models such as BERT [Devlin et al., 2018] or GPT-2 [Radford et al., 2019] . Robust neural prediction is hypothesized to stem from the shared computational objective of both LMs and the human brain: predicting subsequent words based on prior context [Yamins and DiCarlo, 2016, Schrimpf et al., 2021] .Despite the evident behavioral similarities, the extent to which LMs and human brains align functionally for language processing remains an open question. Essentially, the methods that LMs and humans use to acquire language are very different. LMs learn statistical regularities across massive sets of linguistic symbols, whereas humans rely on applying structured linguistic principles across relatively little input. Additionally, LMs that are confined to linguistic data are likely to fail to ground linguistic symbols in real-world contexts [Harnad, 1990 , Bender and Koller, 2020 , Bisk et al., 2020a] . Furthermore, the learning environments and goals of LMs and humans are markedly different. While humans communicate through active inquiry, expressing needs, directed communication, and Figure 1 : Schematic of our experimental approach. The LM takes as input the current word along with its preceding context to produce the current word's LM embedding. This embedding is then used as input to a ridge regression model to predict the human brain responses associated with the word. The Mean Squared Error (MSE) between the predicted and actual MEG responses is calculated. Finally, an LLM-based hypothesis proposer is employed to formulate natural language hypotheses explaining the divergence between the predicted and actual MEG responses.scaffolding conversations [Kuhl, 2011] , LMs are predominantly trained as passive recipients of raw text data. Consequently, LMs may struggle with comprehending social pragmatics and the nuances of words whose meanings fluctuate across different social contexts [Mahowald et al., 2023] .Previous research exploring the relationship between human and LM language processing has typically focused on the types of linguistic features [Oota et al., 2022a , Sun et al., 2023] , neural network architectures [Schrimpf et al., 2021] , or training and fine-tuning methods [Sun and Moens, 2023] that may yield better predictions of brain responses. Diverging from this approach, Aw and Toneva [2023] proposed that the divergence between human and LM language processing might stem from LMs' inadequate understanding of texts. They supported this hypothesis by demonstrating that LMs fine-tuned on summarization tasks align more closely with human brain responses. Yet, this hypothesis is only one of many potential explanations. In this work, we adopt a bottom-up, data-driven methodology to systematically investigate the differences between human and machine language processing. Our main contributions are as follows:1. In contrast to prior studies focusing on the similarities between LMs and human brains, our research emphasizes their differences. We monitor the temporal progression of errors in LM predictions on a word-by-word basis on two datasets with distinct language input modalities ( §2).2. Explaining the prediction errors for every word is challenging due to the vast amount of text. Instead of manually formulating hypotheses, we adopt an LLM-based method that automatically proposes natural language hypotheses to explain the divergent responses between human brains and language models ( §3). The top candidate explanations are related to social/emotional intelligence and physical commonsense ( §4). We validate these hypotheses with human behavioral experiments.3. We present evidence that fine-tuning LMs on tasks related to the two identified phenomena can align them more closely with human brain responses. This implies that the observed divergences between LMs and human brains may stem from LMs' inadequate representation of these specific types of knowledge ( §5).2 Predictive MEG ModelQuestion: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope. The abstract provides a concise summary of the key contributions, while the introduction elaborates on the assumptions and contributions.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
data preparation and preprocessing: 
 While many studies investigating the correlation between brain responses and language models utilize fMRI recordings (e.g., [Caucheteux et al., 2023 , Jain et al., 2020] ), the comparatively low temporal resolution of fMRI hinders its ability to accurately capture the processing of individual words. To address this limitation, our research employed MEG data. We strategically used two different MEG datasets, each with distinct input modalities, to assess potential variations in the brain's response patterns under these conditions.The first dataset [Wehbe et al., 2014a] has eight participants reading Chapter 9 of Harry Potter and the Sorcerer's Stone (5,176 words) and four participants reading Chapter 10 of the same book (4,475 words) . Each word was exposed on a screen for a fixed duration of 500ms. MEG data were collected on an Elekta NeuroMag MEG with 306 channels at 102 cranial points, and sampled at a rate of 1 kHz. The acquired data underwent preprocessing procedures using the Signal Space Separation (SSS) method [Taulu et al., 2004] and its temporal extension, tSSS [Taulu and Simola, 2006 ]. The signal was then time-locked with individual words and down-sampled into non-overlapping 25ms time bins. Given the typical low Signal-to-Noise Ratio (SNR) of MEG, we adopted a denoising technique [Ravishankar et al., 2021] that takes advantage of cross-subject correspondences to get an aggregated, denoised version of MEG responses (refer to Appendix A for more details).To enhance reproducibility and generalizability of our study, we additionally collected MEG data from one participant who listened to six narratives (11,626 words) from The Moth, a platform featuring personal storytelling. These stories were chosen from the stimuli used in a published story listening fMRI dataset [LeBel et al., 2023] . Five of these stories were repeated twice, while one story was repeated five times. The data acquisition was performed using a MEGIN scanner equipped with 306 channels at 102 cranial points. The preprocessing pipeline was similar to that applied to the first dataset. Given that all story repetitions were from the same participant, we averaged the MEG responses for each story's repetitions to enhance SNR without using an alignment method.A substantial number of recent studies exploring the correlation between brain responses and LMs have employed GPT-2 [Pasquiou et al., 2022 , Caucheteux et al., 2022 , 2023 , Toneva et al., 2022] . To ensure consistency and comparability with these studies, we utilized the pre-trained GPT-2 XL model with 1.5B parameters, sourced from HuggingFace's transformers library [Wolf et al., 2020a] , as the backbone language model. Following previous work [Toneva and Wehbe, 2019] , for every word w, we provided the model with a context consisting of the preceding 19 words. We used the output of hidden layers of the LM, subsequently referred to as LM embeddings, to predict the MEG responses associated with each word (Figure 1 ). For comparison, we also replicated some analyses on Llama-2 7B [Touvron et al., 2023a ] (refer to Appendix E for more details).Building upon established research that demonstrates the capability of LM embeddings to linearly predict MEG responses [Wehbe et al., 2014a , Jain and Huth, 2018 , Caucheteux and King, 2022a] , we utilized a linear ridge regression model as the encoding model. Considering the time-correlated nature of MEG data, it was essential to maintain the temporal structure when partitioning the data for training and testing purposes [Yang et al., 2019] . Therefore, we implemented a 10-fold crossvalidation procedure that splits the MEG data into 10 continuous chunks. We denote the actual MEG responses as M and LM embeddings as L. For split i, we set aside one fold as the test set (M i,test , L i,test ) and fitted a ridge regression model with weight matrix W i and bias b i using the remaining folds, denoted as (M i,train , L i,train ). The regularization parameters were chosen via nested cross-validation. Following model training, we applied the trained weight matrix and bias to predict the brain responses from the LM outputs for the test set:M i,test = L i,test Ŵ i + biFinally, the test predictions from all folds were concatenated to form the comprehensive prediction of MEG responses from the LM:M = concat[ M i,test ]This process is performed for each of the different time windows relative to word onset.Prior research has shown that intermediate layers of language models often best predict human brain responses [Toneva and Wehbe, 2019 , Jain and Huth, 2018 , Oota et al., 2022b] . Echoing previous findings, we confirmed that intermediate layers exhibit higher correlations, with layer 7 being the best at predicting brain responses in GPT-2 XL (refer to Appendix B for more details). Therefore, we used the output of layer 7 as the language model embeddings in subsequent experiments.As a sanity check, we examined if the predictive model can effectively predict the brain areas and time course of language processing. These areas include the inferior frontal gyrus, superior temporal gyrus, certain sections of the middle temporal gyrus, and angular gyrus [Blank et al., 2016 , Rogalsky et al., 2015 , Sahin et al., 2009 , Brennan and Pylkkänen, 2012 , Friederici, 2002 , Visser et al., 2010 , Rogalsky and Hickok, 2009] .As shown in Figure 2 , we observe a temporal progression of accurately predicted areas after word onset. The prediction performance peaks first in the occipital lobe between 75-100ms. Given that LM embeddings encode information (e.g., word frequency) correlated to the number of letters in a word and MEG is sensitive to abrupt changes in visual inputs, we attribute this early peak to the initial visual perception of a word. This is followed by heightened prediction performance in the bilateral temporal lobe between 175-250ms, when we expect semantic processing to start. This observation aligns with previous research indicating that most language experiments with naturalistic stimuli reveal bilateral language representations [Wehbe et al., 2014b , Huth et al., 2016 , Deniz et al., 2019 , Toneva et al., 2022] . Finally, between 250-375ms, the anterior temporal lobe and frontal lobe show increased prediction performance, which is likely related to further semantic processing. This sequential pattern of prediction performance replicates the spatio-temporal dynamics of language processing found in previous literature [Wehbe et al., 2014a , Toneva et al., 2022] .Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: Limitations of the work are discussed in section 7.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
identifying phenomena of interest: 
 Our objective is to investigate the elements of MEG responses that cannot be well explained by the LM. We work with an average of cleaned MEG responses from a group of subjects and multiple trials, which illustrate the common elements of language processing across individuals. Therefore, for words where MEG responses are not well predicted, it is likely that this marks a genuine divergence between human brains and the LM. It is important to clarify that our approach trains an encoding model to predict human brain responses based on language model outputs, rather than the reverse. This means our methodology identifies information that is captured by MEG but is not present in the language model, rather than information captured by the language model but is not present in MEG responses.Leveraging the high temporal resolution of MEG, we computed the Mean Squared Errors (MSEs) between actual and predicted MEG responses for each individual word on channels that demonstrated statistically significant correlations (one-sided, p=0.001). For word w, where S is the set of significant channels.M SE(w) = 1 |S| • i∈S ( M (w) i -M (w) i ) 2 (1)Given the vast amount of text, manual pattern discovery becomes challenging. Figure 7 presents sample sentences color-coded based on prediction error, illustrating the challenges in formulating hypotheses from observations.To discover subtle differences between MEG responses and LM predictions, we used a method that automatically describes differences between text corpora using proposer and verifier LMs [Zhong et al., 2023] . This system consists of first prompting an LLM (GPT-3; Brown et al. [2020b] ) with a number of samples from two corpora (D 0 , D 1 ) to generate many hypotheses on how the first corpus differs from the second, and then using a fine-tuned validator model (FLAN-T5-XXL; Chung et al.[2022]) to validate how often each proposed hypothesis is true based on pairs from each corpus sampled from a held-out set. Specifically, the verifier is presented with a prompt containing two sentences from D 0 and D 1 , and asked whether or not the hypothesis is true, and this is repeated across the development set for each hypothesis. For the exact prompts used in the proposer and verifier, please refer to Appendix F. Sentences were then ranked based on their mean MSE. The top 100 least-predicted sentences constituted the D0 set, while the 100 well-predicted sentences constituted the D1 set. This process of hypothesis proposal and verification was repeated across 3 cross-validation folds.The top ten hypotheses from the Harry Potter dataset ranked by validity2 are listed in Table 1 . P-values were adjusted using FDR correction [Benjamini and Hochberg, 1995] . We identified two primary differences between the language model and the human brain: firstly, the processing of social and emotional information, and secondly, the capacity for interaction with the surrounding environment. These are henceforth referred to as social/emotional intelligence and physical commonsense, respectively. Importantly, these hypotheses resonate with conclusions drawn in prior research, as detailed in §4. Similarly, we ran the hypothesis proposer on the Moth dataset. This replication produced slightly varied but fundamentally similar topics to those discovered in the Harry Potter dataset (Table 2 ). This congruence across datasets with different input modalities aligns with previous research showing that after initial sensory processing, the brain's language processing is consistent across reading and listening [Deniz et al., 2019 , Chen et al., 2023] .We conducted an experiment involving human participants for additional validation of our hypotheses. We gathered data from 10 participants using Qualtrics, resulting in a collection of 1,400 trials. In each trial, participants were presented with a hypothesis selected either from the top 10 or bottom 10 hypotheses generated from the Harry Potter dataset, along with a pair of sentences -one from D0 and the other from D1 -in a randomized order. The task for participants was to determine which sentence aligned more closely with the given hypothesis, choosing between "More True for Sentence A", "More True for Sentence B", or "Equally true".The response distribution for each hypothesis is shown in Figure 3 . Note that a given hypothesis is not expected to apply to all divergent sentences (e.g., it might suffice for a sentence to be emotionally intense or grammatically complex to be divergent). If a hypothesis does not align with a sentence from the divergent set, participants should show no preference between the two sentences presented.Chi-square analysis revealed statistically significant differences in the distribution of responses between the top and bottom hypotheses (p = 0.024). A preference towards divergent sentences was observed in the top hypotheses condition while a preference towards "Equally True" was observed in the bottom hypothesis condition. This pattern can be attributed to the role of the proposer, which was instructed to generate hypotheses that effectively distinguish D 0 (comprising divergent sentences) from D 1 . As a result, the bottom hypotheses tend to be those that fail to differentiate between D 0 Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? 
selected phenomena: 
 Comprehending social/emotional and physical commonsense requires humans use a broad spectrum of contextual knowledge. We briefly discuss the insights and challenges highlighted in the existing neuropsychological and NLP literature regarding these domains.Human social and emotional intelligence requires both introspection and predicting the feelings of others [Salovey and Mayer, 1990] . Neuropsychological research on social cognition has identified a network of brain regions that support understanding other people's intentions, actions, and emotions [Saxe et al., 2006] . Crucially, emotions are intrinsic to the human experience and pervasively interact with other mental facilities, including language [Satpute and Lindquist, 2021] . As such, creating agents with social and emotional intelligence has been a longstanding goal of NLP [Gunning, 2018 , Paiva et al., 2021] . However, LLMs still fall short of human abilities for inferring the mental states and emotions of others ("theory-of-mind" tasks) [Sap et al., 2022] .Physical commonsense refers to knowledge about the physical properties of everyday objects and physical phenomena [Forbes et al., 2019 , Bisk et al., 2020b] . From a neuropsychological perspective, language is not the primary means through which humans acquire commonsense physical knowledge. Instead, humans rely on sensory inputs and interactions with their environment [Baillargeon, 1994] .Notably, the category of a physical object affects which brain regions are recruited when interacting with that object. For example, interacting with people activates the theory of mind areas [Saxe et al., 2006] , the visual face areas [Sergent et al., 1992 , Kanwisher et al., 1997] , and body areas [Downing et al., 2001] , interacting with corridors while navigating recruits the visual place [Epstein and Kanwisher, 1998 ] and spatial navigation areas, and interacting with tools recruits the dorsal object-processing stream [Almeida et al., 2010] . Interestingly, reading about these domains has also been found to recruit these same visual regions [Wehbe et al., 2014b , Huth et al., 2016] . Given how physical commonsense knowledge is acquired, it is not surprising that, within NLP, this domain poses a challenge for language models. While these models can potentially learn representations capturing specific physical properties of the world, such as an object's color or a game board's state [Abdou et al., 2021 , Li et al., 2023] , it remains unclear whether purely text-based representations can capture the richness and complexity of physical commonsense as exhibited by humans [Forbes et al., 2019 , Bisk et al., 2020b] . 
improving brain alignment via fine-tuning: 
 We hypothesize that the inability of language models to accurately predict associated brain responses stems from their inadequate representations of social/emotional understanding and physical world knowledge. Drawing inspiration from Aw and Toneva [2023] , we fine-tuned the GPT-2 XL model on datasets specific to the two phenomena to examine if targeted fine-tuning could enhance the model's alignment with brain activity.Furthermore, we examined whether domain-specific fine-tuning would specifically bolster the model's capability in predicting MEG responses associated with words from that domain, as compared to words outside that domain. To this end, we recruited three raters to annotate Chapter 9 of Harry Potter across the two domains. We release these annotations as a resource for the dataset to facilitate further analysis. Details on the annotation process can be found in Appendix I. Examples of each phenomenon within the Harry Potter text can be found in Appendix J.Social/Emotional Intelligence We study social and emotional intelligence using the Social IQa dataset [Sap et al., 2019] . This dataset contains questions about people's feelings and their social implications.We study physical commonsense using the PiQA dataset [Bisk et al., 2020b] . This dataset contains goal-driven questions based on everyday situations. These questions were taken from the website instructables.com, where people share DIY project instructions.We also provide examples from each dataset in Table 3 . In order to keep the architecture of fine-tuned models consistent with the base model, we format the multiple choice task as N language modeling tasks, where N is the number of options. Specifically, for the combined context and question x, we directly concatenate each possible multiple-choice answer {y 1 , ..., y N } to x to form N different sentences. After passing the concatenated sequences through the model, we sum the logits of all tokens corresponding to each multiple-choice option to obtain a score proportional to its log-likelihood. These scores are then gathered into a size (1, N ) tensor, and cross-entropy loss relative to the correct multiple choice answer is used to train the model. Further details on the fine-tuning setup can be found in Appendix H.We evaluated the fine-tuned GPT-2 XL model on the Harry Potter dataset. To identify channels with statistically significant differences between the base and fine-tuned model, we calculated empirical p-values by comparing the true correlation value with 10,000 simulated ones obtained by permuting the brain data. Details of the algorithm can be found in Appendix K.Fine-tuned models are better aligned with the brain on both tasks. As illustrated in Figure 4A , the model fine-tuned on the social dataset exceeds the base model in performance across the majority of channels within the 50ms to 300ms time interval post word onset. Notably, this interval corresponds to the language processing time windows, as identified in §2.4. In a similar vein, the fine-tuned physical model exceeds the base model's performance in almost all channels during the 50-275ms interval post word onset (Figure 4B ). However, interestingly, almost all channels are worse than the base model outside this time window. This time selectivity may indicate that the improvements of the fine-tuned model are tailored towards linguistic comprehension rather than broader brain functionalities.Fine-tuning improves alignment more for words annotated with that category. We compared the reduction in prediction error for words annotated within each category and words outside each category by computing the difference in MSE between the model fine-tuned on the corresponding task and the base model. As demonstrated in Figure 5A , prediction errors for social words exhibit a significant reduction compared to non-social words 200-275ms post word onset. Additionally, there is a significant improvement in MSE for physical words over non-physical words 150-225ms post word onset (Figure 5B ). We also ran additional control experiments to check if MSE improvement is specific to words that match the category of the dataset on which the model was fine-tuned. Specifically, we evaluated the prediction improvement of physical words on the model fine-tuned on the social dataset, and vice versa (Appendix M).Improvements are not related to increased language-modeling ability. Prior work has found that LMs with lower perplexity can better predict brain activity [Schrimpf et al., 2021] . Therefore, additional fine-tuning may have improved the language model's ability to perform the LM task in general, leading to improved alignment. To rule out this possibility, we performed 3-fold crossvalidation on Harry Potter and the Sorcerer's Stone, excluding Chapters 9 and 10, which were used as data in this study. We trained the base model, as well as the fine-tuned models, on the train set in each fold with the language modeling objective, and found that the final average losses on the test sets were similar (See Appendix N for details).Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: The code implementation for all experiments is accessible via an online repository. The Harry Potter dataset is open-sourced. The Moth dataset, which is a component of a larger research initiative, will be publicly released once additional data collection is complete. Instructions on command and environment to reproduce experimental results can be found in the online repository.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
related work: 
 Numerous studies have found that LM hidden states can linearly map onto human brain responses to speech and text measured by MEG, EEG, and fMRI [Wehbe et al., 2014a , Hale et al., 2018 , Jain and Huth, 2018 , Abnar et al., 2019 , Jat et al., 2019 , Gauthier and Levy, 2019 , Toneva and Wehbe, 2019 , Caucheteux and King, 2022a , Jain et al., 2020 , Toneva et al., 2022 , Aw and Toneva, 2022] .At a more foundational level, studies have identified shared computational principles between LMs and human brains. Evidence suggests that both human brains and LMs are perpetually engaged in predicting the subsequent word [Schrimpf et al., 2021] . LM surprisal is found to be positively correlated with brain activation, reaching its peak approximately 400 ms post word onset [Goldstein et al., 2022] . This aligns well with N400, which denotes a decline in brain activation upon encountering unexpected words around 400 ms after word onset [Lau et al., 2009 , Parviz et al., 2011 , Halgren et al., 2002] . Moreover, LM representations can predict the hierarchy of brain responses [Caucheteux and King, 2022b, Caucheteux et al., 2023] . Despite this, Antonello and Huth [2022] have pointed out that a high correlation between brain activity and LMs does not necessarily imply that they operate under similar computational principles.We not only observe this LM-brain alignment but can also actively intervene in it. Research has demonstrated that the alignment between LMs and human brains can be improved by task-specific fine-tuning. A notable instance is the study by Schwartz et al. [2019] , where the fine-tuning of BERT using both fMRI and MEG signals enhanced its ability to predict fMRI responses. Importantly, this improvement was not participant-specific and could be transferred to hold-out individuals. Another study [Aw and Toneva, 2023] showed that task-oriented fine-tuning, particularly for narrative summarization, also facilitated better alignment with brain activity. Furthermore, altering the architecture of BERT such that it aligns better with the brain improves its performance on downstream NLP tasks [Toneva and Wehbe, 2019] . These findings suggest a potentially symbiotic relationship between enhancing task performance in LMs and boosting their alignment with brain responses.7 Conclusions, Limitations, and Future WorkWe explore a critical question connecting language models with human neural activity: How do LMs differ from human brains in processing language? We employed an LLM-based approach to automatically propose hypotheses explaining the elements of human brain responses that cannot be well explained by language models. Social/emotional intelligence and physical commonsense emerged as the two dominant themes. After fine-tuning GPT-2 XL on datasets related to these themes, we observed an improved alignment between LM predictions and human brain responses.Limited by the availability of datasets with aligned brain data, our study was conducted on a relatively narrow range of texts. While we observed consistent patterns across two language modalities, it is important to note that both datasets utilized were exclusively narrative stories. This limited scope raises the possibility that additional, undetected divergences exist, potentially obscured by the quantity of text and the methodology employed for hypothesis generation from the sentences. By developing more robust tools for pattern discovery and incorporating a wider array of textual materials, our approach can be adapted to more comprehensively address the question in future studies.Because of the typical low Signal-to-Noise Ratio (SNR) associated with MEG, we adopted a denoising technique [Ravishankar et al., 2021] that takes advantage of cross-subject correspondences to get an aggregated, denoised version of MEG responses. Specifically, this process involves modeling the MEG responses M t of subject t as a linear function of the MEG responses M s from a source subject s:Mt←s = Ŵt←s M s + bt←sWe estimated the target subject's MEG responses from all other subjects:Mt = 1 N -1 s∈S,s̸ =twhere S is the set of subjects and N is the number of subjects. These individual estimates are then aggregated to generate a denoised version of MEG responses:M = 1 N s∈S MtPrior research has shown that intermediate layers of language models often best predict human brain responses [Toneva and Wehbe, 2019 , Jain and Huth, 2018 , Oota et al., 2022b] . Therefore, we selected the layer that best predicts brain responses. Figure 6 illustrates the Pearson correlation between actual MEG responses and those predicted by LM embeddings across layers and time points relative to word onset. We used the average correlation across all words and time windows as the metric to select the best layer. Echoing previous findings, we confirmed that intermediate layers exhibit higher correlations, with layer 7 being the best at predicting brain responses in GPT-2 XL. Similarly, for Llama-2, layer 3 was identified as the most predictive. Given the vast amount of text in the datasets, manually discovering patterns becomes challenging.Figure 7 provides an illustrative example by presenting a set of sample sentences that are color-coded based on the magnitude of their prediction error. This visualization demonstrates the complexity involved in formulating hypotheses from observations.He had been looking forward to learning to fly more than anything else."Of course he has," said Ron, wheeling around.But Neville, nervous and jumpy and frightened of being left on the ground, pushed off hard before the whistle had touched Madam Hooch's lips. most divergent least divergent 1.3. In addition to the best layer, we also performed analyses on the last layer of the language model.The spatial-temporal pattern of predictions observed in the last layer (Figure 8 ) is similar to that of the best layer, However, there is a notable difference in the magnitude of the values. Specifically, the maximum correlation in the last layer is lower, decreasing from 0.53 in the best layer to 0.43. Language regions are effectively predicted in language processing time windows (refer to §2.4 for more details).We also generate hypotheses from the predictions of the last layer (Table 4 ). These hypotheses exhibit similarities with those derived from the best-performing layer, notably in their inclusion of emotions and social interactions. However, a distinctive aspect of these hypotheses is their association with supernatural and magical elements. Additionally, we observe the emergence of figurative language, aligning with previous research that indicates language models underperform humans in both the interpretation and generation of figurative language [Chakrabarty et al., 2022 , Liu et al., 2022] and the correct representation of idiomatic phrases [Dankers et al., Liu and Neubig, 2022 ].Although GPT-2 is a widely used language model in brain research, it's not the latest model in the field. Models with more parameters and advanced training methods could show different results. Therefore, we replicated some analyses on Llama-2 [Touvron et al., 2023b] . We used the implementation in the HuggingFace library [Wolf et al., 2020b] with 7B parameters. As Figure 6B shows, early layers exhibit high correlations, with layer 3 identified as having the highest correlation. The spatial-temporal pattern of predictions in layer 3 of Llama-2 (Figure 9 ) closely resembles those found in GPT-2 XL. This similarity implies that both language models effectively capture the representations of words. Language regions are effectively predicted in language processing time windows (refer to §2.4 for more details).Hypotheses from the predictions of layer 3 of Llama-2 7B can be found in Table 5 . Interestingly, the focus of these hypotheses is primarily on physical objects and events. In comparison to the hypotheses produced by GPT-2 XL, there is a notable absence of social and emotional aspects, suggesting that Llama-2 7B could have a more advanced comprehension of social and emotional contexts. The prompt for the proposer is: {A_block} {B_block} The dataset includes two chapters from "Harry Potter and the Sorcerer's Stone". The two groups are generated based on the difference between language model and human responses to these sentences. The Group A snippets sentences where language models and humans show divergent responses, while the Group B snippets sentences where language models and humans show similar responses. I am a literary analyst investigating the characteristics of words. My goal is to figure out which sentences induce different responses for language models and human responses.Please write a list of hypotheses about the datapoints from Group A (listed by bullet points "-"). Each hypothesis should be formatted as a sentence fragment.Here are three examples.-"{example_hypothesis_1}" -"{example_hypothesis_2}" -"{example_hypothesis_3}" Based on the two sentence groups (A and B) from the above, more sentences in Group A ... G Manual Hypothesis VerificationWe recruited 10 participants through Qualtrics. Of these, 9 participants completed 100 trials each, while one participant completed 500 trials. In each trial, participants were presented with a hypothesis selected either from the top 10 or bottom 10 hypotheses generated from the Harry Potter dataset, along with a pair of sentences -one from D0 (the divergent sentence set) and the other from D1 (the convergent sentence set) -in a randomized order. The task for participants was to determine which sentence aligned more closely with the given hypothesis, choosing between "More True for Sentence A", "More True for Sentence B", or "Equally true". Note that a given hypothesis is not expected to apply to all divergent sentences (e.g., it might suffice for a sentence to be emotionally intense or grammatically complex to be divergent) so it is expected that some of the responses will be "Equally true". A screenshot of the experiment can be found in Figure 10 .We constructed a contingency table with dimensions (Top Hypothesis, Bottom Hypothesis) by (Prefer Divergent, Equal, Prefer Convergent) (Table 6 ). A binomial test conducted on the contingency table (looking only at the "Prefer Divergent" and "Prefer Convergent" responses) showed that divergent sentences were more likely to be chosen over convergent sentences on average (p < 10 -10 ). A Chi-square test revealed statistically significant differences in the distribution of responses between the top and bottom hypotheses (p = 0.024). Additionally, we utilized the Chi-square test to compare the frequency of "Prefer Divergent," "Equal," and "Prefer Convergent" responses in two conditions. Notably, a preference towards divergent sentences was observed in the top hypotheses condition compared to the bottom hypotheses (p = 0.093). In contrast, in the bottom hypothesis condition, there was a marked preference for "equally true" (p = 0.008). No significant difference was observed in the preference for convergent sentences between the conditions (p = 0.676).H.1 Computational Details GPT-2 XL was trained separately on each of the two datasets in subsection 5.1 on 4 A6000 GPUs with 16-bit quantization and a batch size of 1 per GPU. Deepspeed with ZeRo stage 2 optimization was used in order to parallelize training [Rasley et al., 2020] . The Adam optimizer was used with a learning rate of 1e-5, betas of (0.9, 0.999), epsilon of 1e-8, and no weight decay. Models were trained with early stopping with a patience of 3 [Kingma and Ba, 2017].Let x i represent the concatenation of the context, if applicable, and the question. Then for each answer choice y i , we concatenate it with the question and context, and feed it to the model to obtain a sequence of logits.EQUATIONThen we sum the logits corresponding to the sequence, where t ∈ [1, T ] represents the total length ofEQUATIONFinally, we take the cross-entropy loss of these values relative to a one-hot encoding of the correct option, where t i = 1 if option i is correct, or else 0.P i = exp(logit i ) N j=1 exp(logit j ) L = - N i=1 t i log(P i ) H.2We note that the performance of the final model may not approach that of GPT-2 XL fine-tuned with an output size of N denoting each option, as we keep the output dimension the same as the size of the vocabulary. However, we report the final accuracy achieved by each model on the original datasets here. To decide which category a word belongs to, we employed three raters who used binary coding to indicate if a word belonged to the target category. The consistency among raters was evaluated using Krippendorff's alpha. Their consistency was 0.54 for social/emotion and 0.87 for physical. Finally, if at least two out of the three people annotated a word as fitting a category, we counted it as belonging to that category.• Include words that depict the emotions of characters and/or social interactions.• Exclude words that suggest emotions or social interactions indirectly. For instance, "slam the door" shouldn't be annotated.• Annotate words referring to tangible entities, such as characters (people) and physical objects.• Do not annotate words that represent concrete ideas but lack physical substance, like "laughter".• Pronouns should also be excluded.We give some examples of the two phenomena in the dataset according to the annotations. Words of that category are marked in bold. model fine-tuned on the physical dataset, we observed a marginal, though not statistically significant, boost in performance with social words (Figure 13C ). We propose that this marginal improvement could be attributed to the presence of social and emotional knowledge embedded within the physical dataset. To substantiate this hypothesis, we conducted a thorough review of the physical dataset and identified items that indeed pertain to social or emotional scenarios.Examples of such items include:• how do you give a surprise party?• To help your child feel less afraid when they're going to sleep• How can you get a child to smile in a photo?• To help a friend feel better when they are sad• how to avoid danger? • To determine if someone has romantic feelings for youThese findings led us to conclude that the marginal improvement in processing social words by the model fine-tuned on the physical dataset may result from exposure to social and emotional content.We perform 3-fold cross-validation on the remaining chapters of the Harry Potter book (excluding chapters 9 and 10), where we randomly shuffle paragraphs and assign to train:validation:test sets respectively 77%, 16.5%, and 16.5% of the data. Paragraphs that exceeded the context length were excluded. Both the base GPT-2 XL model as well as each model fine-tuned on the three domains were trained to predict the next word for 3 epochs, with the same hyperparameters used in Appendix H. Results on the test set for each fold are listed below. The average negative-log-likelihood loss per token at the end of training is reported in Table 8 . This study represents a significant intersection between Neuroscience and Machine Learning, striving to push the boundaries of machine learning models while deepening our understanding of how the human brain functions. In a broader context, this research lays the groundwork for future breakthroughs in the field of neuroscience and for making human-computer interfaces more efficient and intuitive.However, the development of human-computer interfaces may cause problems in privacy and security, as these interfaces often require the collection and processing of personal data, increasing the risk of data breaches and unauthorized access. There are also ethical concerns regarding the potential for surveillance and the impact on employment, as advanced interfaces might automate tasks currently performed by humans, potentially displacing workers.NeurIPS Paper ChecklistQuestion: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: The paper specifies all the training and test details necessary to understand the results. The training of the encoding model is detailed in subsection 2.2, and the fine-tuning details of the language models are specified in Appendix H. Additionally, the code for training these models is available in the online repository provided.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: The paper provides detailed information on the computational resources required to reproduce the experiments, including the type and number of GPUs used, as well as the specific hyperparameters employed for fine-tuning the language model.Guidelines:• The answer NA means that the paper does not include experiments. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: The societal impacts of the work are discussed in detail in Appendix O, covering both the potential positive and negative effects on society.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Not applicable. The paper does not involve data or models that carry a high risk of misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.Data and code are available at: https://github.com/FlamingoZh/divergence_MEG 38th Conference on Neural Information Processing Systems (NeurIPS2024).Validity measures the difference in certainty that the hypothesis is true between the two corpora, seeZhong et al. [2023] for more details. 
 
 Appendix:  


Decision: The authors employ a data-driven approach to identify specific words and linguistic contexts where a brain encoding model fails to accurately predict neural responses to language. They discover that factors such as social, emotional, physical, and linguistic complexity contribute to poorer performance of the brain encoding model. To address this, they fine-tune the base GPT model to enhance its capabilities in social and emotional reasoning tasks. This adjustment leads to improved brain encoding predictions for these specific words during the relevant language processing time frame, approximately 75 to 400 milliseconds.

The reviewers raised many technical issues, including experiments with only two models, the possibility that newer models would demonstrate different performance and various technical questions about the experiments. Some of these issues have been solved in the discussion period and all four reviewers are leaning towards acceptance. Overall, this is an empirical paper on an important topic and I believe it makes a nice contribution that justifies publication. 

As a side note - the paper is missing a very relevant citation that also shows that fine tuning to a specific domain can substantially improve alignment with the human brain, although with fMRI data:

Tikochinski, R., Goldstein, A., Yeshurun, Y., Hasson, U., & Reichart, R. (2023). Perspective changes in human listeners are aligned with the contextual transformation of the word embedding space. Cerebral Cortex, 33(12), 7830-7842.‏

I strongly encourage the authors to cite and compare to this paper.

Reviews: review: paper_summary: paper_summary: The authors use a data-driven method to generate hypotheses about particular words/linguistic contexts in which a brain encoding model does not accurately predict the brain response to language. they find that social/emotional/physical complexity, along with linguistic complexity, are all associated with worse brain encoding performance. They fine-tune the base GPT model to perform better at social/emotional reasoning tasks and find that this yields an improvement in brain encoding for these particular words within the relevant language processing window (~75-400 ms).


Update: I have read the other reviews and author rebuttals and will keep my score as-is.
strength_weakness: strength_weakness: - The fine-tuning intervention needs to be appropriately baselined, for example by fine-tuning on other tasks which don't match the hypotheses about what is driving brain encoding performance. If fine-tuning on an appropriate counterfactual sample doesn't improve brain encoding performance, this will strengthen the positive finding. (I'm not totally clear on the discussion in Appendix M -- are you saying that fine-tuning through language modeling on Harry Potter doesn't yield brain encoding improvements? This is a step in the right direction, I think.)
- The effect sizes of improvements here are very small; the results would be much stronger if you could reproduce this effect on a second dataset.
questions: questions: It seems like there is a lot more going on in the example sentences (Appendix C) than the hypotheses cover. For example, many function words are colored as "most divergent" despite not fitting the leading hypotheses. Can you provide a broader sample that might convince the reader of the coverage of the model-derived hypotheses?
limitations: limitations: Yes
paper_summary: paper_summary: This paper explores the differences between LMs and the human brain in processing language. The authors conduct experiments using MEG data from reading and listening tasks to investigate elements of MEG responses that LMs cannot adequately explain. LLMs are used to automatically generate hypotheses, identifying the domains where LMs lack knowledge compared to the human brain. The authors then fine-tune LMs on these specific domains to improve the LMs-brain alignment.
strength_weakness: strength_weakness: Only two LMs are used in the experiment. Additionally, GPT-2 is somewhat outdated compared to recently released open-source LLMs, as it shows limited language understanding and reasoning capability, making it far from achieving human-level AGI. The focus should be more on LLMs such as Mixtral and Gema.

More details about the prompts and API usage should be discussed for the proposer and verifier LLMs.
questions: questions: The relatively narrow datasets used in the paper, focusing on human dialogue and stories, which intuitively consist of emotion representation, could easily introduce bias.
limitations: limitations: See Weakness and Questions
paper_summary: paper_summary: Language models are known to predict MEG signals in humans during reading. In this work, the authors explored for what "kinds" of texts are language models bad at predicting MEG signals. The authors used a novel method to propose possible hypotheses, and found multiple possible categorizations of weak prediction texts. Focusing on social knowledge and spatial commonsense, the authors find that finetuning the language model on texts from these respective domains improves the prediction accuracy.
strength_weakness: strength_weakness: Many aspects mentioned in the work where the language model are not performant on, for example common sense and social reasoning, have large leaps forward in more modern models. A future work could focus on using more modern language models with stronger capacity.
questions: questions: 1. Temporal signal prediction is convincing, another aspect that is interesting is the spectral structure of the predictions.
2. Do layers other than 7 display the same pattern of lower prediction accuracy? Do you observe this consistently across layers, or just at layer 7?
limitations: limitations: Properly addressed.
paper_summary: paper_summary: The paper studies the differences in language representations in LLMs (GPT-2 XL) and the human brain (MEG activity), by using LLM representations to predict MEG activity. They use an LLM-based approach to identify two domains where LLM representations do not capture brain activity well: social/emotional intelligence and physical commonsense. They also show that fine-tuning GPT-2 XL on these domains can improve their alignment with human brain responses.
strength_weakness: strength_weakness: From most to least significant:
1. They used a relatively low layer (layer 7) of GPT2-XL when making the claim that LMs have poor brain alignment for high-level properties (social/emotional intelligence, physical commonsense). They should have studied a higher layer instead/too.
- The paper argues they identified two domains that LMs do not capture well: social/emotional intelligence and physical commonsense.
- However, they used layer 7 of GPT-2 XL, a relatively low layer since GPT-2 XL has 48 layers. They say they use this because it is the best at predicting brain activity, but this seems much lower than prior research: Layer 8 of 12 in BERT and Layer 12 of 19 in T-XL [1]
- Prior work has shown that lower layers of LLMs capture lower-level linguistic properties, while higher layers capture higher-level properties. Prior work also showed that "LM representations can predict the hierarchy of brain responses" (lines 290-291 in their paper).
- In this case, is it possible that the higher layers (30+) of the same GPT-2 XL model will achieve high brain alignment for social/emotional intelligence and physical commonsense? 
- However, re-evaluating more layers is expensive, and I am not requesting them to do this if they can address the comment in writing. 
2. Results in Table 1 are not convincing in supporting their overall claims. The LLM-based system of proposers and verifiers generated hypotheses on how the top 100 least-predicted sentences differ from the top 100 well-predicted sentences. However, it seems that 331 of 708 human participants did not agree with the top hypotheses suggested by the LLM system (Table 1), i.e., either found the sentences to be "Equal" or "Convergent". This casts doubt that the two sets of sentences actually differ in terms of social/emotional intelligence and physical commonsense. This, in turn, casts doubt on their overall claim that LLMs do not capture social/emotional intelligence and physical commonsense well.
3. Additional training data as a potential confound
- The paper showed that training LLMs on a dataset for emotional intelligence improves brain alignment. I agree that by itself this is interesting. However, is this because of: (A) statistical similarities between the text in this training dataset and the MEG stimuli, especially for words related to emotional intelligence, or (B) LLM gains better emotional intelligence understanding, as labels were provided and it learns to predict emotions correctly?
- Alternative reasons that finetuning on social dataset improved brain alignment on social words: (a) because the model was provided additional training data? (b) because the additional dataset was higher quality?
- Possible control experiment: training the model (using language modeling) on the same dataset but without associating each question with the correct label. Basically, do not tell the model which is the correct answer (no supervision).
- It is possible that the authors can answer these questions using their further experiments in Appendix I and M. Still, I hope they explicitly address, within the paper, the issue of additional training data as a potential confound.

[1] M. Toneva and L. Wehbe, “Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain).” 2019
questions: questions: 1. Why is layer 7 of GPT-2 XL the best at predicting brain responses for this dataset? Layer 7 of 48 seems relatively low as opposed to intermediate, whereas prior research showed that intermediate layers are best (lines 115-116 in your paper).
limitations: limitations: The limitations I identified have been mentioned in the paper.

Comments: comment: I have read the authors' rebuttal which has addressed my concerns to some degree. I have raised my rating accordingly. Thanks.
My questions are adequately addressed. I will maintain my current score.
I will raise my score from 6 to 7. The authors provided convincing clarifications for most of the weaknesses (and their sub-points) I raised.
1. Best Performing Layer (resolved)
- Thanks for the clarification on MEG vs fMRI, and larger vs smaller models.
2. Human Study (resolved)
- Thanks for the clarification, especially that "There are a total of ten hypotheses, and each sentence may only satisfy one or two of them." This helped me to understand why so many participant ratings are "Equal" or "Convergent", even for the set of top hypotheses.
3. Fine-Tuning (resolved)
- Thanks for the clarifications.