Instruction-tuning Aligns LLMs to the Human Brain

Authors: khai  loong aw
syrielle  montariol
badr  alkhamissi
martin  schrimpf
antoine  bosselut
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-22 14:25:13

Abstract: Instruction-tuning is a widely adopted finetuning method that enables large language models (LLMs) to generate output that more closely resembles human responses. However, no studies have shown that instruction-tuning actually teaches LLMs to process language in a similar manner as humans. We investigate the effect of instruction-tuning on aligning LLM and human language processing mechanisms in two ways: (1) brain alignment, the similarity of LLM internal representations to neural activity in the human language system, and (2) behavioral alignment, the similarity of LLM and human behavior on a reading task. We assess 25 vanilla and instruction-tuned LLMs on three datasets involving humans reading naturalistic stories and sentences, and find that instruction-tuning generally enhances brain alignment (~6%), but has no similar effect on behavioral alignment. To identify factors underlying this improvement in brain alignment, we compute correlations between brain alignment and various LLM properties, such as model size, problem-solving, and world knowledge understanding. Notably, we find a strong positive correlation between brain alignment and model size (r = 0.95), as well as performance on tasks requiring world knowledge (r = 0.81). Our results demonstrate that instruction-tuning LLMs improves both world knowledge representations and brain alignment, suggesting that the mechanisms that encode world knowledge in LLMs also improve representational alignment to the human brain.

Main paper: 
 introduction: 
 Instruction-tuning is a widely adopted method that finetunes large language models (LLMs) on datasets containing many examples of different tasks and their descriptive instructions, enhancing their ability to generalize to previously unseen tasks by learning to follow provided instructions (Wang et al., 2022c) . Despite costing only a small fraction of compute relative to pretraining (Chung et al., 2022) , instruction-tuning yields large performance improvements on reasoning and problem-solving benchmarks. This generalization has allowed LLMs to tackle open-world reasoning tasks previously achievable only by humans (Zhang et al., 2023) while only using a few (or zero) task-specific training examples.In addition to teaching LLMs to understand and follow human instructions, instructiontuning also improves the ability of LLMs to mimic human-written ground-truth outputs. This fluency allows them to produce more controllable and predictable output that is deemed (1) more desirable to human evaluators (Zhang et al., 2023; Chung et al., 2022; Wang et al., 2022b) , (2) more aligned to human values (Chia et al., 2023) , and (3) more stylistically similar to human outputs (Dasgupta et al., 2022; Safdari et al., 2023) .Consequently, instruction-tuning yields LLMs more similar to humans in both capability and output resemblance. From a neuroscience perspective, this begs the question: Does instruction-tuning make LLMs more functionally similar to the human language system? Previous work has shown that models with higher task performance are more aligned to the human language system (Schrimpf et al., 2021; Goldstein et al., 2022; Caucheteux & King, 2022) , hitting the estimated noise ceiling1 on some datasets. However, there has been no similar study on how instruction-tuning affects alignment with the human language system. In this work, we measure the effect of instruction-tuning on the alignment between language mechanisms in LLMs and the human brain in two ways: (1) brain alignment, an "internal" metric that assesses how well internal feature representations of LLMs match neural representations in the human language system, and (2) behavioral alignment, an "external" metric which evaluates the similarity between LLM and human behavioral measurements. In our study, both LLMs and humans are presented with the same language stimuli comprised of naturalistic stories and sentences. For LLMs, we record their internal representations and per-word perplexity. For humans, we use previously recorded brain activity data from functional magnetic resonance imaging (fMRI) experiments and per-word reading times.To measure brain alignment, we use the Brain-Score (Schrimpf et al., 2018) linear predictivity metric, assessing how well LLM representations predict human brain activity in response to the same language stimuli (Jain & Huth, 2018; Toneva & Wehbe, 2019; Schrimpf et al., 2021; Oota et al., 2023) , using data from three neural datasets: Pereira et al. (2018) , Blank et al. (2014) , and Wehbe et al. (2014) . To evaluate behavioral alignment, we use a benchmark in Brain-Score which calculates the Pearson correlation between LLM per-word perplexity and human per-word reading times from the Futrell et al. (2018) dataset. Perplexity for LLMs and reading times for humans offer insights into comprehension difficulty (Ehrlich & Rayner, 1981; Hale, 2001; Smith & Levy, 2013) , allowing us to examine whether LLMs match humans in their patterns of words and sentences they find challenging or surprising. Because models vary in their brain and behavioral alignment across different architectures and training objectives (Schrimpf et al., 2021) , we estimate the effect of instruction-tuning by evaluating 8 vanilla LLMs and 17 LLMs that were further instruction-tuned, and report a significant increase in brain alignment due to instruction-tuning.To investigate why instruction-tuning increases alignment to human brain activity, we study the relationships between brain alignment and various LLM properties. Specifically, we compute Pearson correlations between an LLM's brain alignment and its properties, including next-word prediction (NWP) ability, model size, a range of problem-solving abilities, and world knowledge spanning different domains. We evaluated the latter two properties using the Big-Bench Hard benchmark (BBH; Suzgun et al., 2022) and the Massive Multi-task Language Understanding benchmark (MMLU; Hendrycks et al., 2021) , respectively.We report three major findings:1. Instruction-tuning generally aligns LLM representations to human brain activity, increasing brain alignment by 6.2% on average (Figure 1 ).2. Investigating the factors underlying LLM-brain alignment, we find that brain alignment is strongly correlated with world knowledge (r = 0.81) and model size (r = 0.95) (Figure 2 ).3. Surprisingly, our results indicate that instruction-tuning LLMs generally does not enhance behavioral alignment with human reading times. Furthermore, behavioral alignment on this dataset is poorly correlated with all other measures we investigate, including task performance and model size (Figure 3 ). 
background & related work: 
 Effect of Instruction-tuning on LLMs. Instruction-tuning is an effective method for enhancing LLM capability and controllability. It trains LLMs using pairs of human instructions and desired outputs. The benefits of instruction-tuning can be categorized into three key aspects (Zhang et al., 2023) : (1) it bridges the disparity between the pretraining objective of LLMs (next-word prediction) and the goal of accurately following human instructions, (2) it achieves greater control and predictability of model behavior compared to standard LLMs, allowing researchers to make them more similar to humans in both capability and output similarity (Chia et al., 2023; Dasgupta et al., 2022; Safdari et al., 2023) , and (3) it often costs only a small fraction of compute relative to pretraining, enabling LLMs to swiftly adapt to target domains (Chung et al., 2022) . We contribute to this research area from a neuroscience perspective, by studying whether instruction-tuning makes LLMs more aligned to the human language system in terms of brain and behavioral alignment.Effect of Finetuning on Brain alignment. Prior works have studied how finetuning affects LMs' alignment to human brain activity. These include finetuning on a wide range of downstream NLP tasks (Oota et al., 2022) , finetuning to summarize narratives (Aw & Toneva, 2023) , and finetuning to directly predict brain activity recordings (Schwartz et al., 2019) . These studies aim to use brain alignment to study how finetuning affects LMs and their representations. (Taori et al., 2023) . The input field is optional for certain instructions.instruction-tuning aligns LLM representations to human brain activity. We also investigate why instruction-tuned LLMs align to brain activity by testing the correlation of brain alignment with various world knowledge domains and problem-solving abilities.LM properties linked to Brain alignment. One exciting research area focuses on disentangling the contribution of various LM properties towards brain alignment. These include studying how brain alignment is driven by next-word prediction ability (Schrimpf et al., 2021; Caucheteux & King, 2022) , multi-word semantics (Merlin & Toneva, 2022) , performance on various NLP tasks (Oota et al., 2022) , and model size (Antonello et al., 2023) .To contribute to this growing body of work, we use instruction-tuned LLMs. They are especially useful as they have been trained to respond to a standard question-answer format, allowing us to evaluate LLMs on a wide array of tasks and in a more fine-grained manner.We expand this area of research by demonstrating that world knowledge is a key property underlying LLM-brain alignment. 
language models: 
 We evaluate the brain alignment of 25 large language models (LLMs) from two model families: T5 (Raffel et al., 2020) and LLaMa (Touvron et al., 2023a) . T5 models are encoderdecoder LLMs pre-trained using a masked infilling objective on the Colossal Common Crawl Corpus (C4), a corpus of 356 billion tokens, and then further finetuned on a multi-task mixture of unsupervised and supervised tasks converted into a text-to-text format. We use all five T5 models, with sizes between 77M to 11B. LLaMA models (Touvron et al., 2023a) are decoder-only LLMs trained on 1.6 trillion tokens from a mixture of corpora including C4, English CommonCrawl, Wikipedia, GitHub. We use the 7B, 13B, and 33B versions.For the instruction-tuned variants of T5 models, we use a variety of models finetuned on FLAN (15M examples for 1,836 different tasks accompanied by instructions, Chung et al., 2022) , Alpaca (52K instruction-following examples generated with methods inspired by Self-Instruct, Wang et al. (2022a) , Taori et al., 2023) , and GPT4ALL (437K instruction-following examples generated with GPT-3.5-turbo, Anand et al., 2023) datasets. For the instructiontuned variants of LLaMa, we use Vicuna's 7B, 13B, and 33B models (Chiang et al., 2023) , which are finetuned on user-shared conversations. We also use StableVicuna-13B, which further refines Vicuna-13B using reinforcement learning from human feedback (RLHF) (Ouyang et al., 2022) on a range of conversational and instructional datasets. We also use the 7B version of Alpaca (Taori et al., 2023) . We provide more details in Appendix A.We also tested 8 models from the GPT2 family; four vanilla models (Small, Medium, Large, XL) and their corresponding versions further instruction-tuned on Alpaca. However, the models performed close to random on reasoning and world knowledge benchmarks (BBH, MMLU). We hypothesize this is due to their size (all < 1.5B parameters), making fine-tuning on a relatively small instruction-following dataset insufficient. Moreover, GPT2 was not originally pretrained to perform instruction-following tasks (unlike T5 models). We include their MMLU, BBH, brain and behavioral alignment results in Appendix G, H, and L. 
brain alignment: 
 Brain alignment refers to the method of evaluating the similarity between LLM representations and human brain activity (Figure 1A and Appendix C). This relies on fMRI recordings of human subjects while they read language stimuli on potentially any topic (here: Pereira et al., 2018; Blank et al., 2014; Wehbe et al., 2014) . The same language stimuli from prior brain recordings are provided as input to LLMs, whose intermediate layer activations are extracted as their representations of the language stimuli. We follow a general approach previously used in several works (Schrimpf et al., 2018; Jain & Huth, 2018; Toneva & Wehbe, 2019; Schrimpf et al., 2021; Oota et al., 2023; Aw & Toneva, 2023) . Specifically, we use the linear predictivity metric implemented in Brain-Score (Schrimpf et al., 2018) , which fits a linear regression model from LLM layer activations to fMRI responses in the human language system. We then apply this linear model to predict held-out fMRI responses from the original corpus of recordings, and compute brain alignment as the Pearson correlation between the predicted and actual fMRI responses. For each LLM, we evaluate its brain alignment for every layer (e.g., LLaMA-7B has 32 layers), and use the highest value as the LLM's brain alignment value, following Schrimpf et al. (2018) .We use three fMRI datasets to measure the brain alignment of LLMs. Each dataset involves a different set of human participants and a different set of language stimuli.PEREIRA2018 (experiments 2 and 3 from Pereira et al., 2018) : In experiment 2, 9 participants read 384 sentences taken from 96 text passages. In experiment 3, 6 participants read 243 sentences from 72 text passages. Each sentence was displayed for 4 seconds on a screen.BLANK2014 (Blank et al., 2014) : The data consists of fMRI recordings of 5 human participants listening to 8 naturalistic stories from the Natural Stories Corpus (Futrell et al., 2018) .WEHBE2014 (Wehbe et al., 2014) : The data includes fMRI recordings of 8 human participants reading chapter 9 of the book Harry Potter and the Sorcerer's Stone (Rowling et al., 1998) .Participants read the chapter at a fixed interval of one word every 0.5 seconds.First, we study the effect of instruction-tuning on LLM brain alignment. We compute each LLM's average brain alignment as the mean of its brain alignment on the 3 neural datasets. We find that instruction-tuning improves brain alignment by an average of 6.2% across tested LLMs (Figure 1B ). This holds across all three neural datasets, with improvements of +6.9% on PEREIRA2018, +8.0% improvement on BLANK2014, and +3.8% on WEHBE2014 (Figure 1C ). Moreover, a smaller instruction-tuned model can attain higher brain alignment than a larger vanilla model from the same family (e.g., Alpaca-7B v.s. LLaMa-13B, see Appendix G). However, some of the larger models (T5-XL, LLaMA) have minimal or no increases, possibly because the vanilla models are already close to the noise ceiling (Appendix G).Next, to longitudinally study how instruction-tuning aligns LLM representations to brain activity, we separately instruction-tune a LLaMA-7B model on the Stanford Alpaca instruction dataset (Taori et al., 2023) for 3 epochs. By evaluating checkpoints regularly during training, we find that instruction-tuning progressively improves brain alignment (Figure 1D , Appendix K). We also disambiguate the effect on brain alignment of (1) the instructionfollowing ability gained from instruction-tuning and (2) added training data. We fine-tune LLaMA-7B with the same process and data, but remove the instruction from each training sample. Brain alignment of this ablated model increases during fine-tuning but stays lower than its instruction-following counterpart (Figure 1D ), indicating that brain alignment improvements are due to both factors. In Appendix J, we also report preliminary results using recent models from the Gemma (Team et al., 2024) and LLaMA-2 (Touvron et al., 2023b) families and observe similar trends. Further, these trends hold across different similarity metrics besides Linear predictivity, e.g., RSA and CKA (Appendix J).To study why LLMs and human brains align in their representations, we compute the Pearson correlation between LLM brain alignment and various properties of LLMs: performance on a benchmark testing various reasoning abilities (BBH; Suzgun et al., 2022) , performance on a benchmark testing world knowledge in various domains (MMLU; Hendrycks et al., 2021) , language modeling ability, and model size. Brain alignment (r) For both benchmarks, we follow the category classifications from the original papers. We perform the evaluations using the instruct-eval repository2 with default settings (3-shots for BBH, 5-shots for MMLU) and preset prompts. We measure the Pearson correlation (and its p-value) between LLM-brain alignment and performance in each category of MMLU and BBH. We obtain p-values after false discovery rate (FDR) correction.World Knowledge. We find that brain alignment is significantly and strongly correlated with world knowledge. Brain alignment is highly correlated with the MMLU Overall score (r = 0.81, p < 0.001, Figure 2A ), which reports the mean performance across all world knowledge subject domains on MMLU. Similarly, brain alignment is also strongly correlated with performance on tasks in the world knowledge category of BBH (r = 0.68, p < 0.005; Figure 2B ). Notably, brain alignment is poorly correlated with all other dimensions of BBH (see Table 2 ), though this could also be due to limitations of the tested models, as indicated by their low raw performance scores on some tasks. Overall, our results provide a strong signal that more accessible representations of world knowledge are a key factor in aligning LLM representations to human brain activity.Language Modeling Ability. Prior works have shown correlations between brain alignment and next-word prediction (NWP) ability (Caucheteux & King, 2022; Schrimpf et al., 2021) . We find similar results for correlation between brain alignment and NWP loss (r = -0.54, p < 0.05, Appendix I). Interestingly, the strength of the correlation is weaker than that between brain alignment and world knowledge performance (r = 0.81). This suggests that world knowledge understanding is a better predictor of brain alignment than NWP ability.Model Size. Finally, we find that brain alignment is significantly and strongly correlated with model size (r = 0.95, p < 0.001, Figure 2C ), as measured by the logarithm of the number of model parameters. Schrimpf et al. (2021) observe such a pattern for language models, and we find the pattern holds for instruction-tuned models, and models trained at a larger scale than their study (7B+ parameters). However, model size alone does not determine brain alignment. Our results show that smaller instruction-tuned LLMs can have greater brain alignment than larger vanilla models. For example, LLaMA-13B obtains brain alignment of 0.220, Vicuna-13B obtains 0.229, LLaMA-33B obtains 0.227, and Vicuna-33B obtains 0.232. Hence, Vicuna-13B has greater brain alignment than LLaMA-33B, due to instruction-tuning, despite being less than 40% its size. We observe a similar trend in another four models: T5-base, Flan-T5-base, T5-large, Flan-T5-large. Also, prior works showed that large random models achieve poor brain alignment (Schrimpf et al., 2021) . These results demonstrate there are LLM properties aside from model size that contribute significantly to brain alignment.Neural datasets. Brain alignment is strongly correlated with world knowledge and model size on PEREIRA2018 and WEHBE2014, but less strongly correlated on BLANK2014, possibly because BLANK2014 has fewer participants (N = 5) leading to greater noise in the results. 
behavioral alignment: 
 In the previous section, we show that instruction-tuning aligns the internal representations of LLMs to human brain activity (Section 4.1). In this section, we explore whether instructiontuning also aligns LLM behavior to human behavior.Following the approach of Schrimpf et al. (2021) implemented in the Brain-Score package (Schrimpf et al., 2020) , we measure behavioral alignment by evaluating the similarity between LLM per-word perplexity and human per-word reading times given the same language stimuli (Appendix D). We use the self-paced reading times dataset from Futrell et al. (2018) , consisting of the reading times of 179 human participants recorded while they were visually presented with 10 naturalistic stories. We provide language stimuli from this data as input to LLMs and evaluate behavioral alignment by computing the Pearson correlation between per-word LLM perplexity and per-word human reading times.One potential concern in the behavioral alignment methodology is that human reading time is influenced by word length; longer words may take more time to read, even if they are not more challenging or surprising. Following Schrimpf et al. (2021) , we address this by computing the surprisal of a multi-token word as the sum of the surprisals of its individual tokens. Importantly, we use consistent tokenizers when comparing vanilla LLMs against their instruction-tuned versions, thereby controlling for the effect of word length when evaluating the impact of instruction-tuning on behavioral alignment. Using the same models as in Section 4, we compare the behavioral alignment of each instruction-tuned LLM against its vanilla counterpart. Our results indicate instructiontuning generally does not improve behavioral alignment to human reading times (Figure 3A ). For half of the LLMs tested, it results in no change or reduced behavioral alignment. Our results align with recent studies (Gao et al., 2023; Steuer et al., 2023) that show improving LLM performance does not necessarily improve alignment with human reading behavior.Next, we test the correlation between LLM behavioral alignment and model size, next-word prediction ability, various reasoning abilities (measured by BBH), and world knowledge across various domains (measured by MMLU). Contrary to our findings on the brain alignment correlations with model size and world knowledge (Section 4.2), we find that LLM behavioral alignment is not correlated with these factors: world knowledge (p = 0.76, Figure 3B ), model size (p = 0.31, Figure 3C ), next-word prediction loss for T5 models (p = 0.54, Figure 3D ), and next-word prediction loss for LLaMA models (p = 0.21, Figure 3E ). 
implications for nlp: building llms: 
 Using brain alignment for LLM interpretability. Prior works have used human brain activity to interpret models (Dong & Toneva, 2023) and build more performant models (Dapello et al., 2020; Safarani et al., 2021; Dapello et al., 2022) . Instruction-tuning has emerged as a breakthrough technique to improve LLM abilities and quality of outputs, and allows LLMs to adapt to new tasks with minimal task-specific training. However, how instruction-tuning alters LLM internal representations to achieve these improvements remains an open question. Brain activity provides a neuroscientific angle to investigate this question. Our results show that instruction-tuning improves both performance on world knowledge benchmarks and human brain alignment, suggesting that the mechanisms that encode world knowledge in LLMs also align LLM representations to the human brain.Instruction-tuned LLMs are useful for studying LLM properties underlying brain and behavioral alignment. To identify why LLM and human brains share representational similarities, prior work has mostly focused on high-level properties such as model size (Antonello et al., 2023) , as well as external behaviors such as predicting missing words (Schrimpf et al., 2021; Caucheteux & King, 2022) . However, a key to understanding these similarities is to identify internal properties of LLMs that underlie brain alignment, including the amount of knowledge LLMs encode, e.g., factual (AlKhamissi et al., 2022) and commonsense (Bosselut et al., 2019; Sap et al., 2020) . Our work is the first to show that we can harness instruction-tuned LLMs for this purpose. Because they have been trained to respond to a general instruction format, we can evaluate LLMs on diverse tasks in a fine-grained manner, allowing the study of LLM properties both internal (e.g., knowledge) and external (e.g., behavior), and how they correlate with brain and behavioral alignment.World knowledge shapes language comprehension and brain activity. Our results show that world knowledge is a key factor in LLM brain alignment. LLMs demonstrating greater world knowledge across all tested subject domains have representations more similar to the human brain. This suggests that world knowledge influences human brain activity and shapes the language comprehension systems in our brain.Examining more dimensions of behavior. Our work and many prior works compare LLM and human next-word surprisal on reading tasks (Wilcox et al., 2020; Schrimpf et al., 2021; Eghbal A. Hosseini et al., 2023) , which evaluates only a single behavioral dimension for LLMs (per-word perplexity) and humans (reading times). For the models we tested, behavioral alignment is not significantly correlated with model size, world knowledge, or next-word prediction ability. While next-word prediction performance correlates with alignment to human reading times across many LMs (Schrimpf et al., 2021) , this trend does not hold up in recent transformer models (Oh & Schuler, 2023) , having a surprising negative correlation with parameter count (Oh et al., 2022) . In the future, we hope to evaluate LLMs on a wider range of behavioral dimensions (e.g., van Duijn et al., 2023; Koo et al., 2023; Kauf et al., 2024) to more holistically evaluate LLM-human behavioral alignment.We study brain alignment using neural datasets of humans reading naturalistic stories and sentences in English. It would be interesting to study brain alignment to human participants attempting the BBH and MMLU benchmarks, but this data unfortunately does not exist. This may explain why brain alignment is not significantly correlated with many categories of problemsolving on BBH, e.g., algorithmic reasoning. In the future, we hope to study brain alignment with human participants performing more diverse sets of tasks, e.g., reading computer program code (Ivanova et al., 2020) . This can identify more factors underlying LLM-brain alignment, and provide insights into how brain activity and the human language system may be shaped by various forms of problem-solving. Furthermore, some of the larger models exceed the noise ceiling estimates in our neural datasets (Appendix G), highlighting the need for more neural datasets and better ways of computing noise ceiling estimates. 
conclusion: 
 We investigate whether instruction-tuning improves the alignment of LLMs to the human language system. We evaluate 25 LLMs with parameter sizes ranging from 77 million to 33 billion, across three neural datasets of humans reading naturalistic stories and sentences. We find that instruction-tuning generally improves the alignment of LLM representations to brain activity. Exploring the factors underlying LLM-brain alignment, we discover that world knowledge and model size are key determinants of brain alignment. This suggests that world knowledge helps shape representations in the human language system, and highlights the importance of integrating world knowledge in developing future LLMs. 
reproducibility statement: 
 We provide our full results in Appendices G, H, and L. We consider this important as our experiments may be computationally expensive to replicate. We investigated 25 LLMs, with the largest models having 33B parameters, on many datasets: brain alignment (PEREIRA2018, BLANK2014, WEHBE2014), behavioral alignment (FUTRELL2018), next-word prediction (WikiText-2) and other evaluations (BBH, MMLU).All models and code repositories used in our study are open-source and their corresponding links are provided in Appendix B and F for full transparency and reproducibility. To calculate brain and behavioral alignment, we used the Brain-Score repository (www.github.com/brain-score/language), a publicly accessible resource for conducting these assessments. We encourage researchers interested in replicating our findings to refer to the provided links and consult the Brain-Score repository for further details on datasets and the evaluation process. To evaluate LLMs on BBH and MMLU, we utilize the widely-used instruct-eval repository (https://github.com/declare-lab/instruct-eval) with default settings. For the parameter count, "M" refers to million and "B" refers to billion. The number of layers for T5 models is a sum of the number of encoder and decoder layers. We follow a general approach for aligning LLM representations to fMRI brain activity used in prior works (Jain & Huth, 2018; Toneva & Wehbe, 2019; Schrimpf et al., 2021) . The approaches used for the three neural datasets are similar. Hence, we explain the main details only for WEHBE2014, followed by key differences for the other two datasets. 
 
 Appendix: WEHBE2014 (Wehbe et al., 2014) : The data includes fMRI recordings of 8 human participants reading chapter 9 of the book Harry Potter and the Sorceror's Stone (Rowling et al., 1998) .Participants read the chapter at a fixed interval of one word every 0.5 seconds. The chapter contains 5,176 words.We extract LLM representations of the text at each layer ∈ R w×d , where w is the number of words in the text and d is the embedding size of the LLM. fMRI activity. Data analyses were performed on fMRI BOLD signals extracted from the language network for all three neural datasets. fMRI activity was recorded at regular intervals of 2 seconds, with a total of 1,351 recorded time-points (TRs, times of repetition).We construct a stimulus-response matrix for fMRI activity ∈ R TR×v i , where TR is the number of TRs and v i is the number of voxels in the brain of participant i.Alignment of LLM representations to fMRI activity. We use PCA to reduce the dimensionality of the LLM representations to result in a matrix ∈ R w×10 . As the number of words in the text is greater than the number of TRs, we down-sample the LLM word-level representations to the TR rate by averaging the LLM representations to the corresponding TRs, producing a matrix ∈ R TR×10 . As the response recorded by fMRI peaks about 6 seconds after stimulus onset, we follow prior methods by including preceding time-points for each time-point. The final LLM representations are constructed by concatenating, for each TR, the LLM representations corresponding to the previous 4 TRs, producing a matrix ∈ R TR×40 . This accounts for the lag in the hemodynamic response that fMRI records. Finally, we learn a linear function, regularized by the ridge penalty, that uses the modified LLM representations ∈ R TR×40 to predict the fMRI activity of participants ∈ R TR×v i . We train the function using 4-fold cross-validation, where each fold corresponds to a separate run of fMRI collection. We also remove 10 TRs from the LLM representations and fMRI activity from the ends of each test run to avoid train-test overlap. To compute the brain alignment, we evaluate the Pearson correlation (r) between the predicted fMRI activity ∈ R TR×v i and recorded fMRI activity ∈ R TR×v i .BLANK2014 (Blank et al., 2014) : The data consists of fMRI recordings of 5 human participants listening to 8 naturalistic stories from the Natural Stories Corpus (Futrell et al., 2018) .fMRI activity was recorded at regular intervals of 2 seconds, with a total of 1,317 TRs. We average the BOLD signals across voxels within each language-responsive region of interest (ROI) of each participant to increase the signal-to-noise ratio, following Schrimpf et al. (2021) .As there are 60 ROIs in total across the 5 participants, this produces a stimulus-response matrix ∈ R 1,317×60 .PEREIRA2018 (experiments 2 and 3 from Pereira et al., 2018) : In experiment 2, 9 participants read 384 sentences taken from 96 text passages. In experiment 3, 6 participants read 243 sentences from 72 text passages. Each sentence was displayed for 4 seconds on a screen.We average the fMRI responses for each sentence, resulting in one data point per sentence per language-responsive voxel of each participant. For experiment 2, there are 384 sentences and 12,195 language-responsive voxels across the participants. For experiment 3, there are 243 sentences and 8,121 language-responsive voxels across the participants. We concatenate the fMRI responses across sentences and participants for each experiment. This produces a stimulus-response matrix ∈ R 384×12,195 for experiment 2 and a stimulus-response matrix ∈ R 243×8121 for experiment 3. Figure 5 : Method for computing Behavioral alignment. The same language stimuli are presented to LLMs and human participants, using the FUTRELL2018 benchmark in Brain-Score, which contains naturalistic stories. We compute the behavioral alignment as the Pearson correlation between LLM perplexity for each word and human reading times for each word.FUTRELL2018 (Futrell et al., 2018) : The data consists of reading times (RTs) of 179 participants on a total of 10,256 words across 485 sentences, taken from 10 stories from the Natural Stories Corpus (Futrell et al., 2018) . The stories were presented online to Amazon Mechanical Turk users visually in a dashed moving window display. Participants press a button to reveal each consecutive word. When they press the button, the most recent word is converted back to dashes, and the next word is revealed. The time between button presses offers an estimate of comprehension difficulty (Ehrlich & Rayner, 1981; Hale, 2001; Smith & Levy, 2013) . We construct a stimulus-response matrix of RTs ∈ R 10,256×179 .We provide the same stimuli to an LLM and record its perplexity for each word, producing an LLM perplexity matrix ∈ R 10,256 . We compute behavioral alignment as the Pearson correlation between per-word LLM perplexity and human RTs. Perplexity for LLMs and reading times for humans offer insights into comprehension difficulty (Ehrlich & Rayner, 1981; Hale, 2001; Smith & Levy, 2013) , allowing us to examine whether LLMs and humans share similarities in terms of which words and sentences they find challenging or surprising. is often computed by recording multiple samples from the same subjects for the same stimuli, providing an upper-limit estimate of the possible correlation in fMRI activity. In our study, we followed prior work for calculating the noise ceiling. For PEREIRA2018 and BLANK2014, noise ceiling estimates are computed using the Brain-Score repository, with details provided in Schrimpf et al. (2021) . For WEHBE2014, noise ceiling estimates are computed using a similar procedure. The T5 models were trained on supervised and unsupervised tasks, while the LLaMA models were trained only on unsupervised text (Section 3). (3) Loss computation: The loss functions for both model families are computed differently, making it inappropriate to directly compare their loss values. (4) Evaluation Metrics: Next-word prediction loss is just one metric, and it may not capture the overall language understanding capabilities of a model. Hence, we additionally evaluate these LLMs' alignment to human brain activity, as well as their performance on problem-solving abilities (BBH) and tasks requiring world knowledge (MMLU). In summary, while NWP loss is a valuable metric for evaluating language models within the same family or architecture, comparing across different model families may not be meaningful. Instruction model. We instruction-tune LLaMA-7B on the Stanford Alpaca dataset (Taori et al., 2023) using the default training process, following the code in www.github.com/ tatsu-lab/stanford alpaca. In particular, the model is instruction-tuned using 52K instruction-following examples generated through methods inspired by Self-Instruct (Wang et al., 2022a) . This model is labeled "Instruction" in Figure 8 .We also train an ablation model with the same process and training data as the default instruction-tuning, but remove the instruction portion from each training sample. This ablation model is labeled "No instruction" in Figure 8 . This ablation experiment disentangles: (1) training data (present in both "Instruction" and "No instruction"), from ( 2) training LLMs to understand and follow instructions (present only in "Instruction").We use all provided training samples from the Alpaca dataset, thus ensuring that the models are trained on the same amount of data. We observe that brain alignment of the "No Instruction" ablation model increases during fine-tuning but stays lower than its "Instruction" counterpart. This shows that brain alignment improvements are due to both (1) training data (present in both models) and (2) the process of training LLMs to understand and follow instructions (present only in the "Instruction" model). In fMRI recordings, a noise ceiling for representational similarity can be computed by sampling from the same participant twice, obtaining an upper limit for how well an ideal model could perform, defined by the internal consistency of neural responses and noise level of the data gathering process.https://github.com/declare-lab/instruct-evalhttps://github.com/declare-lab/instruct-eval 


Decision: ## Summary of the paper

The paper studies how instruction tuning, i.e. supervised fine-tuning on instruction-following data, affects
- Brain alignment: predictability of human brain activity from LLM activations (when humans and LLMs read the same text)
- Behavior alignment: correlation between LLM perplexity and human reading times

The authors show a significant positive impact from instruction tuning on the Brain alignment. The authors argue that in some cases the to the results are close to the noise level of the data, i.e. optimal performance. The authors also show that brain alignment strongly correlates with performance on certain benchmarks such as MMLU.

On the behavior alignment task, the authors show negative or no improvement from instruction tuning.

## Strengths

- This is the first paper to the best of my knowledge that studies the impact of instruction tuning on LLM-Brain alignment
- The results are interesting and not obvious, and contribute to the broader picture of Brain-LLM alignment
- The authors study a broad range of models; during the rebuttal, the authors added results for the recent Gemma and Llama2 models, and the original paper already included many models
- The poor correlation of perplexity and reading times is somewhat unexpected

## Weaknesses

- I agree with reviewer uLs2 that studying RLHF as opposed to supervised finetuning on instruction data would be very interesting. The industry standard for LLM post-training is RLHF and understanding the effect of RL would be extremely interesting, especially if it is different from supervised finetuning.
- Reviewers uLs2 and xjE7 raised concerns about the behavior alignment method: e.g. possibly the reading times should be normalized by the total length of the text in characters. The results on behavior alignment are surprising, and it would be interesting to provide further insights here.
- Reviewer hcZZ pointed out that the paper does not develop new methodology, and instead applies existing methodology in a new setting. I am inclined to agree with the authors that this is not a major issue. 
- I think it would be extremely interesting to push the hypothesis on correlation of Brain Alignment and model size to the limit and try models like Llama2-70, Nemotron-4 340B, and maybe Grok-1 314B.

## Conclusion

Overall, I think this is an interesting paper, and it would be a valuable contribution to the conference, despite some limitations of the analysis.

[At least one review was discounted during the decision process due to quality]

Reviews: review: paper_summary: paper_summary: This work investigates whether instruction-tuning can make LLMs process language similar to humans. In particular, this work studies:
Brain Alignment: How closely LLMs’ internal representations match neural activities in the human language system.
Behavioral Alignment: How similar LLM behaviors are to human behaviors during language tasks.
Comprehensive and carefully-designed experiments are conducted to draw the conclsion:

Instruction-tuning generally aligns LLM representations to human brain activity and instruction-tuning LLMs generally does not enhance behavioral alignment with human reading times.
strength_weakness: strength_weakness: 1. It would be great to provide more insights/future directions to how to build more brain-/human-intelligence-aligned LLMs.

The rest weaknesses here are more like a suggestion not weakness:
2. It would interesting to see the connections between the alignment and the training data, which will provide more insights to determine what's lacking in the current instruction tuning to build more brain-aligned LLMs.

3. I notice the behaviors of LLMs on reasoning tasks is not well aligned with human. I wonder whether it's a training data issue or model arch issue. There are reasoning-specific instruction tuning models such as Mammoth for math reasoning. It would be great if the models are specificly trained for a specific task can increase the alignment.
paper_summary: paper_summary: This paper explores the correlation between instruction-tuning and brain alignment as well as behavioral alignment, demonstrating that instruction-tuning can enhance scores for brain alignment without increasing behavioral alignment scores. Additionally, the paper analyzes the correlation between brain alignment, behavioral alignment, model size, world knowledge, and model capabilities.
strength_weakness: strength_weakness: 1.	Figures 2 and 4 (B, C, D) are noted to be somewhat unclear, suggesting a need for refinement in their presentation for better clarity.
2.	The paper would benefit from an expanded analysis that includes models such as LLaMA2 and Mistral to more effectively demonstrate the impact of instruction tuning. Additionally, the inclusion of an analysis on the correlation between large language model alignment methods, such as RLHF, and brain alignment would contribute to the comprehensiveness of the study.
3.	It is recommended that the use of reading time in behavioral alignment be reconsidered to include the potential impact of word length. The analysis could potentially be more thorough if it differentiates between long and short words based on a defined word length threshold, and also takes into account the reading time adjusted for word length, such as by using a reading time to word length ratio.
4.	Representational Similarity Analysis (RSA) and Brain-Score can both indicate brain alignment, and the authors might explore whether RSA could lead to the same conclusions.
questions: questions: refer to the comments
paper_summary: paper_summary: This paper is an investigation into whether instruction-tuning of LLMs affects the alignment between intermediate LLM representations and brain activity as measured by fMRI.  Alignment with brain representations is measured using the Brain-Score linear predictivity measure from prior work, and this measure is computed across a variety of language models both with and without instruction-tuning.  The experiments find that instruction tuning increases brain alignment by 6.2% on average across different base and instruction-tuned model pairs.  In addition, the brain alignment of the different LLMs is also compared with performance on benchmarks measuring world knowledge and reasoning, and a strong correlation is found with world knowledge benchmark performance but not with reasoning.  Finally, there is also an experiment in the paper comparing "behavioral alignment", which looks at the correlation between per-word perplexity of LLMs and the human per-word reading times, but does not find any significant effect of instruction tuning on this metric.
strength_weakness: strength_weakness: This paper does not introduce new methods of analysis, it runs existing implementations of LLM-brain alignment measurement across a different set of input LLMs.

As the paper notes, much prior work has found that LLM-brain alignment is well correlated with improved LLM benchmark performance, and since instruction tuning is known to substantially improve benchmark performance it seems unsurprising that it would also follow this trend and improve alignment.
paper_summary: paper_summary: This paper aims to study the difference between base and instruction-tuned models in terms of their alignment to both human brain and behaviours. 1) The brain alignment is measured by how well a linear regressor can predict fMRI activity from an LLM's hidden representation. 2) The behavioural alignment is measured by how well per-word perplexity correlates with per-word reading time. The work further studied the how well brain and behavioural alignment correlates with several test set and model properties.
strength_weakness: strength_weakness: Major:
- From the results in the brain alignment part, I agree that instruction tuning aligns "better" than base/pre-trained models, but the Pearson correlation coefficients are still in the "no correlation" to "weak positive" range. If I did not misunderstood: the average improvement (from base to instruction tuning) seems significant, but it is a % score derived from the difference between already weak correlation coefficients. I think that a quantitative definition of "alignment" should be provided or explained, e.g. how good the correlation needs to be in order to qualify for "alignment"; otherwise it is unsound to claim the title of this paper.
- The introduction states that "to investigate why instruction-tuning increases alignment to human brain activity [...]" which formulates the problem as a causal analysis, but the presented methodology is a post-hoc correlation study. I fear that the conclusion might lack generalisability.

Minor:
- Slight overclaim in introduction "however, there has been no similar study on how instruction-tuning". [1] has done an analysis from the perspective of LLM and human attention.
- The methodology used for behavioural alignment is "the similarity between LLM per-word perplexity and human per-word reading times", which is an accepted method but the perplexity and reading time should not be used to define behavioural alignment. ~~In fact the features used in [1] yielded different conclusions.~~ (the reviewer made an incorrect statement in the original review.)
 
[1] Roles of Scaling and Instruction Tuning in Language Perception: Model vs. Human Attention (Gao et al., EMNLP Findings 2023)

Comments: title: Reviewer xjE7's response after rebuttal
Response to the AC
comment: Thank you to the authors for your rebuttal!

Re 1, 3, and 4: Thank you for your explanation and clarification. I think my questions are addressed.

Re 2: I meant to say that that the "*why*" part in the statement is not well-explored.
Thank you for your response. While some of the concerns have been addressed, there are still significant issues, such as the evaluation, that have not been resolved. As a result, I will maintain my current rating.
We thank the reviewers for their feedback and the AC in advance for their meta-review.

We also thank the reviewers for identifying our paper’s strengths:
- **edNc**: comprehensive and carefully designed experiments, very interesting research problem, and meaningful findings.
- **uLs2**: comprehensive analysis of instruction-tuning, using many LLMs from multiple model families.
- **hcZZ**: interesting finding that brain alignment correlates better with world knowledge than reasoning.
- **xjE7**: interesting to read, contributes knowledge to HCI and LLM-human alignment, and the task/setting is relatively novel.

We summarize each reviewer’s suggestions/concerns and our responses:

**Reviewer edNc**: 
- Suggested we discuss more insights to build brain-/human-intelligence-aligned LLMs. In our paper, we will add a subsection discussing papers with insights on LLM and human cognition and how to build brain-aligned LLMs.  (References [1-7] in our rebuttal).
- Suggested we study connections between brain alignment and training data. We explain that we instruction-tuned LLaMA-7B and an ablated version, and how that provides preliminary insights.
- Suggested we test reasoning-specific models (e.g., Math). We explain that reasoning-specific models may not be appropriate as we limit our scope to the human language system and use datasets where participants read/listened to English text.

**Reviewer uLs2**:
- Noted that two figures in our paper could be clearer. We remove model names and adapt point sizes to model sizes for better readability. We provide larger figure versions with model names in the Appendix. 
- Suggested we test additional models, e.g., LLaMA2, Mistral. We explain we prioritized models with diverse model architectures and families (LLaMA, T5, GPT-2). Regardless, in our rebuttal, we show results for LLaMA2 and Gemma models, yielding similar results to our paper’s conclusion.
- Suggested we study the effect of RLHF. We explain that we studied only one RLHF model, which is insufficient to draw rigorous conclusions. Our paper’s focus is instruction-tuning, for which we provide comprehensive results.
- Asked about the effect of word length on our behavioral alignment method. We explain that we follow prior works in implicitly accounting for word length by computing the surprisal of a word with multiple tokens as the sum of surprisals of each token. Importantly, we use consistent tokenizers when comparing each vanilla LLM to its instruction-tuned version, thus controlling for the effect of word length.
- Suggested we run other brain alignment methods (e.g., RSA) aside from Brain-Score linear predictivity. In our rebuttal, we add preliminary results for RSA and CKA, showing they are similar to linear predictivity. We focused on linear predictivity following prior works, but will add more comprehensive RSA and CKA results to our paper. 

**Reviewer hcZZ**:
- Concerned that our paper has no new methodology. We cite the COLM review guide in explaining that developing new methods is only one of many forms of scientific contribution; our analysis of instruction-tuning and LLM world knowledge contributes new insights to the field.
- Implied that since LLM performance is correlated to brain alignment (shown in prior work), it is expected that instruction-tuning improves brain alignment since it improves LLM performance. We explain why we are not confident in this generalization. (a) Prior works evaluate LLM performance using next-word prediction, whereas we study reasoning and world knowledge. (b) Contrary to the reviewer’s point, prior work showed that finetuning may result in no change or decreased brain alignment depending on the finetuning task and brain region.

**Reviewer xjE7**:
- Asked about the weakly positive correlations in our brain alignment results. We explain that some of our LLMs actually obtain brain alignment close to the fMRI “noise ceiling” (further explained in our rebuttal and paper), suggesting they align nearly all useful signals from fMRI data. We follow prior works in computing the noise ceiling.
- Claimed that our study is not a causal analysis. We explain how our experiments showing that instruction-tuning improves brain alignment are, in fact, a causal analysis.
- Noted a related work (reference [1] in their review) on instruction-tuning. We explain that our study is complementary to [1], with many key differences in datasets and methods. We will discuss this related work in our paper.
- Suggested our behavioral alignment method should not be used, and yields different conclusions from [1]. We explain that our behavioral alignment method is commonly used in the literature. Also, it actually yields a similar conclusion to [1]: improving LLM performance does not enhance alignment with human reading behavior. 

We will add these clarifications to improve our paper. Thank you again!

-Authors
Thank you for your response.  I have made a few minor edits to the wording of my review for clarity but have decided to keep my score the same.
Thank you once again for taking the time to review our work!  
We look forward to receiving your feedback on our responses to your initial review. We welcome any additional questions or discussion to ensure your outstanding concerns regarding our submission are thoroughly addressed before the discussion period ends on June 6th.
Thank you once again for taking the time to review our work!  
We look forward to receiving your feedback on our responses to your initial review. We welcome any additional questions or discussion to ensure your outstanding concerns regarding our submission are thoroughly addressed before the discussion period ends on June 6th.
Thank you once again for taking the time to review our work!  
We look forward to receiving your feedback on our responses to your initial review. We welcome any additional questions or discussion to ensure your outstanding concerns regarding our submission are thoroughly addressed before the discussion period ends on June 6th.
Thanks for the reply and explanation.
Totally understand that you want to narrow the scope for the current submission.

I will keep my score unchanged.