Instruction-tuning Aligns LLMs to the Human Brain
Authors: Khai Loong Aw
Syrielle Montariol
Badr AlKhamissi
Martin Schrimpf
Antoine Bosselut
Venue: COLM (2024)
Keywords: fMRI, brain, neuroscience, neuroAI, large language models
TL;DR: This paper explores how instruction-tuning affects language models from a neuroscientific perspective, revealing that it generally improves their alignment with human brain activity, with model size and world knowledge playing key roles.

Abstract: Instruction-tuning is a widely adopted finetuning method that enables large language models (LLMs) to generate output that more closely resembles human responses. However, no studies have shown that instruction-tuning actually teaches LLMs to process language in a similar manner as humans. We investigate the effect of instruction-tuning on aligning LLM and human language processing mechanisms in two ways: (1) brain alignment, the similarity of LLM internal representations to neural activity in the human language system, and (2) behavioral alignment, the similarity of LLM and human behavior on a reading task. We assess 25 vanilla and instruction-tuned LLMs on three datasets involving humans reading naturalistic stories and sentences, and find that instruction-tuning generally enhances brain alignment (~6%), but has no similar effect on behavioral alignment. To identify factors underlying this improvement in brain alignment, we compute correlations between brain alignment and various LLM properties, such as model size, problem-solving, and world knowledge understanding. Notably, we find a strong positive correlation between brain alignment and model size (r = 0.95), as well as performance on tasks requiring world knowledge (r = 0.81). Our results demonstrate that instruction-tuning LLMs improves both world knowledge representations and brain alignment, suggesting that the mechanisms that encode world knowledge in LLMs also improve representational alignment to the human brain.

Published as a conference paper at COLM 2024
Instruction-tuning Aligns LLMs to the Human Brain
Khai Loong Aw", Syrielle Montariol', Badr AlKhamissi Martin Schrimpfi, Antoine Bosselut EPFL awkhaillongg@gaill.com
Abstract
Instruction-tuning a widely adopted finetuning method that enables large language models (LLMs) to generate output that more closely resembles human responses. However, no studies have shown that instruction-tuning actually teaches LLMs to process language in similar manner as humans. We investigate the effect of instruction-tuning on aligning LLM and hu- man language processing mechanisms in two ways: (1) brain alignment, the similarity of LLM internal representations to neural activity in the human language system, and (2) behavioral alignment, the similarity of LLM and human behavior on a reading task. We assess 25 vanilla and instruction- tuned LLMs three datasets involving humans reading naturalistic stories and sentences, and find that instruction-tuning generally enhances brain alignment (~6%), but has no similar effect on behavioral alignment. To identify factors underlying this improvement in brain alignment, we com- pute correlations between brain alignment and various LLM properties, and world knowledge understanding.
such as model size, problem-solving Notably, we find strong positive correlation between brain alignment and model size = 0.95), as well as performance tasks requiring world knowledge (r = 0.81). Our results demonstrate that instruction-tuning LLMs improves both world knowledge representations and brain alignment, sug- gesting that the mechanisms that encode world knowledge in LLMs also improve representational alignment to the human brain.
1 Introduction
Instruction-tuning is a widely adopted method that finetunes large language models (LLMs) on datasets containing many examples of different tasks and their descriptive instructions, enhancing their ability to generalize to previously unseen tasks by learning to follow provided instructions (Wang et al., 2022c). Despite costing only small fraction of compute relative to pretraining (Chung et al., 2022), instruction-tuning yields large performance improvements on reasoning and problem-solving benchmarks. This generalization has allowed LLMs to tackle open-world reasoning tasks previously achievable only by humans (Zhang et al., 2023) while only using a few (or zero) task-specific training examples.
In addition to teaching LLMs to understand and follow human instructions, instruction- tuning also improves the ability of LLMs to mimic human-writter ground-truth outputs. This fluency allows them to produce more controllable and predictable output that is deemed (1) more desirable to human evaluators (Zhang et al., 2023; Chung et al., 2022; Wang et al., 2022b), (2) more aligned to human values (Chia et al., 2023), and (3) more stylistically similar to human outputs (Dasgupta et al., 2022; Safdari et al., 2023).
Consequently, instruction-tuning yields LLMS more similar to humans in both capability and output resemblance. From a neuroscience perspective, this begs the question: Does instruction-tuning make LLMs more functionally similar to the human language system?
Work was done during internship at EPFL
EEqual contribution
Equal supervision senior authors

---

Published as a conference paper at COLM 2024
Previous work has shown that models with higher task performance more aligned to the human language system (Schrimpf et al., 2021; Goldstein et al., 2022; Caucheteux & King, 2022), hitting the estimated noise ceiling' on some datasets. However, there has been no similar study on how instruction-tuning tunnng affects alignment with the human language system.
In this work, we measure the effect of instruction-tunin turion on the alignment between language mechanisms in LLMs and the human brain in two ways: (1) brain alignment, an "internal" metric that assesses how well internal feature representations of LLMs match neural repre- sentations in the human language system, and (2) behaviooal alignment, an "external" metric which evaluates the similarity between LLM and human behavioral measurements. In our study, both LLMs and humans are presented with the same language stimuli comprised of naturalistic stories and sentences. For LLMs, we record their internal representations and per-word perplexity. For humans, we use previously recorded brain activity data from functional magnetic resonance imaging (fMRI) experiments and per-word reading times.
To measure brain alignment, we use the Brain-Score (Schrimpf et al., 2018) linear predictivity metric, assessing how well LLM representations predict human brain activity in response to the same language stimuli (Jain & Huth, 2018; Toneva & Wehbe, 2019; Schrimpf et al., 2021; Oota et al., 2023), using data from three neural datasets: Pereira et al. (2018), Blank et al. (2014), and Wehbe et al. (2014). To evaluate behavioral alignment, we use a benchmark in Brain-Score which calculates the Pearson correlation between LLM per-word perplexity and human per-word reading times from the Futrell et al. (2018) dataset. Perplexity for LLMs and reading times for humans offer insights into comprehension difficulty (Ehrlich & Rayner, 1981; Hale, 2001; Smith & Levy, 2013), allowing us to examine whether LLMs match humans in their patterns of words and sentences they find challenging or surprising. Because models vary in their brain and behavioral alignment across different arch and training objectives (Schrimpf et al., 2021), we estimate the effect of instruction-tuning by evaluating 8 vanilla LLMs and 17 LLMs that were further instruction-tuned, and report significant increase in brain alignment due to instruction-tuning
To investigate why instruction-tuning increases alignment to human brain activity, we study the relationships between brain alignment and various LLM properties. Specifically, we compute Pearson correlations between an LLM's 's brain alignment and its properties, includ- ing next prediction (NWP) ability, model size, a range of problem-solving abilities, and world knowledge spanning different domains. We evaluated the latter two properties using the Big- Big-Bench Hard benchmark (BBH; Suzgun et al., 2022) and the Massive Multi-tast Language Understanding benchmark (MMLU; Hendrycks et al., 2021), respectively.
We report three major findings:
1. Instruction-tuning generally aligns LLM representations to human brain activity, increasing brain alignment by 6.2% on average (Figure 1).
2. Investigating the factors underlying LLM-brain alignment, we find that brain align- ment strongly correlated with world knowledge (r = 0.81) and model size (r 0.95) (Figure 2).
3. Surprisingly, our results indicate that instruction-tuning LLMs generally does not enhance behavioral alignment with human reading times. Furthermore, behavioral alignment on this dataset is poorly correlated with all other measures we investigate, including task performance and model size (Figure 3).
2 Background & Related Work
Effect of Instruction-tuning on LLMs. Instruction-tuning is an effective method for en- hancing LLM capability and controllability. It trains LLMs using pairs of human instructions and desired outputs. The benefits of instruction-tuning can be categorized into three key aspects (Zhang et al., 2023): (1) it bridges the disparity between the pretraining objective
! In fMRI recordings, noise ceiling for representational similarity can be computed by sampling from the same participant twice, obtaining an upper limit for how well an ideal model could perform, defined by the internal consistency of neural responses and noise level of the data gathering process.
2

---

Published as a conference paper at COLM 2024
Language model
rpprssetations
Train linear regression
Language stimuli
Pereira2018
hoot habkit, every teskespers
Blank2014
you would corn ail
pourse Merr Englend.
Human
Wish662014
"Marr had believed he but.."
1MRI activity
Test held-out data
Brain alignment
LM properties
Average Brain alignment (Pearson comr.)
Neural dataset
0235
tre sctioo No rotruction
0.24
TS
+6.9%
Liama
230
0.22
Instruction-uned LM
0.20
+8.0%
Blank2014
Intructionnunng ooects
0.16
0.30
+3.8%
average improvement: 6.2%
Wehbe2014
0.12
0.30
0.10
No
0.10
0.12
0.14
0.16 0.18 0.20 0.22 0.24
15
Vanilla
% change in brain alignment
Figure 1: Instruction-tuning aligns LLM representations to human brain activity. (A) The same language stimuli are presented to LLMs and human participants. Next, we fit linear regression model from LLM layer activations to fMRI responses in the human language system. We apply this linear model to predict held-out fMRI responses from the original corpus of recordings, and compute brain alignment the predicted and actual fMRT responses. We evaluate 25 vanilla and instruction-tune LLMs with sizes between 77M and 33B parameters. We compute the average across 3 neural datasets of humans reading naturalistic stories and sentences: PEREIRA2018 BLANK2014, and WEHBE2014. (B) Instruction-tuning improves average brain alignment by 6.2% on average. Each point above the identity line is an instruction-tuned LLM with greater brain alignment than its vanilla version. Error bars, here and elsewhere, represent median absolute deviation over human participants. (C) Instruction-tuniin improves average brain alignment on all three datasets. (D) We instruction-ture LLLMA-7B on the Alpaca dataset ("Instruction" model) and train an ablation model with the same data, but without the instruction in each training sample ("No Instruction" model). Our results show that brain alignment improvements are due to both (1) training data (present in both models) and (2) training LLMs to understand and follow instructions (present only in the first model).
the Pearson correlation between
of LLMs (next-word prediction) and the goal of accurately following human instructions, (2) it achieves greater control and predictability of model behavior compared to standard make them more similar to humans in both capability and output similarity (Chia et al., 2023; Dasgupta et al., 2022; Safdari et al., 2023), and (3) it often costs only a small fraction of compute relative to pretraining, enabling LLMs to swiftly adapt to target domains (Chung et al., 2022). We contribute to this research area from a neuroscience perspective, by studying whether instruction-tuning makes LLMss more aligned to the human language system in terms of brain and behavioral alignment.
LLMs, allowing researchers
Effect of Finetuning on Brain alignment. Prior works have studied how finetuning affects LMs' alignment to human brain activity. These include finetuning on wide range of downstream NLP tasks (Oota et al., 2022), finetuning to summarize narratives (Aw & Toneva, 2023), and finetuning to directly predict brain activity recordings (Schwartz et al., 2019). These studies aim to use brain alignment to study how finetuning affects LMs and their representations. Our work builds on this line of research by demonstrating that
3

---

Published as a conference paper at COLM 2024
Instruction
Input
Output
"Write short paragraph about the given topic."
"The importance of using renewable energy."
"The use of renewable energy is growing... (paragraph)
Table 1: Example of Instruction-tuning training data format: (Instruction, Input, Output) from the Alpaca dataset (Taori et al., 2023). The input field is optional for certain instructions.
instruction-tuning aligns LLM representations to human brain activity. We also investigate why instruction-tuncd LLMs align to brain activity by testing the correlation of brain alignment with various world knowledge domains and problem-solving abilities.
LM properties linked to Brain alignment. One exciting research area focuses on disentan- gling the contribution of various LM properties towards brain alignment. These include studying how brain alignment is driven by next-word prediction ability (Schrimpf et al., 2021; Caucheteux & King, 2022), multi-word semantics (Merlin & Toneva, 2022), perfor- mance on various NLP tasks (Oota et al., 2022), and model size (Antonello et al., 2023), To contribute this growing body of work, we use instruction-tune especially useful as they have been trained to respond to a standard question-answer format, allowing us evaluate LLMs on a wide array of tasks and in a more fine-grained manner. We expand this area of research by demonstrating that world knowledge is key property underlying LLM-brain alignment.
LLMs. They are
3 Language Models
We evaluate the brain alignment of 25 large language models (LLMs) from two model families: T5 (Raffel al., 2020) and LLaMa (Touvron al., 2023a). T5 models are encoder- decoder LLMs pre-trained using masked infilling objective on the Colossal Common Crawl Corpus (C4), a corpus 356 billion tokens, and then further finetuned on a multi-task mixture of unsupervised and supervised tasks converted into a text-to-text format. We use all five T5 models, with sizes between 77M to 11B. LLaMA models (Touvron et al., 2023a) are decoder-only LLMs trained on 1.6 trillion tokens from a mixture of corpora including C4, English CommonCrawl, Wikipedia, GitHub. We use the 7B, 13B, and 33B versions.
For the instruction-tuned variants of T5 models, we variety of models finetuned FLAN (15M examples for 1,836 different tasks accompanied by instructions, Chung et al., 2022), Alpaca (52K instruction-following examples generated with methods inspired by Self- Instruct, Wang et al. (2022a), Taori et al., 2023), and GPT4ALL (437K instruction-folloingg examples generated with GPT-3.5-turbo, Anand et al., 2023) datasets. For the instruction- tuned variants of LLaMa, we use Vicuna's 's 7B, 13B, and 33B models (Chiang et al., 2023), which are finetuned on user-shared conversations. We also use StableVicuna-138B which further refines Vicuna-13B using reinforcement learning from human feedback (RLHF) (Ouyang et al., 2022) on a range of conversational and instructional datasets. We also use the 7B version of Alpaca (Taori et al., 2023). We provide more details in Appendix A.
of
We also tested 8 models from the GPT2 family; four vanilla models (Small, Medium, Large, XL) and their corresponding versions further instruction-tuned on Alpaca. However, the models performed close to random on reasoning and world knowledge benchmarks (BBH, MMLU). We hypothesize this is due to their size (all < 1.5B parameters), making fine-tuning on a relatively small originally pretrained to perform instruction-folloiing tasks (unlike T5 models). We include their MMLU, BBH, brain and behavioral alignment results in Appendix G, H, and L.
dataset insufficient. Moreover, GPT2 was not
4 Brain Alignment
Brain alignment refers to the method of evaluating the similarity between LLM representa- tions and human brain activity (Figure 1A and Appendix C). This relies on fMRI recordings of human subjects while they read language stimuli on potentially any topic (here: Pereira
4

---

Published as a conference paper at COLM 2024
et al., 2018; Blank et al., 2014; Wehbe et al., 2014). The same language stimuli from prior brain recordings are provided as input to LLMs, whose intermediate layer activations are extracted as their representations of the language stimuli. We follow general approach previously used in several works (Schrimpf et al., 2018; Jain & Huth, 2018; Toneva & Wehbe, 2019; Schrimpf et al., 2021; Oota et al., 2023; Aw & Toneva, 2023). Specifically, we use the linear predictivity metric implemented in Brain-Score (Schrimpf et al., 2018), which fits linear regression model from LLM layer activations to fMRI responses in the human language system. We then apply this linear model to predict held-out fMRI responses from the original corpus of recordings, and compute brain alignment as the Pearson correlation between the predicted and actual fMRI responses. For each LLM, we evaluate its brain alignment for every layer (e.g,, LLaMA-7B has 32 layers), and use the highest value as the LLM's brain alignment value, following Schrimpf et al. (2018).
Datasets We use three fMRI datasets to measure the brain alignment of L.Mss. Each dataset involves a different set of human participants and a different set of language stimuli.
PEREIRA2018 (experiments 2 and from Pereira et al., 2018): In experiment 2, 9 participants read 384 sentences taken from 96 text passages. In experiment 3, 6 participants read 243 sentences from 72 text passages. Each sentence was displayed for 4 seconds on a screen.
BLANK2014 1 (Blank et al., 2014): The data consists of fMRI recordings of 5 human participants listening to 8 naturalistic stories from the Natural Stories Corpus (Futrell et al., 2018).
WEHBE2014 14 (Wehbe et al., 2014): The data includes fMRI recordings of 8 human participants reading chapter of the book Harry Potter and the Sorcerer's 's Stone (Rowling et al., 1998). Participants read the chapter at fixed interval of one word every 0.5 seconds.
4.1 Instruction-tuning aligns LLM representations to human brain activity
First, we study the effect of instruction-tuning on LLM brain alignment. We compute each LLM's average brain alignment as the mean of its brain alignment on the 3 neural datasets. We find that instruction- -enniinnnntuingggg..o improves brain alignment by an average of 6.2% across tested LLMs (Figure 1B). This holds across all three neural datasets, with improvements of +6.9% BLANK2014 4, and +3.8% on WEHBE2014 (Figure
PEREIRA2018, +8.0% improvement 1C). Moreover, a smaller Instruction-tuned model can attain higher brain alignment than a larger vanilla model from the same family (e.g., Alpaca-7B v.s. LLaMa-13B, see Appendix G). However, some of the larger models (T5-XL, LLaMA) have minimal or no increases, possibly because the vanilla models are already close to the noise ceiling (Appendix G).
Next, to longitudinally study how instruction-tuning aligns LLM representations to brain activity, we separately instruction-tune a LLaMA-7B model on the Stanford Alpaca instruc- tion dataset (Taori et al., 2023) for 3 epochs. By evaluating checkpoints regularly during training, we find that instruction-tuning progressively improves brain alignment (Figure ID, Appendix K). We also disambiguate the effect on brain alignment of (1) the instruction- following ability gained from instruction-tuning and (2) added training data. We fine-tune LLaMA-7B with the same process and data, but remove the instruction from each training sample. Brain alignment of this ablated model increases during fine-tuning but stays lower than its instruction-following counterpart (Figure ID), indicating that brain alignment im- provements are due to both factors. In Appendix J, we also report preliminary results using recent models from the Gemma (Team et al., 2024) and LLaMA-2 (Touvron et al., 2023b) families and observe similar trends. Further, these trends hold across different similarity metrics besides Linear predictivity, e.g., RSA and CKA (Appendix) J).
4.2 Factors underlying LLM-brain alignment
To study why LLMs and human brains align in their representations, we compute the Pear- son correlation between LLM brain aligmment and various properties of LLMs: performance on benchmark testing various reasoning abilities (BBH; Suzgun et al., 2022), performance on a benchmark testing world knowledge in various domains (MMLU; Hendrycks et al., 2021), language modeling ability, and model size.
5

---

Published as a conference paper at COLM 2024
Brain alignment A LM properties
0.81, p 0.001
0.68, p 0.005
C
0.95, p
0.24
0.24
0.24
Brain alignment (
Brain alignment
Brain alignment (r)
0.20
0.22
0.22
0.20
T5
TS
T5
0.18
0.10
Liama
0.16
2.16
0.16
0.34
0.54
0.60
0.39
0.50
0.0
11.0
MMLU Overall score
BBH world knowledge
10g model parameters)
Figure World knowledge and model size are important factors underlying LLM-brain alignment. To identify factors underlying brain alignment, we test Pearson correlations between brain alignment and various LLM properties, such as model size, world knowledge in various domains (MMLU benchmark), and various types of problem-solving abilities (BBH benchmark). In the figure, insets display results on individual datasets, with stars reflecting statistical significance (n.s. =p- 0.05, * = < 0.05, p 0.005, etc.) (A) Brain alignment is significantly and strongly correlated with world knowledge as evaluated by the MMLU Overall score (r 0.81), which reports the mean performance across all MMLU subjects. (B) Brain alignment is significantly and strongly correlated with performance on the world knowledge task category in BBH (r 0.68). (C) Brain alignment is significantly and strongly correlated with model size (logarithm of number of parameters) (r = 0.95). In Appendix we provide a larger version of this figure with labels for each data point.
Correlation (r) to
Adjusted p-value
No. tasks
Avg. Model Accuracy
Benchmark, Task category
Brain Alignment
MMLU, Overall Score
0.809
0.0005
57
0.36
MMLU, STEM
0.792
0.0005
18
0.28
MMLU, Humanities
0.791
0.0005
13
0.34
MMLU, Social Sciences
0.807
0.0005
12
0.41
MMLU, Others
0,809
0.0005
14
0.40
BBH, Overall score
0.384
0.18
23
0.28
BBH, Algorithmic reasoning
0.194
0.56
8
0.22
BBH, Language understanding
0.163
0.59
3
0.43
BBH, World knowledge
0.679
0.005
5
0.36
BBH, Multilingual reasoning
-0.035
0.90
1
0.19
BBH, Others
0.478
0.08
6
0.27
Table 2: Brain alignment strongly correlates with world knowledge in all subject domains in MMLU and the world knowledge category in BBH. Brain alignment is not significantly correlated with other problem-solving abilities in BBH (e.g,, algorithmic or multilingual reasoning). We obtain p-values after performing false discovery rate (FDR) correction.
MMLU and BBH. MMLU is designed to measure the world knowledge of LLMs across many subject domains. contains 57 tasks, categorized into four world knowledge sub- ject domains: STEM, Humanities, Social Sciences, and Others (a broad category ranging from finance to marketing to professional medicine). BBH contains 23 tasks, grouped into four types of problem-solving Algorithmic and Multi-Step Arithmetic Reasoning; Natural Language Understanding; Use of World Knowledge; and Multilingual Knowledge and Reasoning. For both benchmarks, we follow the category classifications from the original pa- pers. We perform the evaluations using the instruct-eval repository2 with default settings (3-shots for BBH, 5-shots for MMLU) and preset prompts. We measure the Pearson correla- tion (and its p-value) between LLM-brain alignment and performance in each category of MMLU and BBH. We obtain p-values after false discovery rate (FDR) correction.

---

Published as a conference paper at COLM 2024
World Knowledge. We find that brain aligmment is significantly and strongly correlated with world knowledge. Brain alignment is highly correlated with the MMLU Overall score (r = 0.81, P < 0.001, Figure 2A), which reports the mean performance across all world knowledge domains on MMLU. Similarly, brain alignment is also strongly correlated with performance on tasks in the world knowledge category of BBH (r = 0.68, p 0.005; Figure 2B). Notably, brain alignment is poorly correlated with all other dimensions of BBH (see Table 2), though this could also be due to limitations of the tested models, as indicated by their low raw performance scores on some tasks. Overall, our results provide strong signal that more accessible representations of world knowledge are key factor in aligning LLM representations to human brain activity.
Language Modeling Ability. Prior works have shown correlations between brain align- ment and next-word prediction (NWP) ability (Caucheteux & King, 2022; Schrimpf et al., 2021). We find similar results for correlation between brain alignment and NWP loss (r = -0.54, p 0.05, Appendix 1). Interestingly, the strength of the correlation is weaker than that between brain alignment and world knowledge performance (r 0.81). This suggests that world knowledge understanding is a better predictor of brain alignment than NWP ability.
Model Size. Finally, we find that brain alignment is significantly and strongly correlated with model size (r = 0.95, p < 0.001, Figure 2C), as measured by the logarithm of the number of model parameters. Schrimpf et al. (2021) observe such pattern for language models, and find the pattern holds for instruction-tunet than their study (7B+ parameters). However, model size alone does not determine brain alignment. Our results show that smaller instruction-tuned LLMs can have greater brain alignment than larger vanilla models. For example, LLaMA-13B obtains brain alignment of 0.220, Vicuna-13B obtains 0.229, LLaMA-33B obtains 0.227, and Vicuna-33B obtains 0.232. Hence, Vicuna-13B has greater brain alignment than LLaMA-33B, due to instruction-tuning, despite being less than 40% its size. We observe similar trend in another four models: T5-base, Flan- T5-base, T5-large, Flan-T5-large. Also, prior works showed that large random models achieve poor brain alignment (Schrimpf et al., 2021). These results demonstrate there are LLM properties aside from model size that contribute significantly to brain alignment.
models, and models trained at a larger scale
Neural datasets. Brain alignment is strongly correlated with world knowledge and model size on PEREIRA2018 and WEHBE2014, 4, but less strongly correlated on BLANK2014, possibly because BLANK2014 has fewer participants (N = 5) leading to greater noise in the results.
5 Behavioral Alignment
In the previous section, we show that instruction-tunine aligns the internal representations of LLMs to human brain activity (Section 4.1). In this section, we explore whether instruction- tuning also aligns LLM behavior to human behavior.
Following the approach of Schrimpf et al. (2021) implemented in the Brain-Score package (Schrimpf et al., 2020), we measure behavioral alignment by evaluating the similarity between LLM per-word perplexity and human per-word reading times given the same language stimuli (Appendix D). We use the self-paced reading times dataset from Futrell et al. (2018), consisting of the reading times of 179 human participants recorded while they were visually presented with 10 naturalistic stories. We provide language stimuli from this data as input to LLMs and evaluate behavioral alignment by computing the Pearson correlation between per-word LLM perplexity and per-word human reading times.
One potential concern in the behavioral aligmment methodology is that human reading time is influenced by word length; longer words may take more time to read, even if they are not more challenging or surprising. Following Schrimpf et al. (2021), we address this by computing the surprisal of multi-token word as the sum of the surprisals of its individual tokens. Importantly, we use consistent tokenizers when comparing vanilla LLMs against their instruction-tumn -unden versions, thereby controlling for the effect of word length when evaluating the impact of instruction-tuning on behavioral alignment.
7

---

Published as a conference paper at COLM 2024
Behavioral alignment
LM properties
0.00 B
n.5. (p+0 54)
A
Average Behavioral alignment (r)
T5
0.30
Liama
Instruction-tunee iitiontuued LM
0.25
Liara
GAR 6.G. cas KNIL
NNP loos
(p-0.31)
n.s. 2.1)
0.15
Rechevioret atgimmen
0.10
Lieraa
0.05
0.10
0.15
0.20
0.25
0.30
Ida vis
Vanilla LM
parametens)
Figure 3: Instruction-tuning LLMs generally does not improve behavioral alignment to human reading times. Furthermore, behavioral alignment correlates poorly with all other tested measures: world knowledge, model size, and next-word prediction (NWP) ability. To compute behavioral alignment, we use the FUTRELL2018 benchmark in Brain- Score. The same language stimuli (naturalistic stories) are presented to LLMs and human participants. We then compute the Pearson correlation between per-word LLM perplexity and human reading times as the behavioral alignment. (A) Instruction-tune -unnungn does not generally improve behavioral alignment. Furthermore, behavioral alignment is poorly and not significantly correlated with all other measures: (B) world knowledge (p= 0.76), (C) model size (p = 0.31), (D) NWP loss for T5 models (p = 0.54), and (E) NWP loss for LLaMA models (p- 0.21). In Appendix M, we provide a larger, labeled version of this figure.
5.1 Instruction-tuning generally does not improve behavioral alignment
Using the same models as in Section 4, we compare the behavioral alignment of each instruction-tuned LLM against its vanilla counterpart. Our results indicate instruction- tuning generally does not improve behavioral alignment to human reading times (Figure 3A). For half of the LLMs tested, it results in no change reduced behavioral alignment. Our results align with recent studies (Gao et al., 2023; Steuer et al., 2023) that show improving LLM performance does not necessarily improve alignment with human reading behavior.
5.2 Factors underlying behavioral alignment
Next, we test the correlation between LLM behavioral alignment and model size, next-word prediction ability, various reasoning abilities (measured by BBH), and world knowledge across various domains (measured by MMLU). Contrary to our findings on the brain alignment correlations with model size and world knowledge (Section 4.2), we find that LLM behavioral alignment is not correlated with these factors: world knowledge (p 0.76, Figure 3B), model size (p- 0.31, Figure 3C), next-word prediction loss for T5 models (p 0.54, Figure 3D), and next-word prediction loss for LLaMA models (p 0.21, Figure 3E).
Discussion
6.1 Implications for NLP: Building LLMs
Using brain alignment for LLM interpretability. Prior works have used human brain activity to interpret models (Dong & Toneva, 2023) and build more performant models (Dapello et al., 2020; Safarani et al., 2021; Dapello et al., 2022). Instruction-tuning has emerged as a breakthrough technique to improve LLM abilities and quality of outputs, and allows LLMs to adapt to new tasks with minimal task-specific training. However,
8

---

Published as a conference paper at COLM 2024
how instruction-tuning alters LLM internal representations to achieve these improvements remains an open question. Brain activity provides a neuroscientific angle to investigate this question. Our results show that instruction-tune tunnnn improves both performance on world knowledge benchmarks and human brain alignment, suggesting that the mechanisms that encode world knowledge in LLMs also align LLM representations to the human brain.
6.2 Implications for Neuroscience: Studying LLM-Human Alignment
Instruction-tuned LLMs are useful for studying LLM properties underlying brain and behavioral alignment. To identify why LLM and human brains share representational similarities, prior work has mostly focused on high-level properties such as model size (Antonello et al., 2023), as well as external behaviors such as predicting missing words (Schrimpf et al., 2021; Caucheteux & King, 2022). However, key to understanding these similarities is to identify internal properties of LLMs that underlie brain alignment, in- cluding the amount of knowledge LLMs encode, e.g, factual (AlKhamissi et al., 2022) and commonsense (Bosselut et al., 2019; Sap et al., 2020). Our work is the first to show that we can harness instruction-tuned LLMs for this purpose. Because they have been trained to respond to a general instruction format, we can evaluate LLMs on diverse tasks in fine-grained manner, allowing the study of LLM properties both internal (e.g., knowledge) and external (e.g., behavior), and how they correlate with brain and behavioral alignment.
World knowledge shapes language comprehension and brain activity. Our results show that world knowledge key factor in LLM brain alignment. LLMs demonstrating greater world knowledge across all tested subject domains have representations more similar to the human brain. This suggests that world knowledge influences human brain activity and shapes the language comprehension systems in our brain.
6.3 Limitations and Future Work
Examining more dimensions of behavior. Our work and many prior works compare LLM and human next-word surprisal on reading tasks (Wilcox et al., 2020; Schrimpf et al., 2021; Eghbal A. Hosseini et al., 2023), which evaluates only single behavioral dimension for LLMs (per-word perplexity) and humans (reading times). For the models we tested, behavioral alignment is not significantly correlated with model size, world knowledge, or next-word prediction ability. While next-word prediction performance correlates with alignment to human reading times across many LMs (Schrimpf et al., 2021), this trend does not hold up in recent transformer models (Oh & Schuler, 2023), having a surprising negative correlation with parameter count (Oh al., 2022). In the future, we hope to evaluate LLMs on wider range of behavioral dimensions (e.g,, van Duijn et al., 2023; Koo et al., 2023; Kauf et al., 2024) more holistically evaluate LLM-human behavioral alignment.
Brain alignment datasets with humans performing diverse tasks. We study brain alignment using neural datasets of humans reading naturalistic stories and sentences in English. It would be interesting to study brain alignment to human participants attempting the BBH and MMLU benchmarks, but this data unfortunately does not exist. This may explain why brain alignment is not significantly correlated with many categories of problem- solving on BBH, e.g,, algorithmic reasoning. In the future, we hope to study brain alignment with human participants performing more diverse sets of tasks, e.g., reading computer program code (Ivanova et al., 2020). This can identify more factors underlying LLM-brain alignment, and provide insights into how brain activity and the human language system may be shaped by various forms of groblem-solvingy
Furthermore, some of the larger models exceed the noise ceiling estimates in our neural datasets (Appendix G), highlighting the need for more neural datasets and better ways of computing noise ceiling estimates.
7 Conclusion
We investigate whether instruction-tuning improves the alignment of LLMs to the human language system. We evaluate 25 LLMs with parameter sizes ranging from 77 million to 33 billion, across three neural datasets of humans reading naturalistic stories and sentences. We find that instruction-tuning generally improves the alignment of LLM representations

---

Published as a conference paper at COLM 2024
to brain activity. Exploring the factors underlying LLM-brain alignment, we discover that world knowledge and model size are key determinants of brain alignment. This suggests that world knowledge helps shape representations in the human language system, and highlights the importance of integrating world knowledge in developing future LLMs.
8 Reproducibility Statement
We provide our full results in Appendices G, H, and L. We consider this important as ex- periments may be computationally expensive to replicate. We investigated 25 LLMs, with the largest models having 33B parameters, many datasets: brain alignment (PEREIRA2018, BLANK2014, WEHBE2014), behavioral alignment (FUTRELL2018) next-word prediction (WikiText-2) and other evaluations (BBH, MMLUU.
All models and code repositories used in our study are open-source and their corre- sponding links are provided in Appendix B and for full transparency and reproducibil- ity. To calculate brain and behavioral alignment, we used the Brain-Score repository (www.g. gi thub. com/ cww/brain-seeeeeeeeeeeeeemeeeeee Abrain-scoreeaannneen score/language) a publicly accessible resource for conducting these assessments. We encourage researchers interested in replicating our findings to refer to the provided links and consult the Brain-Score repository for further details on datasets and the evaluation process. To evaluate LLMs on BBH and MMLU, we utilize the widely-used instruct-eval repository (https: //gi wwt.decaaaeeeaab iastee-clelanstttteeelllllllllll with default settings.
References
Badr AlKhamissi, Millicent Li, Asli Celikyilmaz, Mona Diab, and Marjan Ghazvininejad. A Review on Language Models as Knowledge Bases, April 2022. URL http://arxiv.orrg abs/2204. 06031. arXiv:2204.06031 [cs].
Yuvanesh Anand, Zach Nussbaum, Brandon Duderstadt, Benjamin Schmidt, and Andriy Mulyar. Gpt4all: Training an assistant-style chatbot with large scale data distillation from gpt-3.5-turbo. https:// thub. com/ cow/nomiccaanacccccmmmmmoo htmm//mmmi-aa/gttaalll 2023.
Richard Antonello, Aditya Vaidya, and Alexander G. Huth. Scaling laws for language encod- 11863. arXiv:2305.11863
ing models in fMRI, May 2023. URL [cs].
Khai Loong Aw and Mariya Toneva. Training language models to summarize narratives im- proves brain alignment. In The Eleventh International Conference on Learning Representations, 2023.
Idan Blank, Nancy Kanwisher, and Evelina Fedorenko. A functional dissociation between language and multiple-demand systems revealed in patterns of BOLD signal fluctuations. Journal of Newrophysiology, 112(5):1005-1118, September 2014. ISSN 0022-3077, 1522-1598. doi: 10.1152/jj.00844.2000 2013. 2013.
URL https: http:///wwwwhhhhllgg.ororo... htwphhyssoooo..orgoogoooo wwtp:///wo//o//ooo////////// 1152/jn. 00884.
Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin Choi. Comet: Commonsense transformers for automatic knowledge graph construction. In Annual Meeting of the Association for Computational Linguistics, 2019. URL https://api. semanti .oraar.org///o wwtg//orruSS::1111mmmmm 189762227..
Charlotte Caucheteux and Jean-Rémi King. Brains and algorithms partially converge in natural language processing. Communicatiins Biology, 5(1):134, February 2022. ISSN 2399-3642. doi: 10.1038/ /44000000022-00003--11 542803-022-03836
0 URL https: http://wwww..t..eeeeee nawure. coom/rrtille//////
Yew Ken Chia, Pengfei Hong, Lidong Bing, and Soujanya Poria. INSTRUCTEVAL: Towards Holistic Evaluation of Instruction-Tunee Large Language Models, June 2023, URL http:
// //arxiv.et www.ii.org//bb////... wwtg////////aeeeeoe 04757. arXiv:2306.04757 [cs].
10

---

Published as a conference paper at COLM 2024
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%" chatgpt quality, March 2023, URL
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma,, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. Scaling Instruction-Finetuned Language Models, December 2022. URL http: /ttt:///rxiv.... wwwpiiiooorggboooe org/abs/ 2210. 11416. arXiv:2210.1140 [cs].
Joel Dapello, Tiago Marques, Martin Schrimpf, Franziska Geiger, David D. Cox, and James J. DiCarlo. Simulating Primary Visual Cortex at the Front of CNNs Improves Robustness to Image Perturbations, In Neural Information Processing Systems (NeurlPS), June 2020. doi: 00.1101/2020.00.11.15444
Joel Dapello, Kohitij Kar, Martin Schrimpf, Robert Geary, Michael Ferguson, David D. Cox, and James J. DiCarlo. Aligning Model and Macaque Inferior Temporal Cortex Represen- tations Improves Model-to-Human Behavioral Aligmment and Adversarial Robustness. preprint, Neuroscience, July 2022. URL http:://bbiooiii.oggllooooooeeoe 07.01.4984 498495.
1181/2022.
Ishita Dasgupta, Andrew K. Lampinen, Stephanie C. Y. Chan, Antonia Creswell, Dhar- shan Kumaran, James L. McClelland, and Felix Hill. Language models show human- like content effects on reasoning, July 2022. URL http: httt:://arxi.oorr orgg hrtpii..rrgabb//22////// abs/2207.07000 ntXiv:2207.07051 [cs].
Dota Tianai Dong and Mariya Toneva. Interpreting multimodal video transformers using brain recordings. In ICLR 2023 Workshop on Multimodal Representation Learning: Perks and Pitfalls, 2023. URL http:tttss///eeeeett/
net www.:neetfooriie wtt.fformmmmmd--33333 LL3mooogg
Eghbal A. Hosseini, Martin Schrimpf, Yian Zhang, Samuel Bowman, Noga Zaslavsky, and Evelina Fedorenko. Artificial neural network language models predict human brain responses to language even after a developmentally realistic amount of training. January 2023. doi: 10.1101 222.1005/2220004451666661 URL 10. 04. 144.51666 a.strraat.trcttt...............t
bioRxiv, pp- httg///nnneeeeeee/////////////// http://biorxiv http://boorxxvorooo//oo
Susan F. Ehrlich and Keith Rayner. Contextual effects on word perception and eye movements during reading. Journal of Verbal Learning and Verbal Behavior, 20(6):641- 655, December 1981. ISSN 00225371. doi:
URL
https://linkinghuub elsevier. wttp::/r.w..ereerreeeee/////2/22
Richard Futrell, Edward Gibson, Harry J. Tily, Idan Blank, Anastasia Vishnevetsky, Steven Piantadosi, and Evelina Fedorenko. The natural stories corpus. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May 2018. European Language Resources Association (ELRA). URL https://acllanthology www.///-///110000
Changjiang Gao, Shujian Huang, lixing Li, and Jiajun Chen. Roles of scaling and in- struction tuning in language perception: Model vs. human attention. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computa- tional Linguisties: EMNLP 2023, pp. 13042-13055, Singapore, December 2023. Associ- ation for Computational Linguistics. doi: 06653//122223 finddiinnnnnnnnnllpp. https://aclanthology.. org 20233. findings-emne emnlp. 868.
URL
Ariel Goldstein, Zaid Zada, Eliav Buchnik, Mariano Schain, Amy Price, Bobbi Aubrey, Samuel A. Nastase, Amir Feder, Dotan Emanuel, Alon Cohen, Aren Jansen, Harshvardhan Gazula, Gina Choe, Aditi Rao, Catherine Kim, Colton Casto, Lora Fanda, Werner Doyle, Daniel Friedman, Patricia Dugan, Lucia Melloni, Roi Reichart, Sasha Devore, Adeen
11

---

Published as a conference paper at COLM 2024
Flinker, Liat Hasenfratz, Omer Levy, Avinatan Hassidim, Michael Brenner, Yossi Matias, Kenneth A. Norman, Orrin Devinsky, and Uri Hasson. Shared computational principles for language processing in humans and deep language models. Nature Neuroscience, 25(3): 369-380, March 2022. ISSN 1097-6256, 1546-1726. doi: https://wwwt https::/wwwwnnatt nature. www/artiicls////eeee..ct 5atp:///555999-2222 0106-44.
URL
John Hale. A probabilistic earley parser as a psycholinguistic model. In Second meeting of the North American Chapler of the Association for Computational Linguistics on Language technologies 2001 NAACL '01, pp. 1-8, Pittsburgh, Pennsylvania, 2001. Association for Computational Linguistics. doi: 10.3115 011115//077336.17333333 URL http: http:://porr /portal...a..////////// acm. org/ citation. (tt.:iinnccmmmdodd11110100 00073333 1073357.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring Massive Multitask Language Understanding, January 2021. URL http:://rxxiii
hwwgg/ab//0000 03300. arXiv:2009 0003:000000033000 [cs].
Anna A Ivanova, Shashank Srikant, Yotaro Sueoka, Hope H Kean, Riva Dhamala, Una-May O'Reilly, Marina U Bers, and Evelina Fedorenko. Comprehension of computer code relies primarily on domain-general executive brain regions. eLife, 9:e58906, December 2020. ISSN 2050-084X. doi:
URL httpt:t:/eelffesciimeee
org/ httgg/aricceeeeeeee.
Shailee Jain and Alexander G Huth. Incorporating Context Language Encoding Models for fMRL. preprint, Neuroscience, May 2018. URL http:://bboox http:///bioxxxiv.o//ooooo/////// http://www....rrrooooooo////o/// 1ookup/doi/10. 1101/32760)
Carina Kauf, Emmanuele Chersoni, Alessandro Lenci, Evelina Fedorenko, and Anna A. Ivanova. Comparing plausibility estimates in base and instruction-tuned large language models, 2024.
Ryan Koo, Minhwa Lee, Vipul Raheja, Jong Inn Park, Zae Myung Kim, and Dongyeop Kang. Benchmarking cognitive biases in large language models as evaluators, 2023.
Gabriele Merlin and Mariya Toneva. Language models and brain alignment: beyond word- level semantics and prediction, December 2022. URL arXiv:2212.005% [cs, q-bio].
Byung-Doh Oh and William Schuler. Why Does Surprisal From Larger Transformer-Basse Language Models Provide Poorer Fit to Human Reading Times? Transactions of the Association for Computational Linguistics, 11:336-350, March 2023. ISSN 2307-387X. doi:
(..1162/taclaa000
URL htt.httts////oioorrrr wtt.////////..//a.aaaeaeoe 1112//tac_a..055444
Byung-Doh Oh, Christian Clark, and William Schuler. Comparison of Structural Parsers
and Neural Language Models as Surprisal Estimators. Frontiers in Artificial Intelligence,
5:777963, March 2022. ISSN 2624-8212.
doi: 10.3389 2r2.2....02..77666... URL https:
//www. httprtiirriiinongrrrrillee///e/e 33t.:///....33./11///
2022. 777963/ful1.
Subba Reddy Oota, Jashn Arora, Veeral Agarwal, Mounika Marreddy, Manish Gupta, and Bapi Surampudi. Neural Language Taskonomy: Which NLP Tasks are the most Predictive of fMRI Brain Activity? In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3220-3237, Seattle, United States, 2022. Association for Computational Linguistics. doi: 10.18653/v1 2022.naacl-ne moal-mmin.2225 235. URL https:tt//// / hwtpt//aoaaactoccaaaaoacaooo.aae wwanthoogg..org/2eee0 2r2. naacl-main. 235.
Subba Reddy Oota, Manish Gupta, Raju S. Bapi, Jobard,, Frederic Alexandre, and Xavier Hinaut. Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey), July 2023. URL
10246. 2XXi:23377.1026 [cs, q-bio].
Long Ouyang, Jeff WU, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. ArXio, abs/2203.02155,, 2022. URL https: //api. semanti oet.aattccccclll.a..oo////oooooo org//orrpusID:2444229999
12

---

Published as a conference paper at COLM 2024
Francisco Pereira, Bin Lou, Brianna Pritchett, Samuel Ritter, Samuel J. Gershman, Nancy Kanwisher, Matthew Botvinick, and Evelina Fedorenko. Toward a universal decoder of linguistic meaning from brain activation. Nature Communications, 9(1):63,, March 2018. ISSN 2041-1723. doi: 10.1038 1..1008//41467-18--088088 articless/s414477011--333
URL. https:/ /www. wwwure. com/
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the Limits of Transfer Learning with Unified Text-to-Text Transformer, July 2020. URL http:///axiv. www.voorg/aaaoe http://..or/////1/...... http://abs/111..1..55 arXiv:1910.10633 [cs, stat].
J.K. Rowling, M. GrandPre, M. GrandPré, T. Taylor, Arthur A. Levine Books, and Scholastic Inc. Harry Potter and the Sorcerer's 's Stone. Harry Potter. A.A. Levine Books, 1998. TSBN 97805903534033 URL https: //bokks google Ti
Shahd Safarani, Arne Nix, Konstantin Willeke, Santiago Cadena, Kelli Restivo, George Denfield, Andreas Tolias, and Fabian Sinz. Towards robust vision by multi-task learning on monkey visual cortex. Advances in Neural Information Processing Systems, 34:739-751, 2021.
Mustafa Safdari, Greg Serapio-Garcia, Clément Crepy, Stephen Fitz, Peter Romero, Luning Sun, Marwa Abdulhai, Aleksandra Faust, and Maja Matarić. Personality Traits in Large Language Models, June 2023. URL http: //axxvv http://aaxvv...r/bbbbbee wtgg/bs///g// 001 84. arXiv:2307.001 00014 [cs].
Maarten Sap, Vered Shwartz, Antoine Bosselut, Yejin Choi, and Dan Roth. Commonsense Reasoning for Natural Language Processing- In Proceedings of the 58th Annual Meeting of the Association for Computationnl Linguistics: Tutorial Abstracts, pp. 27-33, Online, 2020. URL
Association for Computational Linguistics. doi: 10.18653/v1 https://ww..acwettt org/
Martin Schrimpf, Jonas Kubilius, Ha Hong. Najib J. Majaj, Rishi Rajalingham, Elias B. Issa, Kohitij Kar, Pouya Bashivan, Jonathan Prescott-Roy, Franziska Geiger, Kailyn Schmidt, Daniel L. K. Yamins, and James J. DiCarlo. Brain-Score: Which Artificial Neural Network for Object Recognition is most Brain-Like? preprint, Neuroscience, September 2018. URL or hwtp://wwwrwg/oo//..///ooo wwwgg/oookp//o///// 1101 1111111111110000111 /077077.
Martin Schrimpf, Jonas Kubilius, Michael J Lee, N. Apurva Ratan Murty, Robert Ajemian, and James DiCarlo. Integrative Benchmarking to Advance Neurally Mechanistic Models of Human Intelligence. Neuron, 2020. ISSN 0896-6273. doi: 10.1016 2.1016/..euron.2220.000
Martin Schrimpf, Idan Asher Blank, Greta Tuckute, Carina Kauf, Eghbal A. Hosseini, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. The neural architecture of language: Integrative modeling converges on predictive processing. Proceedings of the National Academy of Sciences, 118(45):e1105666111, November 2021. ISSN 0027-8424, 1091- 6490. doi: 0t0007//ppaa.2205466111 URL https://pnas..or//o//// 2105646118.
p073/pnas.
Dan Schwartz, Mariya Toneva, and Leila Wehbe. Inducing brain-relevant bias in natural language processing models, October 2019. URL http:://arxiv.or//// /ttgg//////////////////// 03266.. arXiv:1911.03266 [cs, q-bio].
Nathaniel Smith and Roger Levy. The effect of word predictability on reading time is logarithmic. Cognition, 128(3):302-319 2013. ISSN 0010-0277. doi: https:/ doi.org/10 110. 0016://.oonnitton.201...00000110 URL /ttp:////www.. wtw.eeeedreett.ccccciieeeeeeeeee pii/5001062771300441
Julius Steuer, Marius Mosbach, and Dietrich Klakow. Large GPT-llike models are bad babies: A closer look at the relationship between linguistic competence and psycholinguistic measures. In Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mosquera, Bhargavi Paranjabe, Adina Williams, Tal Linzen, and Ryan Cotterell (eds.), Proceedings of the BabyLM Challenge al the 27th Conference on
13

---

Published as a conference paper at COLM 2024
Computational Natural Language Learning, pp. 142-157, Singapore, December 2023. As- URL
sociation for Computational Linguistics. doi: 10.18653 /tt////22..cllllbbbblaaa https: fttps:://clanthologg.orrrg///
conll 12.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them, October 2022. URL http:
www.:orr//bbb//e/ee/ 09261. arXiv:2210.09261 [cs].
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following
llama model. //tp ctt.ub..comttttti www.///ttttaaaaeeeeeese htttssslaaassttaoord.lllla.s/
2023.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie Héliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette-Choo, Clément Crepy, Daniel Cer, Daphne Ippolito, David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George- Christian Muraru, Grigory Rozhdestvenskiy, Henryk Michalewski, Ian Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean-Baptiste Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, Justin Mao-Jones, Katherine Lee, Kathy Yu, Katie Millican, Lars Lowe Sjoesund, Lisa Lee, Lucas Dixon, Machel Reid, Maciej Mikula, Mateo Wirth, Michael Sharman, Nikolai Chinaev, Nithum Thain, Olivier Bachem, Oscar Chang, Oscar Wahltinez, Paige Bailey, Paul Michel, Petko Yotov, Rahma Chaabouni, Ramona Comanescu, Reena Jana, Rohan Anil, Ross Mcllroy, Ruibo Liu, Ryan Mullins, Samuel L Smith, Sebastian Borgeaud, Sertan Girgin, Sholto Douglas, Shree Pandya, Siamak Shakeri, Soham De, Ted Klimenko, Tom Hennigan, Vlad Feinberg, Wojeiech Stokowiec, Yu hui Chen, Zafarali Ahmed, Zhitao Gong, Tris Warkentin, Ludovic Peran, Minh Giang, Clément Farabet, Oriol Vinyals, Jeff Dean, Koray Kavukcuoglu, Demis Hassabis, Zoubin Ghahramani, Douglas Eck, Joelle Barral, Fernando Pereira, Eli Collins, Armand Joulin, Noah Fiedel, Evan Senter, Alek Andreev, and Kathleen Kenealy. Gemma: Open models based on gemini research and technology, 2024. URL http://aaxvvvogg/ra//
08295.
Mariya Toneva and Leila Wehbe. Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain), November 2019. URL http://arxiv.org// wwt../s///bbb../.5// 11833. arXiv:1905.1188 [cs, q-bio].
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and Efficient Foundation Language Models, February 2023a. URL http://arxiv. org/abs. 2302. 13971. arXiv:2302.13971 [cs].
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brían Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Maric-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, lliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023b. URL https:/ www.iiwwww.//b....eee org/abs/2300) 09288.
14

---

Published as a conference paper at COLM 2024
Max van Duijn, Bram van Dijk, Tom Kouwenhoven, Werner de Valk, Marco Spruit, and Peter van der Putten. Theory of mind in large language models: Examining perfor- mance of 11 state-of-the-art models children aged 7-10 advanced tests. In Jing Jiang, David Reitter, and Shumin Deng (eds.), Proceedings of the 27th Conference on Com- putational Natural Language Learning (CoNLL), pp. 389 402, Singapore, December 2023. Association for Computational Linguistics. doi: 2w.1663/v1/2023..om https://aclanthollog. wtg/2023.0.. con1l-1. wrg//223..ccm111111
URL
Yizhong Wang, Yeganch Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions, 2022a.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Gary Lai, Ishan Purohit, Is- hani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Maitreya Patell Kuntal Kumar Pal, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Shailaja Keyur Sampat, Savan Doshi, Siddhartha Mishra, Sujan Reddy, Sumanta Patro, Tanay Dixit, Xudong Shen, Chitta Baral, Yejin Choi, Noah A. Smith, Hannaneh Hajishirzi, and Daniel Khashabi. Super- NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks, htapgg/bb//22200 07705. arXiv:2204.07705 [cs].
October 2022b. URL
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Pa- tel, MeMrrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Shen. Super-NaturallInstuuttoi
Generalization via declarative instructions on 1600+ NLP tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 5085-5109, Abu Dhabi, United Arab Emirates, December 2022c. Asso- ciation for Computational Linguistics. doi: 386533/112022.emm/// mainn-. URL https://aclanttiologyy wtgg//22 emnlp-main. 340. 340.
Leila Wehbe, Brian Murphy, Partha Talukdar, Alona Fyshe, Aaditya Ramdas, and Tom Mitchell. Simultaneously Uncovering the Patterns of Brain Regions Involved in Different Story Reading Subprocesses. PLoS ONE, 9(11)=e11275, November 2014. ISSN 1932-6203. doi: 10.1371 /o..3///juumalllooe.ooo 01 0112575.
URL https: //tx www..ppls...ooo....... org/10.111 /
Ethan Gotlieb Wilcox, Jon Gauthier, Jennifer Hu, Peng Qian, and Roger Levy. On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behavior, June 2020. URL
orgg/bs/////6 01912. arXiv:2006.01912 [cs].
Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, and Guoyin Wang. Instruction Tuning for Large Language Models: A Survey, August 2023. URL http///xxiv.or//////////// 10792. arXiv:2308.10790 10799 [cs].
15

---

Published as a conference paper at COLM 2024
A Language Models: Parameter count and Number of Layers
Model
Parameter Count
Number of Layers
t5-small
77 M
16
flan-t5-small
77 M
16
t5-base
250 M
24
flan-t5-base
250 M
24
flan-alpaca-base
250 M
24
t5-large
800 M
48
flan-t5-large
800 M
48
flan-alpaca-large
800 M
48
t5-xl
3
48
flan-t5-xl
3 B
48
flan-alpaca-xl
3
48
flan-gpt4all-xl
3 3B
48
flan-sharegpt-l
3
48
flan-alpaca-ppé4y.
3
48
t5-xxl
11B
48
flan-t5-xxl
11B
48
flan-alpaca-xx
11
48
Ilama-7b
7 B
32
alpaca-7b
7 B
32
vicuna-7b
7 B
32
Ilama-13b
13
40
vicuna-13b
13 B
40
stable-vicuna-13b
13
40
Ilama-33b
33
60
vicuna-33b
33
60
gpt2-small
124 M
12
gpt2-small-alpaca
124 M
12
gpt2-medium
355 M
24
gpt2-medium-alpaca
355 M
24
gpt2-large
774 M
36
gpt2-large-alpaca
774 M
gpt2-xl
1.5 B
48
gpt2-xl-alpaca
1.5
48
Table 3: Parameter count and number of layers for all vanilla and instruction-tuned LLMs. For the parameter count, "M" refers to million and "B" refers to billion. The number of layers for T5 models is sum of the number of encoder and decoder layers.
16

---

Published as a conference paper at COLM 2024
B
Language Models: Links to models weights
Model
Link to model weights
t5-small
www.
huggingface.
co/ggoogle/t5--111-smmal.]
flan-t5-small
www.
huggingface
co/ggoogle/Flan--555ms1]]]]
t5-base
flan-t5-base
www. www.hugg
www.hugggngaac..commm oag.g///o.....ooooooo////aaaaaaa
flan-alpaca-ban
www.huugingface.colooo
15-large
www www. wuggingfaccgeeennoooooggggggggge
flan-t5-large
www 1
flan-alpaca-large
t5-xl
www www.h/uwiiiiieeeerroooo/o//////.
flan-t5-xl
www. www. www.lrgghiggheccccocc/oo///e//ar www. www. www www.hhggihggiieeeo////eeeeeee/er www www.lnngggnggaae.oooooogg//////. www.tnugginggccccccooo///o//ooar
flan-alpaca-xl
flan-gptHall-l
flan-sharegp--x
flan-alpaca-gpt4-i t5-xxl
flan-t5-xxl
flan-alpaca-xll
www. www.luggirffecccccccccceee/////r
llama-7b
www. www.ggthh..ccm//eeebborrrreeeree
alpaca-7b
www. www.g//hw...o/aaatau.aa////aaaaa www.uuggiigggccccoo/////////otto
vicuna-7b
llama-13b
vicuna-13b
www.tuggiinnffeeeeoo///sssss/sss
stable-vicuna-13b
Ilama-33b
vicuna-33b
gpt2-small
ht
gpt2-small-alpaca
gpt2-medium
gpt2-medium-alpaca
gpt2-large
gpt2-large-alpaca gpt2-xl
gpt2-xl-alpaca
Table 4: Link to model weights for all vanilla and instruction-tuned LLMs. We provide
these links for reproducibility purposes.
17

---

Published as a conference paper at COLM 2024
C
Method for computing Brain alignment
Language model
representations
1. Train linear regression
Language stimuli
Perotr2018
We nepplig zige the remernaiion olocal every bessaapere......
England wore jurrey
Human
Wihbx20144
Day ha irree thnn Dudley
[MRI activity
2. Test on held-out data
Figure 4: Method for computing Brain alignment, the similarity of an LLM's internal
representations human brain activity.
We follow a general approach for aligning LLM representations to fMR brain activity used in prior works (Jain & Huth, 2018; Toneva & Wehbe, 2019; Schrimpf al., 2021). The approaches used for the three neural datasets are similar. Hence, we explain the main details only for WEHBE2014, followed by key differences for the other two datasets.
C.1 WEHBE2014
WEHBE2014 (Wehbe et al., 2014): The data includes fMRI recordings of 8 human participants reading chapter of the book Harry Potter and the Sorceror's 's Stone (Rowling et al., 1998). Participants read the chapter at fixed interval of one word every 0.5 seconds. The chapter contains 5,176 words.
LLM representations. We extract LLM representations of the text at each layer Rexd where w is the number of words in the text and d is the embedding size of the LLM.
fMRI BOLD signals extracted from the language network for all three neural datasets. fMRI activity was recorded at regular intervals of 2 seconds, with a total of 1,351 recorded time-points (TRs, times of repetition). We construct stimulus-response matrix for fMRI activity R where TR the number of TRs and D; is the number of voxels in the brain of participant
fMRI activity. Data analyses were performed
Alignment of LLM representations to fMRI activity. We use PCA reduce the dimension- ality of the LLM representations to result a matrix E R@x10 As the number of words in the text is greater than the number of TRs, we down-sample the LLM word-level repre- sentations to the TR rate by averaging the LLM representations to the corresponding TRs, producing a matrix E IR Tix10 As the response recorded by fMRI peaks about 6 seconds after stimulus onset, time-point. The final LLM representations are constructed by concatenating, for each TR, the
follow prior methods by including preceding time-points for each
LLM representations corresponding to the previous 4 TRs, producing matrix E I TRx40 This accounts for the lag in the hemodynamic response that fMRI records. Finally, we learn linear function, regularized by the ridge penalty, that uses the modified LLM representations E IR TRx40 to predict the fMRI activity of participants E TR We train the function using 4-fold rros-validattin, where each fold corresponds to a separate run of fMRI collection. We also remove 10 TRs from the LLM representations and fMRI activity from the ends of each test run to avoid train-test overlap. To compute the brain alignment, we evaluate the Pearson correlation (r) between the predicted fMRI activity E 7 and recorded fMRI activity E R
C.2 BLANK2014
BLANK2014 (Blank et al., 2014): The data consists fMRI recordings of 5 human participants listening to 8 naturalistic stories from the Natural Stories Corpus (Futrell et al., 2018).
18

---

Published as a conference paper at COLM 2024
fMRI activity was recorded at regular intervals of 2 seconds, with total of 1,317 TRs. We average the BOLD signals across voxels within each language-responsive region of interest (ROI) of each participant to increase the signal-to-noise ratio, following Schrimpf et al. (2021). As there are 60 ROIs in total across the 5 participants, this produces a stimulus-response matrix E R1317x60
C.3 PEREIRA2018
PEREIRA 2018 (experiments 2 and 3 from Pereira et al., 2018): In experiment 2, 9 participants read 384 sentences taken from 96 text passages. In experiment 3, 6 participants read 243 sentences from 72 text passages. Each sentence was displayed for 4 seconds on screen.
We average the fMRI responses for each sentence, resulting in one data point per sentence per mnnuuage-respossive voxel of each participant. For experiment 2, there are 384 sentences and 12,195 language-responsing voxels across the participants. For experiment 3, there are 243 sentences and 8,121 language-responiie voxels across the participants. We concatenate the fMRI responses across sentences and participants for each experiment. This produces rtimolus matrix R384x12,195 for experiment 2 and a stimulus-response matrix 2223x8121 for experiment 3.
D Method for computing Behavioral alignment
Language model
perplexity for each
Language stimull
Futro02018
we'v to zummy to Morth England.
Human
Pearson correlation
Compute
reading times each
Figure 5: Method for computing Behavioral alignment. The same language stimuli are presented to LLMs and human participants, using the FUTRELL2015 benchmark in Brain- Score, which contains naturalistic stories. We compute the behavioral alignment the Pearson correlation between LLM perplexity for each word and human reading times for each word.
FUTRELL12018 (Futrell et al., 2018): The data consists of reading times (RTs) of 179 par- ticipants on total of 10,256 words across 485 sentences, taken from 10 stories from the Natural Stories Corpus (Futrell et al., 2018). The stories were presented online to Amazon Mechanical Turk users visually in dashed moving window display. Participants press button to reveal each consecutive word. When they press the button, the most recent word converted back to dashes, and the next word is revealed. The time between button presses offers an estimate of comprehension difficulty (Ehrlich & Rayner, 1981; Hale, 2001; Smith & Levy, 2013). We construct stimulus-response matrix of RTs E R 110,256x177)
We provide the same stimuli to an LLM and record its perplexity for each word, producing an LLM perplexity matrix E R 10,256 We compute behavioral alignment as the Pearson correlation between per-word LLM perplexity and human RTs. Perplexity for LLMs and reading times for humans offer insights into comprehension difficulty (Ehrlich & Rayner, 1981; Hale, 2001; Smith & Levy, 2013), allowing us to examine whether LLMs and humans share similarities in terms of which words and sentences they find challenging or surprising.
19

---

Published as a conference paper at COLM 2024
E MMLU and BBH benchmarks
MMLU is designed to measure knowledge from many domains Hendrycks et al. (2021). It contains 57 tasks, categorized by the subject domain of world knowledge tested: STEM, Humanities, Social Sciences, and Others. The STEM category includes questions on com- puter science, physics, mathematics, etc. The Humanities category includes questions on philosophy, law, history, etc. The Social Sciences category includes questions on politics, sociology, economics, geography, etc. The Others category includes questions on business topics such finance, accounting, well general knowledge of global facts.
BBH designed to evaluate various problem-solving and reasoning abilities of LLMs (Suzgun et al., 2022). BBH contains 23 tasks, categorized by the type of problem-solving ability tested: (1) Algorithmic and Multi-Step Arithmetic Reasoning, (2) Natural Language Understanding, (3) Use of World Knowledge, and (4) Multilingual Knowledge and Rea- soning. The world knowledge category of BBH contains tasks that test for factual and general knowledge. Tasks requiring factual knowledge include: "Sports Understanding" and "Movie Recommendation" Tasks requiring general knowledge include: "Causal Judge- ment", which tests knowledge about causal-resoninng suppositions, and "Ruin Names", which requires knowledge about human perception and usage of humor in the English language.
For both benchmarks, we adopt the same category classification used in their original papers. We measure the performance of LLMs on BBH and MMLUU using the instruct-eval repository with default settings (3-shots, 5-shots respectively) and preset prompts.
Code Repositories
We use the Brain-Score repository to evaluate brain alignment for the PEREIRA2018 and BLANK2014 datasets, well behavioral alignment for the FUTRELL2018 dataset. Link: www.gi thub. commbbrain-ssorr/langggggg
We use open-source repository evaluate brain alignment for the WFHBE2014 4 dataset. Link: www. gi thub. com/
www.ssssmmaiiiinooooo which builds on www. www.gi thub.
We use the Instructt to evaluate MMLU and BBH scores. Link: www.gi thub.
We use the Stanford Alpaca repository for instruction-tuning. Link: www.githun com/ tatsu--lab/staaforr.llpee..
20

---

Published as a conference paper at COLM 2024
G
Results for Brain alignment
PEREIRA2018
BLANK2014
WEHBE2014
Average
t5-small
0.166
0.168
0.071
0.135
flan-t5-small
0.202
0.178
0.079
0.153
t5-base
0.222
0.188
0.074
0.162
flan-t5-base
0.234
0.178
0.076
0.163
flan-alpaca-base
0.227
0.179
0.076
0.161
t5-large
0.270
0.082
0.071
0.141
flan-t5-large
0.311
0.104
0.080
0.165
flan-alpaca-large
0.322
0.126
0.082
0,177
t5-x1
0.285
0.192
0.072
0.183
flan-t5-xl
0.314
0.215
0.072
0.200
flan-alpaca-xl
0.312
0.209
0.075
0.199
flan-gpt4all-il
0.300
0.206
0.078
0.195
flan-sharegpt-xl
0.323
0.211
0.070
0.201
flan-alpaca-ppt4xx
0.302
0.205
0.073
0.193
t5-xxl
0.343
0,297
0.096
0.246
flan-t5-xxl
0,350
0.268
0.103
0.240
flan-alpaca-xx
0.346
0.268
0.102
0.239
Ilama-7b
0.405
0.154
0.118
0.226
alpaca-7b
0.420
0.167
0.118
0.235
vicuna-7b
0.399
0.152
0.119
0.223
Ilama-13b
0.412
0.133
0.115
0.220
vicuna-13b
0.423
0.148
0.116
0.229
stable-vicuna-13b
0.415
0.144
0.115
0.225
Ilama-33b
0.426
0.145
0.109
0.227
vicuna-33b
0.436
0.156
0.105
0.232
gpt2-small
0.305
0.123
0.088
0,172
gpt2-small-alpaca
0.298
0.121
0.081
0.167
gpt2-medium
0.329
0.081
0.089
0.166
gpt2-medium-alppan
0.325
0.080
0.090
0.165
gpt2-large
0.342
0.077
0.101
0.173
gpt2-large-alpaca
0.336
0.074
0.101
0.170
gpt2-xl
0.358
0.140
0.102
0.200
gpt2-xl-alpaca
0.343
0.139
0.093
0.192
Table 5: Brain alignment results for all vanilla and instruction-tuned LLMs. We provide these results for reproducibility purposes.
PEREIRA2018
BLANK2014
WEHBE2014
Average
Noise ceiling
0.359
0.210
0.104
0.224
Table 6: Noise ceiling estimates for all neural datasets. fMRI measurements inherently include noise, i.e., fluctuations not due neurons firing, from sources such as environmental factors, the scanner, and movements of human participants. Consequently, "noise ceiling" often computed by recording multiple samples from the same subjects for the same stimuli, providing
upper-limit estimate of the possible correlation in fMRI activity. In our study, we followed prior work for calculating the noise ceiling. For PEREIRA2018 and BLANK2014, 4, noise ceiling estimates are computed using the Brain-Score repository, with details provided in Schrimpf et al. (2021). For WEHBE2014 noise ceiling estimates are computed using similar procedure.
21

---

Published as a conference paper at COLM 2024
H
Results for Next-word prediction, MMLU, BBH
WikiText-2 NWP Loss
MMLU Score
BBH Score
flan-t5-small
0.851
0.294
0.287
flan-t5-base
1.235
0.341
0.308
flan-alpaca-base
1.074
0.304
0.266
flan-t5-large
0.625
0.419
0.370
flan-alpaca-large
0.648
0.397
0.276
flan-t5-xl
0.650
0.493
0.402
flan-alpaca-xl
0.604
0.466
0.270
flan-gptiall-x
0.625
0.337
0,212
flan-sharegpt-xl
0.664
0.446
0.363
flan-alpaca-gpt--xl
0.593
0,456
0.348
flan-t5-xxl
0.638
0.545
0.443
flan-alpaca-xxl
0.607
0.508
0.229
alpaca-7b
4.201
0.404
0.328
vicuna-7b
4.387
0.472
0.331
vicuna-13b
4.130
0.521
0.387
stable-vicuna-13b
4.623
0.495
0.380
vicuna-33b
3.940
0.590
0.426
gpt2-small-alpaca
4.193
0.270
0.273
8p12-medium-alpacca
3.333
0.260
0.276
gpt2-large-alpaca
3.614
0.260
0.278
gpt2-xl-alpaca
3.283
0.270
0.284
Table 7: WikiText-2 NWP loss, MMLU Overall Score, and BBH Overall Score for all instruction-tuned LLMs. Results for vanilla LLMs are not shown they are not adapted for the question formats in the MMLU and BBH benchmarks. Results in gray are close to random performance. We provide these results for reproducibility purposes.
Notes on comparing next-word prediction (NWP) loss across model families. The T5 and LLaMA models belong to separate model families. We wish to caution that comparing next-word prediction loss across different model families may not be meaningful. This due several reasons related architectural differences, training methodologies, and objectives. (1) Architecture: T5 models have an encoder-decoder architecture while LLaMA models have a decoder-only architecture. (2) Training Objectives: The T5 models were trained on supervised and unsupervised tasks, while the LLaMA models were trained only on unsupervised text (Section 3). (3) Loss computation: The loss functions for both model families are computed differently, making it inappropriate to directly compare their loss values. (4) Evaluation Metrics: Next-word prediction loss is just one metric, and it may not capture the overall language understanding capabilities of model. Hence, additionally evaluate these LLMSs' alignment to human brain activity, as well as their performance on problem-solving abilities (BBH) and tasks requiring world knowledge (MMLU). In summary, while NWP loss a valuable metric for evaluating language models within the same family or architecture, comparing across different model families may not be meaningful.
22

---

Published as a conference paper at COLM 2024
I Results for Correlations of Brain Alignment with LLM properties
Brain alignment
LM properties
Statistical significance: p (n.s.). ), p (*). p 0.005 (*"), etc.
A
0.81,
8.52e-05
B
0.68, p 2.72e-03
0.24
0.24
BeneS-al
Brain alignment (r)
Brain alignment (r)
strura-33
T5
vicune Eibi--kons-133
0.22
0.22
Liama
0.20
0.20
a
Tan-elieca--eee spit-al
Faan-a space sp244
T5
0.18
0.18
Llama
0.16
Ban Doose
flnnis large
Perera2018
0.16
Tanals
fan-t5 fan-t--argge large
Pareira201
Banc5 small
Bia x2014
Bani5 small
Blark2014
Wenbe2014
Wenbe2014
0.14
0.14
0.30
0.35
0.40 0.45 0.50 0.55
0.60
0.30
0.35
0.40
0.45
0.50
0.55
MMLU Overall score
BBH world knowledge
C
0.95, p 9.43e-09
D
-0.54, p 2.68e-02
0.24
T5
Rop Rae
0.24
T5
Llama
x 3un
Llama
Maate S
Brain alignment (r)
Brain alignment (r)
si Ao
0.22
0.22
Pereiro2018
Blank2014
Weh
RoR Saarrrt
0.20
Parsenn 20
0.18
0.16
Ban
[anntt-large
201e
0.16
Cace base
Blank2014
f S-amal
Wenbs2014
0.14
0.14
9.5
10.0
10.5
11.0
11.5
15
10
log10(# model parameters)
NWP loss
Figure 6: Correlation between brain alignment and various LLM properties: (A) MMLU benchmark overall score, (B) BBH benchmark score for world knowledge tasks, (C) number of parameters of the model, and (D) Next-word prediction (NWP) loss.
23

---

Published as a conference paper at COLM 2024
Brain alignment
LM properties
Stalistical significance: p 0.05 (n.s.). 0.05 (*), 0.005 (*)) etc.
A
B
0.92,
7.19e-11
-0.54, p 4.94e-03
0.26
0.26
T5
T5
0.24
Llama
fan
0.24
Llama
Brain alignment (r)
1.3
0.22
0.22
Peretra3016
Brain alignment
Bankz204
0.20
lent G
0.20
Ran (0l-llg
Fa Bwil gynt-ol
15
lpece-lage
Fan-ES-brge
Starge
BB Sight
0.16
0.16
Ban
Pa1e
Blonk2014
TM
Wenbo2014
0.12
0.12
9.0
9.5
10.0
10.5
11.0
11.5
15
10
log10(# model parameters)
NWP loss
Figure 7: Correlation between brain alignment and various LLM properties for all LLMs (including vanilla models): (A) number of parameters of the model, and (B) Next-word prediction (NWP) loss.
J Additional Brain Alignment results (Linear, RSA, CKA) for Gemma and LLaMA-2 models
Pretrained models
Instruction-tuned models
Model
Linear
CKA
RSA
Model
Linear
CKA
RSA
Gemma-2B
0.76
1.94
0.63
Gemma-2B
0.72
2.44
0.70
Gemma-7B
0.90
2.25
0.62
Gemma-7B
0.94
2.55
0.72
I.LMAA2-7B
0.97
1.81
0.65
I.L.MA-27BB
1.02
2.00
0.63
LLaMA-2-13B
1.08
2.51
0.68
LLaMA-2-13B
1.08
2.77
0.65
Table Additional Brain Alignment results (Linear, RSA, CKA) for Gemma and LLaMA-2 models. "Linear" refers to the linear predictivity metric used in the main paper, normalized by the noise ceiling computed in Appendix G.
In Section 4, we study the representational similarity between LLMs and human brain activity (brain alignment) using the linear predictivity similarity metric (" ("Linear"). This metric involves training linear regression model to predict fMRI brain activity based LLM representations. Here, we additionally validate our results using the RSA and CKA similarity metrics (Table 8), confirming that the brain alignment results are consistent across different similarity metrics.
In Section 4.1, we observed that instruction-tuning generally improves brain alignment for models from the T5 and LLaMA-1 families. We further validate this trend with recently released models from the Gemma and LLaMA-2 families (Table 8). These results hold across various similarity metrics, including Linear predictivity, RSA, and CKA.
24

---

Published as a conference paper at COLM 2024
K
Results for Instruction-tuinng LLaMA-7B on Alpaca dataset
Brain alignment
LM properties
B
0.235
Instruction No instruction
Brain alignment tO
BBH Overall score
0.32
Overall
Instruction No instruction
0.230
0.31
0.30
Instruction
0.29
No instruction
0.225
0.5
100
1.5
2.0
2.5
3.0
2.5
Instrutioo-tuiing epochs
Instruction tuningg opochs
Instruction tuning epochs
Figure 8 Improvements brain alignment from instruction-tuning are due to both additional training data, well training to understand and follow instructions.
Instruction model. We instruction-tune LLaMA-7B on the Stanford Alpaca dataset (Taori et al., 2023) using the default training process, following the code in www. thub. com/ tatsu-lab/ httbb//tanford..lppp. instruction-following examples generated through methods inspired by Self-Instruct (Wang et al., 2022a). This model is labeled "Instruction" in Figure 8.
In particular, the model is instruction-tuned using 52K
No instruction model (Ablation). We also train an ablation model with the same process and training data as the default instruction-tuning, but remove the instruction portion from each training sample. This ablation model is labeled "No instruction" in Figure 8. This ablation experiment disentangles:
(1) training data (present in both "Instruction" and "No instruction" ), from (2) training LLMs to understand and follow instructions (present only in "Instruction"".
We use all provided training samples from the Alpaca dataset, thus ensuring that the models are trained on the same amount of data. We observe that brain alignment of the "No Instruction" ablation model increases during fine-tuning but stays lower than its "Instruction" counterpart. This shows that brain alignment improvements are due to both (1) training data (present in both models) and (2) the process of training LLMs to understand and follow instructions (present only in the "Instruction" model).
25

---

Published as a conference paper at COLM 2024
L
Results for Behavioral alignment
FUTRELL2018
t5-small
0.229
flan-t5-small
0.054
t5-base
0.333
flan-t5-base
0.152
flan-alpaca-base
0.290
15-large
0.303
flan-t5-large
0.145
flan-alpaca-large
0.291
t5-xl
0,225
flan-t5-xl
0.113
flan-alpaca-xl
0.181
flan-gpt4all-X
0.251
flan-sharegpt-xl
0.285
flan-alpaca-ppt-xl
0.250
t5-xxl
0.260
flan-t5-xxl
0.274
flan-alpaca-xxl
0.267
llama-7b
0,204
alpaca-7b
0.206
vicuna-7b
0.205
llama-13b
0.184
vicuna-13b
0.184
stable-vicuna-13b
0.196
Ilama-33b
0.158
vicuna-33b
0.164
gpt2-small
0.367
gpt2-small-alpaca
0.335
gpt2-medium
0,350
gpt2-medium-alpaae
0.345
gpt2-large
0.331
gpt2-large-alpaca
0.338
8pt2-xl
0.318
gpt2-xl-alpaca
0.336
Table 9: Behavioral alignment results for all vanilla and instruction-tuned LLMs. We provide these results for reproducibility purposes.
FUTRELL2018
Noise ceiling
0.858
Table 10: Noise ceiling estimates for the FUTRELL2018 reading-times dataset. Noise ceiling estimates are computed using the Brain-Score repository, with details provided in Schrimpf et al. (2021).
26

---

Published as a conference paper at COLM 2024
M Results for Correlations of Behavioral Alignment with LLM properties
A
n.s. (p=0.76)
C
(p=0.54)
0.30
tan-alpace dasn Sappeaa
0.30
tian-alpaca-base
Nam Nandsool
0.25
flan gpt4all
gat4idd
0:25
Behavioral alignment
Behavioral alignment
0.20
vicuna-
0.20
kor--ppkGupp-1133 -13
fannalpacaa
vicune-336
0.15
faan base
Pan---largg
0.15
ftan-tS-base
fen-t5-eerge
FeneSod
faan-S-ed
0.10
T5
Liama
T5
flan amail
Non----salll
MMLU Overall score
NWP loss
B
n.s. (p=0.31)
D
n.s. (p=0.21)
0.30
Tan-ely Eon- Stin fir ARERGregtt
vicuna-78( alpaca
Ren aadielllggbt
0.20
Behavioral alignment
Behavioral alignment
stablewicunaad13
cunna--7 70
stable-iicune-13
0.19
Vicuna
vicume-13b
base fichtt--large
0.18
T5
0.17
Liama
x
Liama
Ran-5-small
vicuna-33b
0.05
9.0
9.5
10.0
10.5
11.0
11.5
4.6
4,4
1.2
4.0
log10(# model parameters)
NWP loss
Figure 9: Correlation between behavioral alignment and various LLM properties: (A) MMLU benchmark overall score, (B) number of parameters of the model, (C) Next-word prediction (NWP) loss for T5 models, and (D) NWP loss for LLaMA models.
27


Reviews:
[{"review_id": "nClM2ilXkR", "reviewer": "Reviewer_edNc", "summary": "This work investigates whether instruction-tuning can make LLMs process language similar to humans. In particular, this work studies:\nBrain Alignment: How closely LLMs’ internal representations match neural activities in the human language system.\nBehavioral Alignment: How similar LLM behaviors are to human behaviors during language tasks.\nComprehensive and carefully-designed experiments are conducted to draw the conclsion:\n\nInstruction-tuning generally aligns LLM representations to human brain activity and instruction-tuning LLMs generally does not enhance behavioral alignment with human reading times.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate that the reviewer found our experiments comprehensive and carefully designed, our research problem very interesting, and our findings meaningful.\n\n1. Discuss more insights to build brain-/human-intelligence-aligned LLMs\n\nWe thank the reviewer for the suggestion. In our revision, we will add a subsection discussing papers with insights on LLM and human cognition [1-2] and how to build brain-aligned LLMs [3]: next-word prediction [4], data/model size [5], fine-tuning [6], semantic understanding [7], among others.\n\n2. Study connections between brain alignment and training data\n\nWe trained LLaMA-7B on the Alpaca instruction dataset and an ablated version without the instruction portion for each sample (Figure 1D, Appendix K). The ablated model increases less in brain alignment than the model tuned with instructions, suggesting that brain alignment benefits from both the training examples, but also the instruction format; this opens the way for further exploration.\n\n3. Test reasoning-specific models (e.g., Math)\n\nThe reviewer suggests measuring brain alignment for reasoning-specific datasets to elucidate why LLM performance on reasoning tasks is not well correlated with brain alignment improvement from instruction tuning. \nWe limit our scope to the human language system and use datasets where participants read/listened to English text and did not engage in specific tasks related to abilities such as mathematical reasoning. Prior work tackled an fMRI dataset of participants reading code, which could be linked with models’ ability to understand and generate code [8]. However, they measure alignment with the Multiple Demand System of the brain which is known to be involved in domain-agnostic tasks like problem-solving, logic, and spatial memory tasks.\n\n[1] Dissociating language and thought in large language models: a cognitive perspective\n\n[2] Roles of Scaling and Instruction Tuning in Language Perception: Model vs. Human Attention\n\n[3] Getting aligned on representational alignment\n\n[4] The neural architecture of language: Integrative modeling converges on predictive processing\n\n[5] Scaling laws for language encoding models in fMRI\n\n[6] Divergences between Language Models and Human Brains\n\n[7] Lexical semantic content, not syntactic structure, is the main contributor to ANN-brain similarity of fMRI responses in the language network\n\n[8] Convergent Representations of Computer Programs in Human and Artificial Neural Networks", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Very interesting research problem. since the instruction tuning has been a de-facto choice for training LLMs, this study can provide some insights into whether instruction tuning can lead to human-level or human-like intelligence. Whatever the answer is, the research question per se is important and worth investigating.\n2. The experiment setup is valid and comprehensive: four types of langauge abilities, 25 LLMs (instruction tuned or not) with different sizes.\n3. Intersting and meaningful findings.", "reasons_to_reject": "1. It would be great to provide more insights/future directions to how to build more brain-/human-intelligence-aligned LLMs.\n\nThe rest weaknesses here are more like a suggestion not weakness:\n2. It would interesting to see the connections between the alignment and the training data, which will provide more insights to determine what's lacking in the current instruction tuning to build more brain-aligned LLMs.\n\n3. I notice the behaviors of LLMs on reasoning tasks is not well aligned with human. I wonder whether it's a training data issue or model arch issue. There are reasoning-specific instruction tuning models such as Mammoth for math reasoning. It would be great if the models are specificly trained for a specific task can increase the alignment.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ksbIr3XpJt", "reviewer": "Reviewer_uLs2", "summary": "This paper explores the correlation between instruction-tuning and brain alignment as well as behavioral alignment, demonstrating that instruction-tuning can enhance scores for brain alignment without increasing behavioral alignment scores. Additionally, the paper analyzes the correlation between brain alignment, behavioral alignment, model size, world knowledge, and model capabilities.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate that the reviewer found our analysis of instruction-tuning (IT) comprehensive, using many LLMs from multiple model families.\n\n1. Figures 2 and 4 are somewhat unclear\n\nWe will remove model names and adapt point sizes to model sizes for better readability. We provide larger figure versions with model names in the Appendix.\n\n2. Test LLaMA2, Mistral. Study RLHF’s effect on brain alignment\n\nInstead of testing many models with highly similar architectures, we prioritized models from different families, particularly those with varying model sizes (LLaMA, T5, GPT-2). Below are results normalized relative to the estimated noise ceiling for LLaMA2 and Gemma, showing that IT models (right score) have higher alignment than vanilla (left score).\n\nRegarding the impact of RLHF, which aligns LLMs to human preferences, on LLMs alignment to human brain activity and behavior: we studied one RLHF model (StableVicuna-13B), but our findings were insufficient to draw rigorous conclusions. Our findings are more comprehensive for IT, the focus of our paper, though we agree rigorous investigation of RLHF’s effect on brain alignment is an interesting future direction. \n\n|Model|Linear|CKA|RSA|\n|-|-|-|-|\n|Gemma-2B|.76/.72|1.94/2.44|.63/.70|\n|Gemma-7B|.90/.94|2.25/2.55|.62/.72|\n|LLaMA2-7B|.97/1.02|1.81/2.0|.65/.63|\n|LLaMA2-13B|1.08/1.08|2.51/2.77|.68/.65|\n\n3. Effect of word length on behavioral alignment\n\nBehavioral alignment evaluates the correlation between per-word reading times and LLM per-word surprisal for the same stimuli. We follow [1] and implicitly account for word length by computing the surprisal of a word with multiple tokens as the sum of surprisals of each token. Importantly, we use consistent tokenizers when comparing vanilla LLMs and IT versions, thus controlling the effect of word length when evaluating the impact of IT on behavioral alignment.\n\n4. Run other brain alignment methods (e.g., RSA) aside from Brain-Score linear predictivity\n\nWe focused on linear predictivity following [1], but will add more comprehensive results for RSA [2] and CKA [3] to our revision. Early experiments (see table above) indicate similar patterns for LLaMA2 and Gemma models, except for RSA for LLaMA2.\n\n[1] The neural architecture of language: Integrative modeling converges on predictive processing\n\n[2] Representational similarity analysis - connecting the branches of systems neuroscience\n\n[3] Similarity of Neural Network Representations Revisited", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThe study evaluates 25 LLMs from the T5 and LLaMA families, as well as 8 models from the GPT2 family, including their original versions and various instruction-tuned versions, offering a comprehensive analysis of instruction tuning across these three model families.\n2.\tThe paper examines the correlation between brain alignment, behavioral alignment, and the characteristics of LLMs.", "reasons_to_reject": "1.\tFigures 2 and 4 (B, C, D) are noted to be somewhat unclear, suggesting a need for refinement in their presentation for better clarity.\n2.\tThe paper would benefit from an expanded analysis that includes models such as LLaMA2 and Mistral to more effectively demonstrate the impact of instruction tuning. Additionally, the inclusion of an analysis on the correlation between large language model alignment methods, such as RLHF, and brain alignment would contribute to the comprehensiveness of the study.\n3.\tIt is recommended that the use of reading time in behavioral alignment be reconsidered to include the potential impact of word length. The analysis could potentially be more thorough if it differentiates between long and short words based on a defined word length threshold, and also takes into account the reading time adjusted for word length, such as by using a reading time to word length ratio.\n4.\tRepresentational Similarity Analysis (RSA) and Brain-Score can both indicate brain alignment, and the authors might explore whether RSA could lead to the same conclusions.", "questions_to_authors": "refer to the comments", "extra_scores": {}}, {"review_id": "JczOR9iObd", "reviewer": "Reviewer_hcZZ", "summary": "This paper is an investigation into whether instruction-tuning of LLMs affects the alignment between intermediate LLM representations and brain activity as measured by fMRI.  Alignment with brain representations is measured using the Brain-Score linear predictivity measure from prior work, and this measure is computed across a variety of language models both with and without instruction-tuning.  The experiments find that instruction tuning increases brain alignment by 6.2% on average across different base and instruction-tuned model pairs.  In addition, the brain alignment of the different LLMs is also compared with performance on benchmarks measuring world knowledge and reasoning, and a strong correlation is found with world knowledge benchmark performance but not with reasoning.  Finally, there is also an experiment in the paper comparing \"behavioral alignment\", which looks at the correlation between per-word perplexity of LLMs and the human per-word reading times, but does not find any significant effect of instruction tuning on this metric.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "We appreciate that the reviewer found our finding—that brain alignment correlates better with world knowledge than reasoning—interesting.\n\n**Response to concerns:**\n\n1. No new methodology\n\nWe respectfully disagree this is valid grounds for rejection. Developing new methods is one of many forms of scientific contribution; our analysis of instruction-tuning and LLM world knowledge contributes new insights to the field. From the COLM review guide: “Our goal as researchers is not only to develop methods and build artifacts but also to understand both the methods we use and natural language.” However, we understand these concerns and will update the paper to better highlight our contributions.\n\n2. Unsurprising that instruction-tuning improves brain alignment\n\nThe reviewer implies that because some methods that improve performance on certain benchmarks also increase brain alignment (though we note the absence of specific references), it is not surprising that instruction tuning would do so as well. We’re not as confident in this generalization, and we note that: \n\n(a) many of the prior works we know of that agree with the reviewer’s generalization are testing next-word prediction, not reasoning and world knowledge (which our paper is the first to study)\n\n(b) prior work has shown that finetuning may result in no change or decreased brain alignment depending on the finetuning task and brain region [1], contradicting the reviewer’s point, and \n\n(c) our work investigates novel questions. We show that instruction-tuning LLMs improves brain alignment and brain alignment correlates with better representations of world knowledge. Our results provide new insights into world knowledge representations by suggesting that mechanisms encoding world knowledge in LLMs also improve brain alignment. This opens new avenues for future research to explore how enhancing world knowledge can align LLMs to human brains. We will update the paper to make these distinctions clearer.\n\n[1] Neural Language Taskonomy: Which NLP Tasks are the most Predictive of fMRI Brain Activity? (Oota et al., 2022)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The finding that brain alignment correlates better with benchmarks measuring world knowledge than it does with benchmarks measuring reasoning is interesting.  Though as the authors note, this finding could be different if the subjects whose brain activity was being measured were performing tasks requiring more reasoning rather than reading, and the correlation with world knowledge benchmarks was still less than with model size, which has a very strong correlation that has also been seen in prior work.", "reasons_to_reject": "This paper does not introduce new methods of analysis, it runs existing implementations of LLM-brain alignment measurement across a different set of input LLMs.\n\nAs the paper notes, much prior work has found that LLM-brain alignment is well correlated with improved LLM benchmark performance, and since instruction tuning is known to substantially improve benchmark performance it seems unsurprising that it would also follow this trend and improve alignment.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eCnbooeWlr", "reviewer": "Reviewer_xjE7", "summary": "This paper aims to study the difference between base and instruction-tuned models in terms of their alignment to both human brain and behaviours. 1) The brain alignment is measured by how well a linear regressor can predict fMRI activity from an LLM's hidden representation. 2) The behavioural alignment is measured by how well per-word perplexity correlates with per-word reading time. The work further studied the how well brain and behavioural alignment correlates with several test set and model properties.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "We appreciate that the reviewer found our paper interesting to read, contributes knowledge to HCI and LLM-human alignment, and the task/setting is relatively novel. \n\n1. Brain alignment of LLMs is weakly positive\n\nThe reviewer is concerned about the weakly positive correlations in our brain alignment results. However, some of our LLMs actually obtain brain alignment close to the “noise ceiling” (Appendix G), suggesting they align nearly all useful signals from fMRI data. For background, fMRI measurements include noise (i.e., fluctuations not due to neurons firing) from factors such as the environment, scanner, and movements of the subject. As a result, a “noise ceiling” is an upper-limit estimate of possible correlation with fMRI activity (usually computed using multiple samples from the same subjects for the same stimuli). There is much work to be done to improve noise ceiling estimates, but we followed prior work in its calculation.\n\n2. The study is not a causal analysis\n\nOur experiments on instruction-tuning’s effect on brain alignment are, in fact, a causal analysis. We directly compare each vanilla LLM (e.g., LLaMA-7B) before and after instruction-tuning (e.g., Alpaca-7B) (Figure 1B), controlling for factors not linked to instruction-tuning.\n\n3. Related work on instruction-tuning [1]\n\nThanks for the relevant pointer! Our study is complementary to theirs. We focus more on brain alignment, using 3 neural and 1 behavioral dataset (while they only touch on brain alignment in the appendix), use human reading times for behavioral alignment (while they use human attention), and study LLaMA, T5, and GPT2 families (while they only explore LLaMA). \n\n4. Behavioral alignment method should not be used, and yields different conclusions from [1]\n\nOur work actually yields a similar conclusion to [1] and [3] for behavioral alignment: improving LLM performance does not enhance alignment with human reading behavior (Figure 3A). [1] study LLM attention and human reading attention while we study LLM surprisal and human reading times, same as [2-3].\nMoreover, human reading time is commonly used in the literature to measure behavior alignment [2-3].\n\n[1] Roles of Scaling and Instruction Tuning in Language Perception: Model vs. Human Attention\n\n[2] The neural architecture of language: Integrative modeling converges on predictive processing\n\n[3] Large GPT-like Models are Bad Babies: A Closer Look at the Relationship between Linguistic Competence and Psycholinguistic Measures", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper is interesting to read and contributes knowledge to the field of HCI and LLM-humanalignment. I find this task/setting to be relatively novel.\n- Technically, the work presented a correlation analysis at two levels:\n  - how base/instruction-tuning affects brain and behaviour alignment.\n  - how test set (knowledge being tested) and model (size, capability) properties then correlate with the brain and behaviour alignment coefficients.", "reasons_to_reject": "Major:\n- From the results in the brain alignment part, I agree that instruction tuning aligns \"better\" than base/pre-trained models, but the Pearson correlation coefficients are still in the \"no correlation\" to \"weak positive\" range. If I did not misunderstood: the average improvement (from base to instruction tuning) seems significant, but it is a % score derived from the difference between already weak correlation coefficients. I think that a quantitative definition of \"alignment\" should be provided or explained, e.g. how good the correlation needs to be in order to qualify for \"alignment\"; otherwise it is unsound to claim the title of this paper.\n- The introduction states that \"to investigate why instruction-tuning increases alignment to human brain activity [...]\" which formulates the problem as a causal analysis, but the presented methodology is a post-hoc correlation study. I fear that the conclusion might lack generalisability.\n\nMinor:\n- Slight overclaim in introduction \"however, there has been no similar study on how instruction-tuning\". [1] has done an analysis from the perspective of LLM and human attention.\n- The methodology used for behavioural alignment is \"the similarity between LLM per-word perplexity and human per-word reading times\", which is an accepted method but the perplexity and reading time should not be used to define behavioural alignment. ~~In fact the features used in [1] yielded different conclusions.~~ (the reviewer made an incorrect statement in the original review.)\n \n[1] Roles of Scaling and Instruction Tuning in Language Perception: Model vs. Human Attention (Gao et al., EMNLP Findings 2023)", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: ## Summary of the paper

The paper studies how instruction tuning, i.e. supervised fine-tuning on instruction-following data, affects
- Brain alignment: predictability of human brain activity from LLM activations (when humans and LLMs read the same text)
- Behavior alignment: correlation between LLM perplexity and human reading times

The authors show a significant positive impact from instruction tuning on the Brain alignment. The authors argue that in some cases the to the results are close to the noise level of the data, i.e. optimal performance. The authors also show that brain alignment strongly correlates with performance on certain benchmarks such as MMLU.

On the behavior alignment task, the authors show negative or no improvement from instruction tuning.

## Strengths

- This is the first paper to the best of my knowledge that studies the impact of instruction tuning on LLM-Brain alignment
- The results are interesting and not obvious, and contribute to the broader picture of Brain-LLM alignment
- The authors study a broad range of models; during the rebuttal, the authors added results for the recent Gemma and Llama2 models, and the original paper already included many models
- The poor correlation of perplexity and reading times is somewhat unexpected

## Weaknesses

- I agree with reviewer uLs2 that studying RLHF as opposed to supervised finetuning on instruction data would be very interesting. The industry standard for LLM post-training is RLHF and understanding the effect of RL would be extremely interesting, especially if it is different from supervised finetuning.
- Reviewers uLs2 and xjE7 raised concerns about the behavior alignment method: e.g. possibly the reading times should be normalized by the total length of the text in characters. The results on behavior alignment are surprising, and it would be interesting to provide further insights here.
- Reviewer hcZZ pointed out that the paper does not develop new methodology, and instead applies existing methodology in a new setting. I am inclined to agree with the authors that this is not a major issue. 
- I think it would be extremely interesting to push the hypothesis on correlation of Brain Alignment and model size to the limit and try models like Llama2-70, Nemotron-4 340B, and maybe Grok-1 314B.

## Conclusion

Overall, I think this is an interesting paper, and it would be a valuable contribution to the conference, despite some limitations of the analysis.

[At least one review was discounted during the decision process due to quality]
Decision: Accept