Roles of Scaling and Instruction Tuning in Language Perception: Model vs. Human Attention
Authors: Changjiang Gao
Shujian Huang
Jixing Li
Jiajun Chen
Venue: EMNLP 2023 Findings (2023)
Keywords: Large Language Models
Attention
Human Resemblance
Instruction Tuning

Abstract: Recent large language models (LLMs) have revealed strong abilities to understand natural language. Since most of them share the same basic structure, i.e. the transformer block, possible contributors to their success in the training process are scaling and instruction tuning. However, how these factors affect the models' language perception is unclear. This work compares the self-attention of several existing LLMs (LLaMA, Alpaca and Vicuna) in different sizes (7B, 13B, 30B, 65B), together with eye saccade, an aspect of human reading attention, to assess the effect of scaling and instruction tuning on language perception. Results show that scaling enhances the human resemblance and improves the effective attention by reducing the trivial pattern reliance, while instruction tuning does not. However, instruction tuning significantly enhances the models' sensitivity to instructions. We also find that current LLMs are consistently closer to non-native than native speakers in attention, suggesting a sub-optimal language perception of all models. Our code and data used in the analysis is available on GitHub.

Roles of Scaling and Instruction Tuning in Language Perception: Model vs. Human Attention
Changjiang Gao' Shujian Huang" Jixing Li² Jiajun Chen'
National Key Laboratory for Novel Software Technology, Nanjing University Department of Linguistics and Translation, City University of Hong Kong gaocjesmail.. nju. edu. [huangsj, chenjj]enju. jedduu.eju.edu.... edu. cn jixingli@cityu.. hk
Abstract
et al., 2020; Schrimpf al., 2021). However, given recent breakthroughs LLMs remains be tested whether the newest LLMs still align with hu- man perception data. Based on the training pipeline of current LLMs, there are two possible sources for the potential change: the scaled model/data size and the aligning process after pretraining, such instruction tuning (Ouyang al., 2022; Wei al., 2022: Sanh et al., 2022). This paper aims pro- vide further understanding of LLMs' mechanism by investigating the roles of these two factors language understanding.
Recent large language models (LLMs) have revealed strong abilities to understand natural language. Since most of them share the same basic structure, i.e. the transformer block, pos- sible contributors to their success the train- ing process are scaling and instruction tuning. However, how these factors affect the mod- els' language perception unclear. This work compares the self-attention of several existing LLMs (LLaMA, Alpaca and Vicuna) in differ- ent sizes (7B, 13B, 30B, 65B), together with eye saccade, an aspect of human reading atten- tion, to assess the effect of scaling and instruc- tion tuning language perception. Results show that scaling enhances the human resem- blance and improves the effective attention by reducing the trivial pattern reliance, while in- struction tuming does not. However, instruction tuning significantly enhances the models' sensi- tivity to instructions. We also find that current LLMs are consistently closer to non non-native than native speakers in attention, suggesting a sub- optimal language perception of all models. Our code and data used the analysis available GitHuub.
In this paper, we analyze self-attention as a chan- nel to the language perception ablility of LLMs, because it is the key mechanism of transformer language models, and it naturally resembles the hu- man attention mechanism in language perceiving (Merkx and Frank, 2021). The attention patterns under investigation includes those from trending open-sourced LLMs with different sizes, i.c., 7B, 13B, 30B, 65B, and different training stages, i.e.. pretrained (LLaMA) and instruction- (Alpaca and Vicuna) models. Given these models are all trained mainly with English data, we also com- pare the attention patterns of LLMs with human saccade (a form of eyc- eye-movement) data, including both native speakers (L1) and non-native learners (L2) of English (Li et al., 2019) to see if the models correlates more with L1 subjects.
1 Introduction
Large language models (LLMs), e.g., GPT-4, Chat- GPT and Vicuna, have shown nearly human-level understanding of text inputs, indicating better human-like language perception compared with their predecessors such as BERT (Devlin et al., 2019) and GPT-2 (Radford al., 2019). How- ever, the mechanism behind such improvement largely unknown. One way to interpret such mech- anism is comparing model computation processes to human data. For example. prior work compared language models (mainly GPT-2) with human neu- roimaging data during language comprehension, suggesting that models with higher human resem- blance also perform better in NLP tasks. (Hasson
Our analysis three-fold. First, we compare the general attention distributions LLMs to mea- sure the impact of these two varying factors on the model ($3.1). Second, we perform linear regres- sions (LRs) between the model self-attention and human saccade matrices to see how human-like they are, and how the resemblance is affected by the two factors ($3.2). Third, analyze the rela- tion between the given attention and some trivial patterns, revealing the difference between different levels language perception abilities ($3.3). The main findings are:
Corresponding author

---

Scaling significantly affects the general atten- tion distribution on plain text, while instruc- tion tuning has limited effect. However, in- struction tuning enhances the model's s sensi- tivity to instructions ($5.1).
LLMs are better at understanding human instruc- tions. However, there little discussion on how the instruction tuning process affects the language perception.
Eye-movement in the human reading process has drawn attention in the computational linguistic field. It is proposed that scanpaths, i.e. tracks of eye-movement, reveals characteristics of the pars- ing processes and relates to working memory dur- ing reading (von der Malsburg and Vasishth, 2013) 3), and the irregularity of scampaths is affected by sen- tence structures and age (von der Malshurg et al., 2015). Such work provides theoretical ground for our analysis on saccade data.
Higher human resemblance is significantly correlated to better language modeling. Scal- ing improves the resemblance by scaling law (Henighan et al., 2020), while instruction tun- ing reduces it. All models have higher resem- blance to L2 rather than to LI, suggesting further room for the improvement in language perception ($5.2).
L2 saccade has higher dependency on trivial attention patterns than L1, indicating a more rigid way of perception. Scaling significantly lowers LLMs' reliance on trivial attention pat- terns, while instruction tuning does not ($5.3).
Joint study of language models and human eye-movement is also receiving growing interest. Prior studies have shown the correlation between language processing of human and artificial neural networks (Kell et al., 2018), especially pretrained language models such as GPT-2 (Caucheteux et al., 2021, 2022: Schrimpf et al., 2021; Lamarre et al., 2022). Especially, Oh and Schuler (2023a,h) ex- plored the fit of model surprisal human read- ing times, noting the effect of models' sequencial memorization and dataset size. Hollenstein and Beinborn (2021) studied the correlation between relative word importance derived from models and humans, and Morger et al. (2022) extended that research to a multilingual setting, as well anal- ysis model attention. Some studies also human attention supervision to assist specific tasks (Sood et al., 2020; Bansal al., 2023), use sequencial models to predict eye-tracking data (Chersoni et al., 2022). Different from them, we use human attention as references for the language perception abilities.
2 Related Work
Self-attention analysis is a common way to in- terpret transformer models. Several studies has shown that it is correlated to linguistic information, such syntactic or semantic structures (Mareček and Rosa, 2018; Raganato and Tiedemann, 2018; Voita al., 2019; Clark al., 2019). There also some attempts and debates explain the models' prediction with attention (Jain and Wallace, 2019; Wiegreffe and Pinter, 2019; Serrano and Smith, 2019; Vashishth et al., 2019). Different from them, we use self-attention patterns as measurement of the language perception abilities of LLMs. Also, some new ways to interpret model self-attention have been proposed, such as Attention Flow (Abnar and Zuidema, 2020), vector norm (Kobayashi et al.. 2020, 2021). 1). ALTI (Ferrando et al., 2022), Value Zeroing (Mohebbi et al., 2023), and logit analysis (Ferrando et al., 2023). These methods can raise the interpretability of model attention w.r.t. linguistic structural information, but are not tested with hu- man eye- trackink data. Also, some of them suffer from huge computational cost. Instead, choose the raw attention scores, as it is produced in the models' original computational processes, and they already correlate the human data, demonstrat- ing their interpretability.
Methods
To show the effect of scaling and instruction tuning different models, we compare the self-attention scores of different LLMs given the same input, by viewing model attention as probability distributions and calculating the general attention divergence based on Jensen-Shannon divergence.
To analyze model self-attention, we take the hu- man saccade as a reference, and design human semblance metric based on linear regression scores.
Instruction tuned LLMSs achieve better perfor- mances in performing tasks (Ouyang et al., 2022; Wei et al., 2022: Sanh et al., 2022). It seems that the
In addition, we select several trivial, context- free attention patterns, and design a trivial pattern reliance metric to help demonstrate the difference

---

between models and human subject groups.
the two corpora within the original sentence spans (re-normalized) to represent con-instruction and in- struction scenarios. Also, as a control group, we at- tached a noise prefix made of 5 randomly sampled English words "Cigarette first steel convenience champion.", and calculate the divergence between prefixed and plain text to account for the effect of meaningless prefix.
3.1 General Attention Divergence
We assesses the mean Jensen-Shannen (J-S) di- vergence between attention from different mod- els on all sentences. For two attention matrices Roword X Xhward the J-S divergence is:
A, B E
Aword Σ
3.2 Human Resemblance
DIs(A,
(
2
3.2.1 Human Resemblance Calculation
1=1
+ DKL (RLL(B:1A4) [AA)]
Given the model self-attention and the human sac- cade data in matrix form for each sentence, we first extract the lower-triangle parts of the matrices (marking right-to-left attendance), because prior studies have found that such eye-movements in reading are related to working memory efficiency (Walczyk and Taylor, 1996) and parsing strategies (von der Malsburg and Vasishth. 2013), and the auto-regressive LLMs only have right-to-left self- attention. Then, we flatten the extracted parts of each sentence attention matrix, and concatenate them to get attention vectors {fiummnn for each subject and { (umoyee)) for head k in model layer j. We then stack { (umoded along the attention heads and get a matrix {V (Vmodde)) for the j-th layer. The LR for model layer and subject is defined as:
where DKL is the Kullback-Leibler (K-L) diver- gence, A₁ and B; are the i-th rows in the two matri- ces. Note the attention matrices are re-normalized keep the sum of every row 1.0. We calcu- late the J-S divergence between heads-averaged attention scores per sentence, and demonstrate the mean values. For models with the same number of layers, the divergence is calculated layerwise; For models with different numbers of layers, we divide their layers into four quarters and calculate the divergence quarter-wise.
To identify high and low divergence, reference value is required. We propose to use the attention divergence between two models that share the same structure, size, and training data, but have small dif- ference in training strategies as the reference. A divergence value higher than the reference suggests relatively large change attention, while one lower than that indicates small (but not necessarily negli- gible) change.
Vit Muddee w + marran
arg min w.
2
This means we take the linear combination of attention scores from all heads in layer to predict the human saccade. Note that the model attention here are not re-normalized, in order to preserve the relative weight w.r.t. the 
 token. The regression scores R?, (ranging from 0 to I, larger value means better regression) are averaged over subjects to rep- resent the human resemblance of layer j. and the highest layer score considered as the regression score of the whole model, R2 R.nodel
Divergence by Scaling and Instruction Tuning Models with different scales or training stages are compared with the above divergence to show the effect of different factors.
Sensitivity to Instructions To evaluate the effect of instruction tuning, we measure given model's sensitivity to instructions by the same general atten- tion divergence metric, but on different input, i.e.. plain input and input with instructions. If the di- vergence between them is significantly higher than reference, we say the model has high sensitivity to instructions.
Because the human saccade pattern varies across individuals, there exists a natural upper bound for human resemblance of models, i.e., the human- human resemblance, or the inter-subject correla- tion. To calculate this correlation within the two language groups, LI and L2, we take the group mean to regress over each individual data indepen- dently, and use the average regression score R2 as the inter-subject correlation for that group. The final human resemblance score is R2 R R22 R ranging from to 100%.
To construct instructed text, attach two pre- fixes plain text sentences: "Please translate this sentence into German:". and "Please paraphrase this sentence:". They are within the reported capac- ity of most LLMs, and require no extra information to follow. Then, we collect model attention on

---

3.2.2 Human Resemblance vs. Next-Token Prediction
of
Croed
A previous study finds a positive correlation be- tween the next-token predication (NTP) perfor- mance and the "Brain Score" of neural networks (Schrimpf et al., 2021), which supports the intu- ition that the more human-like models are, the bet- ter generation they output. To test this finding in our setting, we calculated the per-token prediction loss (the negative log likelihood loss normalized by sequence lengths) of the models employed in this study on the Reading Brain datasct, and calculate the Pearson's s Correlation between their prediction loss and max layerwise human resemblance scores.
12
(a) Individual
(b) Group Mean
Figure 1: Examples of individual and group mean sac- cade of the sentence "Could humans live on Mars some day?".
on the articles. The human subjects are 52 native English speakers (LI) and 56 non-native learners of English (L2). We choose this dataset for the follow- ing reasons: (1) The text is presented to the subjects sentence by sentence instead of page by page, mak- ing in-sentence reading patterns clearer. (2) has synchronous eye-tracking and fMRI data, which allows both behavioral- and neural-level analysis. In this article we mainly focus on the behavioral part, but we also have some preliminary results on the fMRI analysis (See Appendix D).
3.3 Trivial Pattern Reliance
3.3.1 Trivial Pattern Reliance of Models
We select three representative trivial patterns of transformer attention: attending to the first word in the sentence, attending the previous word (Vig and Belinkov, 2019), and self-attending (Clark al., 2019), and analyze the effect of scaling and instruction tuning on models' reliance on them. First, we represent the trivial patterns as binary adjacent matrices, where attending relations are marked with 1, others with 0, and use their flattened lower-triangle parts as the independent variables. The model attention are similarly processed to be the dependent variables. Then, an LR is fit between the two, and the regression scores are collected to represent the trivial pattern reliance of each model layer.
We take the saccade number, i.c. times eye- movements from word to another, instead of sac- cade duration, to represent the human reading atten- tion, because this reduces the effect of other factors such as word length. The saccade data of human subject for given sentence is foword X Rwword matrix. Figure gives examples of individual and group mean saccades.
3.3.2 Trivial Pattern Reliance of Humans
4.2 LLMs
This rescarch compares the models' resemblance to LI and L2 humans because intuitively, LI is better than L2 in language understanding. To test this intuition, we compare their reliance on the three trivial patterns to see whether their attention mode differs. Similar to model attention, the flattened human saccade vectors are used as the dependent variable to fit the LRs with the trivial patterns.
We employ LLaMA (Touvron et al., 2023) and its instruction-tunet versions, Alpaca (Taori et al., 2023) and Vicuna (Chiang al., 2023) in this search. We use them because: (1) The model sizes and different instruction- tuning methods covers the majority of current open-sourced LLMs. (2) These models are being widely used and customized glob- ally. Thus, analysis on them is representative and meaningful.
4 Experiment Settings
4.1 Data
LLaMA is a series of pretrained causal language models with parameter sizes 7B, 13B, 30B and 65B, trained on over IT publicly available text tokens, and reaches state-of-the-at on most LLM benchmarks (Touvron al., 2023). The training corpus is mainly English. We use all sizes LLaMA, which covers the sizes of most current
We use the Reading Brain dataset (Li et al., 2019) for both the text input and the human data. The dataset includes English STEM articles. Each ar- ticle has 29. 6+0.68 sentences of 10.33+0.15 words. It also includes human eye-tree tracking and fMRI data recorded synchronously during self- self- reading

---

open-sourced LLMs. We also use the 774M GPT-2 Large (Radford al., 2019) to represent smaller pretrained models in the scaling analysis.
5
Results
Attention Divergence between 7B and 13B
0.010
Alpaca is fine-tuned from the 7B LLaMA model on 52K English instruction-folloiing demonstra- tions generated by GPT-3 (Brown et al., 2020) us- ing self-instruct (Wang et al., 2023). To analyze the effect of scaling, also trained a 13B version of Alpaca using the official data and training strategy of Alpaca. Our 13B Alpaca model scores 43.9 and 46.0 on the MMLU dataset in zero-shot and one- shot setting respectively, proving the soundness of our fine-tuning, (The corresponding scores are 40.9 and 39.2 for the official Alpaca 7B model.')
Mean 15 Divergence
-+-
Alpaca
Ref
0.005
0.002
0.000
100%
Figure 2: Mean J-S divergence between 7B and 13B model attention in each layer quarter, measured instruction sentences.
Vicuna models fine-tuned from the 7B and 13B LLaMA models 70K user-shared conversa- tions with ChatGPT. The data contains instruction and in-context learning samples in multiple lan- guages, thus the Vicuna models can also be viewed as instruction-tun -unde.
5.1
General Attention Divergence
5.1.1
Scaling Significantly Changes the General Attention
The general attention divergence between models in different sizes is shown in Figure 2. The results shows that LLaMA, Alpaca and Vicuna all reveal divergence significantly higher than the reference values when scaled from 7B 13B. This means scaling causes large change in the overall attention distribution.
4.3 LLMs Attention Collection
To obtain model attention, text is provided to the models sentence by sentence perform next token predict, and we collect the attention tensors in each layer. This resembles the human self-paced task- free reading process, where only one sentence is visible at time to the subjects, and the natural human behavior is to predict the next word (Kell et al., :;; Schrimpf et al., 2021).
5.1.2
Instruction Tuning Has Limited Effect on General Attention
The general attention divergence between pre- trained and instruction tuned models shown Figure 3. It can be observed that only Vieuna 13B brings divergence above the reference values, while all the others don't. The same result can also be observed instruction-preffi suggests that instruction tuning can only bring lim- ited change to the models' general attention distri- bution. This can also be verified by other results in the following parts.
Because models use the SentencePiece tokeniza- tion (Kudo and Richardson, 2018), but the human saccade data is recorded words, the attention matrices need to be reshaped. Following Clark et al. (2019) and Manning et al. (2020), we take the sum over the "to" tokens, and the average over the "from" tokens in split word. Also, the 
 token is dropped as it is not presented to human subjects.
sentences. This
5.1.3
Instruction Tuning Enhances Sensitivity to Instructions
Reference of Divergence We use the attention di- vergence of Vieuna v0 and Vicuna v1.1 13B as the reference values. The two versions share the same training data and strategy, but have minor differ- ence in separators and loss computation". We use the divergence calculated on 13B Vicunas across all divergence analyses, because it has lower mean and variance than the 7B one, making our comparison fairer and stricter.
The divergence between the models' attention ob- tained on the original and instruction-attached sen- tences are shown in Figure 4. There are two major observations. Firstly, LLaMA, Alpaca and Vicuna all show divergence significantly above the refer- ence values between the two types of sentences. especially in the shallower layers, which means they all have sensitivity to instructions. Secondly, while the divergence of LLaMA drops in the deeper layers, the divergence of Alpaca and Vicuna does
Based on this GitHub repository LLM evaluation. "Acoording to notes in the Vicuna GitHub repository.

---

Pretrained (7B)
between Instructed (7B)
ALpA
"de LLaMA
Mean 15 Divergence
0.0125
Micuma-LLaMA
Mean Divergence
0.025
-+-:
0.0100
0.020
Ref.
0.0075
0.0050
0.0000
Layer
Layer 31
Layer 0
Layer 31
Attn, Div. between Pretrained and Tuned Models (138)
Attn. Div. between Plain and Instructed Text (138)
0.04
Alpaco LiaMA
Moain Dierrgence
0.03
Vicuns LiaMA
Ref.
Hean Divergence
0.03
Alpaca
Vicuna
0.02
0.02
0.01
0.00
Layer
Layer
Layer
Layer
Figure General attention divergence between pre- trained and instruction-tuna tuned models in 7B and 13B, measured on non-instruction sentences.
Figure 4: Attention divergence on original and Intruction-attachan text of 7B and 13B models.
this research, giving practical meaning to our human resemblance analysis. Also, based this result, we know the pursuit for higher human semblance consistent with the target of training better LLMs. Similarly, factors negatively impact- ing the human resemblance could also harm the models' language modeling performance, such as instruction tuning brings lower human resemblance as well as higher NTP loss.
not go down, but rather up, suggesting higher sen- sitivity to instructions in the deeper layers. This effect is not observed the poise-prefixed scenario (See Appendix A).
From these two observations, we can conclude that the pretrained LLaMA has already obtained some sensitivity to instructions, i.e. to change its attention mode when it meets instructions, and this sensitivity is further strengthened by instruction tuning, which makes this sensitivity higher in the deeper layers. Because in causal language models like LLaMA, the last few layers are related to gen- eration, this effect of instruction tuning may bring more suitable generation for a certain instruction.
Size
Name
Loss
1.(%))
1.2(%)
774M
GPT-2
0.3264
34,99
40.03
LLaMA
0.2408
53.04
62.44
Alpaca
0.2646
52.51
61.71
Vicuna
0.2593
$1.90
61.19
LLaMA
0.2406
55.66
64,20
13B
Alpaca
0.2634
55.05
64.46
Vicuna
0.2847
54.26
61.31
5.2 Human Resemblance
30B
LLaMA
0.2372
63.16
69.40
5.2.1 Higher Human Resemblance, Lower Prediction Loss
63B
LLaMA
0.2375
64.05
70.07
Table 1: Comparison between NTP loss and the human resemblance scores
The per-token NTP loss and the maximum layer- wise human resemblance scores are compared in Table and Figure 5. One can tell from it that, within the considered range parameter sizes, the human resemblance is negatively correlated with the NTP loss, where the resemblance goes higher nearly linearly with the loss going lower. The Pear- son's correlation -0.875 for Ll (p 0.002) and -0.917 for L2 (p 0.0005), supporting signifi- cant and strong negative linear correlation.
5.2.2 Scaling Enhances Human Resemblance
The layerwise human resemblance of pretrained models in different scales are shown in Figure 6, where the LLaMAs are very different from GPT- 2 Large in two ways. First, the layerwise human resemblance of the LLaMAs are much higher than GPT-2, but the gaps among the LLaMAs are small; Second, human resemblance GPT-2 drop quickly in the deeper layers, while it remains high across all layers the LLaMAs. one can refer from these
This result shows that, the human resemblance we defined positively related to the language modeling performance the LLMs considered in

---

Correlation between Loss and Human Resemblance
70%-
LayerWise Human Resemblence Modes Diferent Scates
L1
Maa Fuman Besambiaanc
600%
Mear lbumen
50
12%
GFTD-774M
LaatA-77
Latte 135
00
100%
Latt--300
Layero
Layer
0.24
0.26
0.28
0.30
0.32
Par-oken
Layonwise Resamblance Diffarent Scales
Figure 5: The correlation between the per-token nega- tive log likelihood loss max layerwise human resemblance of LLMs.
Veern lamen Sansmbiere
CFT2-724M
LaMk6-70
results that LLaMA has undergone a phase change in terms of human-like attention from the level of GPT-2. However, the scaling from 7B to 30B does not cause another phase change.
Model Leyers
Figure 6: Layerwise I.I and human resemblance for different sized pretrained models.
Table and Figure 7 show the max layer- wise human resemblance increases linearly while
the model parameter size increases exponentially, where the Pearson's 's correlation 0.989 for (p < 0.002) and 0.964 for I,2 (p < 0.01). This agrees with the scaling law of LLMs (Henighan
Correlation between Log Scale and Human Resemblance
Max Hummnn Resemblance
et ul., 2020).
50 a0
This result shows that, within the scope of this study, scaling significantly enhances the human semblance. If the scaling law continues take effect, the human resemblance is expected reach 98.800 88.80% for L2 and 88.82% for LI at the scale of 100B. However, because the law is expected to fail after reaching a threshold (Hestness et al., 2017), the relation between human resemblance and model scale is expected to change before reach- ing that scale.
Log Parameter Scale (B)
Figure 7: The change of max layerwise human resem- blance caused by the scaling of model sizes, where the x-axis log scaled.
both sizes, which suggests that instruction tuning causes decline in the human resemblance of atten- tion. However, the decline is minor compared with the increase brought by scaling, which agrees with the limited effect of instruction tuning.
Model
Resemblance (%)
Name GPT-2
Size
L1
L2
774M
34.99
40.03
7B
53.04
62.44
We also conduct relative t-tests to detect signif- icant difference in layerwise human resemblance (p 0.05 n. The numbers of significant lay- ers are shown in Table 4, and the complete lists of layers are in Table 6, Appendix C). The tables show that instruction tuning enhances or reduces human resemblance on about the equal numbers of layers on the 7B models, but reduces human re- semblance to much more layers on the 13B models. Also, the layers with significantly lower resem- blance are distributed across all layers. This result means that, while the total effect of instruction tun- ing is small, the drop human resemblance caused by it significant and widespread.
13B
55.56
64.20
LLaMA
30B
63.16
69.40
65B
64.05
70.07
Table 2: Max layerwise human resemblance of models ranging from 774M to 65B.
5.2.3 Instruction Tuning Harms Human Resemblance
Table shows the max layerwise human resem- blance of the pretrained and instruction tuned mod- els. shows that the instruction tuned Alpaca and Vicuna models have lower max layerwise human resemblance to both LI and L.2 than LLaMA in

---

To sum up, instruction tuning does small but sig- nificant harm to the human resemblance of LLMs, which in turn may bring potential damage to their language perception. This challenges the common assumption that instruction tuning can activate the model's ability to perceive language with human- aligned behaviors. However, this negative effect could be compensated by scaling, for scaling in- creases the human resemblance in larger degree. This provides a novel reason for further scaling.
5.3 Trivial Pattern Reliance
5.3.1 L2 Relies More on Trivial Patterns
The results of LR scores between trivial patterns and LI and 1.2 human saccade is shown in Table 5, where L2's score is higher than Ll's in minimum, maximum and mean values. Independent t-test also supports significant advantage of L2 over L1 on the regression scores (p 5 x 10 $). This means one significant difference between L1 and L2 English speakers is that L.2 people show more trivial and fixed attention mode than LI people while reading, which suggests weaker language understanding ability as those patterns contain no linguistic or factual information.
Model
Resemhlancc(5h (h)
Size
Name
L1
L.2
LLaMA
53.04
62.44
7B
Alpaca
52.51
61.71
Vieuna
51.90
61.19
LLaMA
55.66
64.20
This finding can help us interpret the difference between the models' L1 and L2 human resem- blance. Note that all models considered in this research are trained mainly on English, but show higher attention resemblance to L2 subjects, this result suggests that their language perception is not idcal.
13B
Alpaca
55.05
63,46
Vicuna
54.26
61.31
Table 3: Max layerwise human resemblance scores of pretrained and instruction tuned models. This shows instruction tuning causes small decline in the max layer- wise human resemblance.
Group Min
Max
Mean
SF
0.0048
0.1525
0.0473
0.0037
Model
Higher
Lower
0.0134
0.2165
0.0894
0.0060
Size
Name
L1
L2
L1
L2
7B Alpaca
3
8
4
7
Table 5:
Regression scores of human saccade the trivial patterns, across L1 and L2 subjects. SE stands for standard errors.
Vicuna
9
10
9
Alpaca
4
25
13B
Vicuna
5
16
Table 4: Numbers of Alpaca and Vicuna layers with significantly higher or lower human resemblance com- pared with LLaMA in the same sizes.
5.3.2 Scaling Reduces Trivial Pattern Reliance
The relative difference between 13B and 7B mod- els in trivial pattern reliance is shown in Figure 8, where the decline in the deeper layer quarters is substantial across all models. There is also consistent increase in the second quarter, but the amplitude small, so the total trivial pattern re- liance is reduced. This phenomenon is also ob- served between ILLMMA 13B and 30B (Figure 11 in Appendix B). This means scaling can effectively reduce the trivial pattern reliance, especially in the deeper layers, indicating more contextualized and adaptive language perception.
5.2.4 All Models Show Higher L2 Resemblance
Comparing the L1 and L2 human resemblance of all models (LLaMA, Alpaca and Vieuna, 7B and 13B) in Table and Table 3, an advantage of L2 over LI can be observed steadily. Independent t-test also shows that, all models considered in this research show significantly higher resemblance to L2 than to LI (p- = 0.05/ Prest to avoid false positives). This means the models are closer to non-native English learners than to native English speakers in attention, though they all trained mainly on English data. Furthermore, this trend is not reversed by the scaling or instruction tuning within the scope of this research, which suggests that it is a innate feature of LLMs. We will look further into this phenomenon in the next section.
/.05/Prest
5.3.3 Instruction Tuning Increases Trivial Pattern Reliance
Figure 9 shows the relative difference between pre- trained and instruction tuned models in trivial pat- tern reliance, where the tuned ones show reliance gain in the deeper layers. There is also a small drop in the second quarter, but the total trivial pattern reliance is increasing. Also, total increase here is

---

Reliance (13B 78)
Limitations
Reralive Difference
One key limitation of this study is that we can- not apply our current method to closed-soureed LLMs such as ChatGPT, for their layerwise atten- tion scores are unavailable. With the rapid devel- opment of open LLMs, is hopeful that we can examine these findings on the largest and most ad- vanced LLMs in the near future.
-10%
LLMMA
-00%
Alpaca
-30%
25%
30%
75%
100%
Leyer Quarters
Figure 8: The difference between trivial pattern reliance 7B and 13B model attention.
Besides, though demonstrate the difference in LLMs resemblance LI and L2 people, and partially explained the effect by trivial pattern re- liance, not enough to explain the underlying cause or consequences of this phenomenon. We do have some observations, such as the inter-subject correlation in the LI group (5.1560 + 0.0073) is lower than L2 (0.2463 + 0.0090), and L1 subjects tend to look back to pervious words less frequently than L2 subjects. However, the underlying mecha- nism is still unclear. We hope to dive deeper into this non-native resemblance in our following study. Another limitation of this work comes from the way we obtain model attention. We used the atten- tion scores when predicting the next single token, which makes sense. However, the attention patterns could be different in the following decoding steps. Also, it is possible that combining attention scores across multiple layers can improve the analysis. We will investigate this in our future work.
also smaller in amplitude than the reduction caused by scaling, again supporting the limit effect of in- struction tuning.
Also, one can tell from Figure 8 and Figure that scaling and instruction tuming changes the triv- ial pattern reliance in opposite directions. How- ever, because the effect of instruction tuning is also smaller than that of scaling, as long the model size continues grow, this gain in trivial pattern reliance brought by instruction tuning could be compensated or even covered, offereing another reason for further scaling.
A Trivial Pettern Reliance (Tuned Pretrained)
Delelive Diffrence
Vccna-LLaMA/78)
1.0%
LLaMA/13B)
Vicun-LLaMW/1311
5%9
In addition, the Reading Brain dataset we used this research is monolingual, which does not reflect the language perception process in the cross-lingual setting, which is also important for the application of LLMs. We also hope to examine our findings in the cross-lingual scenario in the future.
0%
55%
75%
50%
Layer Quarters
Figure 9: The difference in trivial pattern reliance of attentions in pretrained and instruction-tuned models.
Ethics Statement
6 Conclusion
The authors declare no competing interests. The human behavioral data we use is publicly available and does not contain personal information of the subjects.
This research evaluates the effects of scaling and in- struction tuning on LLMs' attention. We find that scaling effectively changes the general attention distribution, enhances the human resemblance, and reduces the trivial pattern reliance, while instruc- tion tuning does the opposite, but also increases the models' sensitivity to instructions. Further- more, we find current open-sourced LLMs closer to non-native English speakers in language percep- tion, which contains more trivial attention patterns. To the best of our knowledge, are the first an- alyze the effect of instruction tuning on the human resemblance of model attention. We hope this can inspire future work on analyzing LLMs.
Acknowledgements
We would like to thank the anonymous reviewers for their insightful comments. Shujian Huang is the corresponding author. We thank Yunzhe Lv and Wenhao Zhu for finetuning the Alpaca 13B model. This work supported by National Science Foundation of China (No. 62376116. 62176120). the Liaoning Provincial Research Foundation for Basic Research (No. 2022- 2022-KF-26-0000 KF-26-02).

---

References
the 2022 Conference on Empirical Methods in Nat- ural Language Processing, pages 8698 8698-8714 8714, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Samira Abnar and Willem Zuidema. 2020. Quantity- ing Attention Flow in Transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4190-41997 On-
Javier Ferrando, Gerard Gállego, Ioannis Tsiamas, and Marta R. Costa-jussà. 2023. Explaining How Transformers Use Context to Build Predictions. In Proceedings of the 61st Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 5486-5513, Toronto, Canada. Association for Computational Linguisties.
line. Association for Computational Linguisties.
Aakash Bansal, Bonita Sharif, and Collin McMillan. 2023. Towards modeling human attention from eye movements for neural source code summarization. Proceedings of the ACM on Humaan-Compute Inter- action, 7(ETRA):-19.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Uri Hasson, Samuel A. Nastase, and Ariel Goldstein. 2020. Direct fit to nature: An evolutionary perspec- tive on biological and artificial neural networks. Neu- ron, 105(3):416 434.
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Radford. Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schul- man, Dario Amodei, and Sam McCandlish. 2020. Scaling Laws for Autoregressive Generative Model- ing. ArXiv:2010.1497 [cs].
Charlotte Caucheteux, Alexandre Gramfort, and Jean- Rémi King. 2021. Model-based analysis of brain activity reveals the hierarchy of language in 305 sub- jects. arXiv preprint arXiv:21 10.0008.
Charlotte Caucheteux, Alexandre Gramfort, and Jean- Rémi King- 2022. Deep language algorithms predict semantic comprehension from brain activity. Scien- sific Reports, (2(1)) 16327.
Joel Hestness, Sharan Narang, Newsha Ardalani, Gre- gory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. 2017. Deep learning scaling is predictable, empirically. arXiv preprint arXiv: 1712.00409
Emmanuele Chersoni, Nora Hollenstein, Cassandra Ja- cobs, Yohei Oseki, Laurent Prévot, and Enrico San- tus, editors. 2022. Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics. Association for Computational Linguisties, Dublin, Ireland.
Nora Hollenstein and Lisa Beinborn. 2021. Relative Importance in Sentence Processing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 1Ith International Joint Conference on Natural Language Processing ( (Volume 2: Short Pupers), pages 141-150, Online. Association for Computational Linguisties.
Wei-L.in Chiang, Zhuohan Li, 7i Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang. Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open- source chatbot impressing 4 with 90%* * chatgpt quality.
Sarthak Jain and Byron C. Wallace. 2019, Attention is not Explanation. Proceedings of the 2019 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies. Volume (Long and Short Pa- pers), pages 3543-3556. Minneapolis, Minnesota. Association for Computational Linguisties.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? analysis of BERT's attention. In Pro- ceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276-286, Florence, Italy, Association for Com- putational Linguisties.
Alexander JE Kell, Daniel LK Yamins, Erica N Shook, Sam Haignee,,,,, and Josh H McDermott. 2018. A task-optimized neural network replicates human auditory behavior, predicts brain responses, and reveals a cortical processing hierarchy. Neuron, 98(3):630-66
Jacob Devlin, Ming-Was Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies. Volume I (Long and Short Papers). pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. 2020. Attention is Not Only Weight: Analyzing Transformers with Vector Norms. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7057-7075, Online, Association for Computa- tional Linguistics.
Javier Ferrando, Gerard I. Gállego, and Marta R. Costa- jussà. 2022. Measuring the Mixing of Contextual Information in the Transformer. In Proceedings of

---

Goro Kobayashi. Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. 2021. incorporating Residual and Nor- malization Layers into Analysis of Masked Language Models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4547-4568, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Byung-Dof Oh and William Schuler. 2023a. Transformer Based LM Surprisal Predicts Hu- man Reading Times Best with About Two Billion Training Tokens.
Byung-Doh Oh and William Schuler. 2023b. Why Does Surprisal From Larger Transformer-Basee Language Models Provide a Poorer Fit to Human Reading Times? Trunsactions of the Association for Com- putational Linguistics, 11:336-350.
Taku Kudo and John Richardson. 2018. SentencePiece:
A simple and language independent subword tok- enizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66-71, Brussels, Belgium. Association for Computational Linguistics.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll L. Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. ArXiv:2203.00 02155 [cs].
Mathis Lamarre, Catherine Chen, and Fatma Deniz. 2022. Attention weights accurately predict language representations in the brain. In Findings of the Asso- ciation for Computational Linguistics: EMNLP 2022, pages 4513 4529, Abu Dhabi, United Arab Emirates. Association for Computational Linguisties.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners.
Alessandro Raganato and Jörg Tiedemann. 2018. An analysis of encoder representations in transformer- based machine translation. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 287-297, Brussels, Belgium. Association for Com- putational Linguistics.
Ping Li, Anya Yu, Benjamin Schloss, Chun-Tine Ting Hsu, Lindsey Ma, Chih-Ting Chang, and Marissa Scotto. 2019. The reading brain project: An open science data-sharing initiative. In 26th Annual Meeting of the Cognitive Neuroscience Society.
Christopher D Manning, Kevin Clark, John Hewitt, Ur- vashi Khandelwal, and Omer Levy. 2020. Emer- gent linguistic structure in artificial neural networks trained by self-supervision. Proceedings of the Na- tional Academy of Sciences, I (1744))3004663000
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M. Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, De- bajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica. Sheng Shen, Zheng Xin Yong. Harshit Pandey, Rachel Bawden, Thomas Wang. Trishala Neeraj. Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry. Jason Alan Fries, Ryan Techan, Tali Bers, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. 2022. Mul- titask Prompted Training Enables Zero-Shot Task Generalization. ArXiv:21 10.08207 [cs].
David Mareček and Rudolf Rosa. 2018. Extracting syn- tactic trees from transfomeer encoder self- attentioss In Proceedings of the 2018 EMNLP Workshop Black- boxNLP: Analyzing and Interpreting Neural Net- works for NLP, pages 347-349, Brussels, Belgium. Association for Computational Linguistics.
Danny Merkx and Stefan L. Frank. 2021. Human sen- tence processing: Recurrence or attention? In Pro- ceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pages 12-22, Online. Association for Computational Linguistics.
Martin Schrimpf, Idan Asher Blank, Greta Tuckute, Ca- rina Kauf, Eghbal A Hosseini, Nancy Kanwisher, Joshua B Tenenbaum, and Evelina Fedorenko. 2021. The neural architecture of language: Integrative modeling converges on predictive processing. Pro- ceedings of the National Academy of Sciences, (18845))21056661118
Hosein Mohebbi, Willem Zuidema, Grzegorz Chrupalaa, and Afra Alishahi. 2023. Quantifying Context Mix- ing in Transformers. In Proceedings of the 17th Con- ference of the European Chapter of the Association for Computational Linguistics, pages 3378-3400, Dubrovnik, Croatia. Association for Computational Linguistics.
Sofia Serramo and Noah A. Smith. 2019. Is attention in- terpretable? In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages 2931 1-951, Florence, Italy. Association for Computational Linguistics.
Felix Morger, Stephanie Brandl, Lisa Beinbor, and Nora Hollenstein. 2022. A Cross-lingual Compari- son of Human and Model Relative Word Importance. In Proceedings of the 2022 CLASP Conference on jmbbbddidd, pages 11-23, Gothenburg. Sweden. Association for Computational Linguistics.
Ekta Sood, Simon Tamnert, Philipp Müller, and Andreas Bulling. 2020. Improving natural language process- ing tasks with human gaze- guided neural attention.

---

Advances Neural Information Processing Systems, 0616327-634..
on Natural Language Processing (EMNLP-IJCNLP), pages 11-20, Hong Kong, China. Association for Computational Linguistics.
Rohan Taori, Ishaan Guirajani, Tianyi Zhang. Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instructioon-ollooinng Ilama model. https:// gi thubt cwm/ httpl//aaatttttdaa.aa//
A Attention divergence on plain and noise-prefixed text
As shown Figure 10, the divergence between plain and noise-prefixed text of Alpaca and Vicuna is not significantly higher than LLaMA in most of the layers, which further supports our findings on the higher instruction sensitivity of instruction- tuned models.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Aune Lachaux. Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023, Llama: Open and efficient foundation language models.
Shikhar Vashishth, Shyam Upadhyay, Gaurav Singh
Plain Noise Prefixed (7B)
Tomar, and Manaal Faruqui. 2019. Attention in- terpretability across nip tasks. arXiv preprint arXiv:1909.1122 11218.
0.030
LiaMA
Mean js Divergence
- Alpaca
0.020
Ref.
Jesse Vig and Yonatan Belinkov. 2019. Analyzing the structure of attention in transformer language model. Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 63-76, Florence, Italy. As- sociation for Computational Linguistics.
0.015
0.010
0.005
Layer 0
Layer 31
Attn. Div. between Plain and Noise-Prefixed Text (138)
-.fo. LLaMA
Elena Voita. David Talbot. Fedor Moiseev, Rico Sen- nrich, and Ivan Titov. 2019. Analyzing multi-head self-attention: Specialized heads do the heavy lift- ing. the rest can be pruned. In Proceedings the 57th Annual Meeting of the Association for Computa- tional Linguistics, pages 5797-5808. Florence, Italy. Association for Computational Linguisties.
Means J5 Divergence
0.03
Appcce
Vicuna
0.02
0.01
0.00
Layer
Layer
Titus von der Malsburg, Reinhold Kliegl, and Shravan Vasishth. 2015. Determinants of Scanpath Regularity in Reading. Cognitive Science, 39(7):1775-1703
Figure 10: Attention divergence on the original and noise-attached text of 7B and 13B models.
Titus der Malsburg and Shravan Vasishth. 2013. Scanpaths reveal syntactic underspecification and re- analysis strategies. Language and Cognitive Pro- cesses, 28(10):1545-1778
B Layerwise trivial pattern reliance
Figure 11 shows the layerwise trivial pattern liance of 7B and 13B models, well as the results for LLaMA 30B. The reliance keeps decreasing as the model size scales up, especially in the deeper layers.
Jeffrey J. Walczyk and Robert W. Taylor. 1996. How do the efficiencies of reading subcomponents relate to looking back in text? Journal of Educational Psychology, 88(3):537-545.
Yizhong Wang. Yeganch Kordi, Swaroop Mishra, Al- isa Liu, Noah Smith, Daniel Khashabi, and Han- naneh Hajishirzi. 2023. Self-Instruct: Aligning Language Models with Self-Generated Instructions. ArXiv:2212.1055 [cs].
C Layers with significant changes in human resemblance after instruction tuning
Jason Wei. Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. 2022. Finetuned language mod- els are zero-shot learners. In International Confer- ence on Learning Representations.
Table 6 shows the list of layers in 7B Alpaca and Vi- cuna with significant change in human resemblance after instruction tuning compared with LLaMA, and Table 7 shows the list in 13B.
Sarah Wiegreffe and Yuval Pinter. 2019. Attention is not
D Preliminary IMRI results
not explanation. In Proceedings of the 2019 Confer-
ence on Empirical Methods Natural Language Pro- cessing and the 9th International Joint Conference
Some preliminary analysis on the fMRI data in the Reading Brain dataset is done, and the results com-

---

Relation
Group
Alpaca 2.
Vicuna
RB Trivial Pattem Predictability of Models
13,
LLaMA
Regresston Scores
0.3
Alpece
28,
0.2-
Higher
2,5,
3, 13, 15,
12, 15.
18, 21, 23,
1.2
21, 27
26, 28, 29,
0.1-
30
4. 10, 23, 31
1,
0.0
Layer
Leyer 31
L1
5,9, 11,
Model Layers
20, 31
Lower
RB Trivial Pattem Predictability of 136 Models
0, T0, 14,
0, 1,
L2
19, 23,
5, 11,
Degraxsoon Scones
Alpaca
28, 31
14. 20, 31
Vicune
0.2-
Table 6: Layers of Alpaca and Vicuna with significantly higher or lower human resemblance compared with LLaMA in 7B.
0.1
Layyrr
Model Leyers
Layer
Relation
Group
Alpaca
Vicuna
16, 32, 33, 36
4, 16, 27,
Trivial Pottern Predictability 30B Models
33. 36
LLMA
Higher
1, 16, 19, 27, 32, 33,
4, 16, 19,
Regrosion Scones
0.3
L2
23, 27. 33,
36
36, 37
0, 2,
0, 1,
5,
5, 6.
0.3
9. 14.
10. 14. 17.
L1
18, 24. 25.
18, 24, 30.
0.0
28, 30, 35,
34,
Model Layers
39
0, 3,
0,
Regreszion Score Differnces
RB Trivial Pattern Predictability Differences (300-130)
Lower
5, 6,6,
6,
0.04
9, 10,
9,
11. 12, 14.
12, 14. 17.
L2
15. 17, 18
20,
0.00
24, 25.
30
28, 29, 30.
-0.02
31, 35, 38.
0.06
39
25%
50%
75%
100%
Layer Quaertilss
Table 7: Layers of Alpaca and Vicuna with significantly higher or lower human resemblance compared with LLaMA 13B.
Figure 11: Lyerwise trivial pattern reliance of 7B and 13B models. "RB" stands for "Reading Brain".
ing enhances human fMRI resemblance, instruction tuning does not (Table 8); (4) Resemblance to L2 fMRI is consistantly higher than that to Ll.
ply with our findings on saccade data. We investi- gate several linguistics-related brain regions, and find that the highest correlation of fMRI signals to model attention are from the anterior left temporal gyrus (Figure 12). The fMRI data from this region is pre-processed into matrix forms saccade data. Figure 14 gives examples of individual and group mean IMRI matrices.
The findings are: (1) The models' NTP loss sig- nificantly (p 0.02 for both LI and I.2) and strongly negatively (r -0.777 for L1, r -0.79 for L2) correlates with fMRI resemblance (Figure 13); (2) The models' highest layerwise fMRT resemblance significantly (r > 0.99, p < 0.001 for both LI and L2) correlates with the model scale in logarithm (Figure 15, 16); (3) While seal-

---

Layeror Resuemblance Dilarrnt Stales
Figure 12: An illustration of the anterior left temporal gyrus in the human brain.
Vinee humee fasemberr
LaMM-6555
Correlation belween NTP Loss and Human Resemblance
Layers
Max Humsnn Reseentlance
Layerwise Human Resemblence Modes Diferent Scates
20
25%
15
20%
10
Meann Htmmen
GFTD-774M
Luatte-re
0.24
0.26
PerToken Loss
0.28
0.30
0.32
LuattA 405
LLattA-300
Lattte 136
Layerg
Figure 13: The correlation between per-token nega- tive log likelihood loss and max layerwise fMRI resem- blance of the LLMs.
Figure 15: Layerwise LI and L2 human fMRI resem- blance for different sized pretrained models.
Model
Resemblance %)
Size
Name
LLaMA
11.02
15.45
Alpaca
10.99
15.61
Vieuna
11.35
15.44
LLaMA
14.69
20.19
13B
Alpaca
14.67
19.44
Vicuna
14.02
18.57
Table Max layerwise human fMRI resemblance scores of pretrained and instruction tuned models. This
shows instruction tuning causes small decline in the max layerwise human resemblance.
Correlation between Log mblance
Max Human ressenblence
20
LIANN E-EALL PARIEE
428
Log
40
Figure 16: The change max layerwise human fMRI resemblance caused by scaling. The x-axis is log scaled.
(a) Individual
Group Mean
Figure 14: Examples of individual and group mean MMR signals of the sentence "Could humans live on Mars some day?".


Reviews:
[{"review_id": "kJfz4TaAew", "reviewer": "Reviewer_pjsV", "summary": "", "questions": "* The tokens in the instruction influence the attention over the tokens of interest. When you report results in Figure 4, is it based on renormalising the instruction+plain text attentions over only the tokens in the plain text?\n* What are the attentions like between plain text and noise + plain text? Would this say anything about how the extra pre-pended instruction text impacts attentions?\n* Also on Fig4, the divergence patterns in Figure4 are interesting, in that I have no good hypothesis as to why the max difference is about 1/3 into the model. \n* Each result is based on one model only. Do you think retraining the model (same size, same data, same curriculum) would result in pretty much the exact same attentions? \n* without having read the Clark and Manning papers (line 312), is there any other good options for this operation of squishing the token level attention matrix down to same dimensions as the word level one? Or is this used procedure the only real option?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the positive comments. Below are our answers to the questions and suggestions.\n\n## Answers to the questions:\n\n> - The tokens in the instruction influence the attention over the tokens of interest. When you report results in Figure 4, is it based on renormalising the instruction+plain text attentions over only the tokens in the plain text?\n\nThank you for raising this question. The J-S divergence reported in Figure 4 is based on renormalising the instruction+plain text attentions over only the tokens in the plain text. We will clarify this in the revised version of the manuscript.\n\n> - What are the attentions like between plain text and noise + plain text? Would this say anything about how the extra pre-pended instruction text impacts attentions?\n\nThank you for the constructive suggestion. We have now tested this noise+plain text scenarios on the 7B models, and found that the divergence between plain and noise+plain text (the noise consists of 5 randomly generated English words) of Alpaca and Vicuna is not significantly higher than LLaMA in most of the layers, which further supports our findings on the higher instruction sensitivity of instruction-tuned models. We will add this new result to the revised version.\n\n> - Also on Fig4, the divergence patterns in Figure4 are interesting, in that I have no good hypothesis as to why the max difference is about 1/3 into the model.\n\nWe agree that this is an intriguing phenomenon and needs further explanation. Since the first 1/3 of all layers are relatively shallow, we think that higher difference in these layers may indicate that, the instructions will alter the model perception at lower levels. The difference fades in the pre-trained-only model (LLaMA) in the deeper layers, but retains in the instruction tuned models (Alpaca, Vicuna). Further experiments will be needed to confirm our hypotheses.\n\n> - Each result is based on one model only. Do you think retraining the model (same size, same data, same curriculum) would result in pretty much the exact same attentions?\n\nAlthough the initial states,  random seed, etc., will affect the attention patterns of the LLMs to some extent, we do not think this will make too much difference to the result of the analyses. As pointed out by prior studies, some of the attention heads are indispensable for the correct prediction of transformer models (Viota, et al., 2019), so the attention patterns carried by these heads will be replicated in a successfully re-pretrained LM, in some of the layers and attention heads. And since our human resemblance calculation is based on regression over all attention heads and maximum over all layers, we expect the shared attention patterns to be located, and the human resemblance score to stay approximately the same. \n\n> - without having read the Clark and Manning papers (line 312), is there any other good options for this operation of squishing the token level attention matrix down to same dimensions as the word level one? Or is this used procedure the only real option?\n\nThank you for pointing out this issue. There could be other ways to do this, such as preserving only the first token in each word (which loses information in other sub-word tokens); or aggregating the sub-word tokens based on their frequency or structural importance (which brings more computational complexity). Instead, we follow Clark and Manning’s method because we think it is reasonable. Viewing words as groups of tokens, the sum over \"to\" tokens (columns in the attention matrix)  and the average over \"from\" tokens (rows in the attention matrix) means we aggregate the attention scores between the context and all tokens in that group, while keeping the sum of attention scores in each row as 1. \n\n## Presentation improvements: \n\n> - Can you give equivalent numbers on the 7B model at line 289/290? Saves reader digging those up and would better justify your point that the 13B model you trained is sound.\n\nThank you for the suggestion. The MMLU score for the Stanford Alpaca 7B model is 40.93 for zero-shot and 39.18 for one-shot. (The zero-shot score is higher than the one-shot score, which could be due to longer context, and this is not observed in LLaMA-7B or Alpaca-13B.)  We will add this information in our revised version.\n\n## Response to the reproducibility score\n\nWe will make parameters and other settings clearer in the next version. All the model weights, codes and intermediate results will also be publicly available on GitHub after acceptance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "It reports an interesting finding, namely that the self-attention distribution of transformer language models is not improved by instruction tuning, is improved by scaling, and most interestingly is closer to non-native speaker attention distributions than native speakers. These results are surprising  since instruction tuning results in (certain types) of improved performance and generalisation, whilst the greater similarity to non-native speakers hints at being able to focus on the self-attention mechanism to improve models further.\nSeveral experiments reported to provide evidence for the claims made. These are done on public data and with public models. \nThis is also an important research area, as we have made great steps recently in model ability, but understanding how these models perform so well is less advanced. ", "reasons_to_reject": "No significant reason. ", "questions_to_authors": "* The tokens in the instruction influence the attention over the tokens of interest. When you report results in Figure 4, is it based on renormalising the instruction+plain text attentions over only the tokens in the plain text?\n* What are the attentions like between plain text and noise + plain text? Would this say anything about how the extra pre-pended instruction text impacts attentions?\n* Also on Fig4, the divergence patterns in Figure4 are interesting, in that I have no good hypothesis as to why the max difference is about 1/3 into the model. \n* Each result is based on one model only. Do you think retraining the model (same size, same data, same curriculum) would result in pretty much the exact same attentions? \n* without having read the Clark and Manning papers (line 312), is there any other good options for this operation of squishing the token level attention matrix down to same dimensions as the word level one? Or is this used procedure the only real option?", "paper_topic_and_main_contributions": "The paper investigates how scaling and instruction turning of LMs (mainly LLMs) impacts the self-attention distributions. It compares these to human attention based on a publicly available dataset built from recording human eye tracking whilst reading sentences. \nThe main conclusions are interesting, namely that scaling up LMs results in producing attention distributions closer to the human ones, and that instruction tuning the models actually reduces this similarity. \nThe paper is well written, contains a good amount of evidence for the claims made.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "* Can you give equivalent numbers on the 7B model at line 289/290? Saves reader digging those up and would better justify your point that the 13B model you trained is sound. "}, "extra_text": {}}, {"review_id": "Vy82O4hW5n", "reviewer": "Reviewer_dzum", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the very useful and constructive comments. Below are our answers to the issues and suggestions.\n\n## Replies to the reasons for rejection\n\n> - I think the paper is too atheoretical. I do not see a clear motivation for human evaluation -- it is sort of presented as-is. I do not think that this human evaluation helps us to understand how these models differ from each other than any other dataset (that is, not this dataset of reading times, or any dataset of reading times) would do.\n\nWe agree that our motivation may not be clearly conveyed in the manuscript, and we will make it clearer in the revised version. Our motivation for using human behavioral data for model evaluation stems from prior work that compares language models with human neuroimaging data during language comprehension (e.g., Hasson et al., 2020; Schrimpf et al., 2021). These studies suggest that models with higher human resemblance also perform better in NLP tasks. However, given recent breakthroughs in LLMs such as ChatGPT and LLaMA, it remains to be tested whether these newest models, with their larger scale and extra fine-tuning techniques, still align with human perception data.\n\nOur human evaluation provides new insights into the effects of scaling and instruction tuning, revealing interesting results (e.g., while instruction tuning is widely used to improve the overall performance of LLMs, it may not enhance human resemblance, whereas further scaling does). These insights cannot be obtained solely through NLP task scores. Additionally, our methods can be applied not only to the Reading Brain dataset but also to other datasets that capture human saccade behavior. We chose the Reading Brain dataset because it simultaneously records eye-tracking and fMRI data. We already have some preliminary results on the fMRI part and plan to further investigate it in the future.\n\n> - I do think that looking at eye-tracking saccade co-occurrences is interesting but this approach has been done in the literature before -- scanpath work should be cited to motivate the approach since it is not novel (though it is an interesting way of capturing something)\n\nThank you for pointing out the inspiring work on Scanpath. These studies reveal the reading characteristics of different groups of people using the trajectories of eye-movement, which we think will improve our theoretical basis. We will add these references to the revised version. Also, we are not stating that we are the first to analyze the human saccade data. Instead, the novelty of our work mainly lies in the comparison between saccade data and model self-attention, which is especially affected by scaling and instruction-tuning.\n\n> - I would not refer to these saccade data as \"human attention\" -- I would steer clear of calling *anything* *human attention* without very carefully situating the analyses in over a century of psychophysical literature and being extremely clear about what cognitive processes are going on. The model \"attention\" is also not \"attention\" in the same way we talk about it colloquially -- something I suspect the authors know.\n\nWe recognize that the term \"human attention\" may be too vague, and we will substitute it with more suitable terms such as \"human saccade patterns\" in the revised version. We are aware that the self-attention weights in transformer models are based on query-key-value matrix multiplications and softmax calculations, and are not the same as the \"attention\" in colloquial usage. However, from a higher conceptual level, the self-attention mechanism represents the reliance on contextual information, which also included in the human reading behavior. We believe this provides the ground for our comparison analyses.\n\n> - I find the focus on L1/L2 also perplexing. The authors should be extremely clear about what they think differentiates the L1/L2 data and why this is relevant to LLMs. As it is, I think that the authors frame L2 readers as having a kind of \"deficit\" in processing that is not necessarily appropriate. Rather, L2 reading may simply engage different kinds of cognitive skills that L1 readers may use less or not at all.\n\nWe agree that this is a concern. However, we have stated in our paper that the LLMs employed in this research are all mainly trained (both pre-trained and instruction-tuned) on English data, so one can expect that they are closer to native speakers of English. However, our results showed that the models are actually closer to the L2 speakers. This is why we consider the models to be “sub-optimal”. We have never referred to L2 speakers as \"deficit\" readers. We merely stated that their saccade patterns are different from those of the L1 speakers, e.g. in trivial-pattern reliance. \n\n> - The authors repeatedly state that they have \"proved\" things throughout the manuscript and that is simply not how scientific inquiry goes.\n\nThank you for pointing this out. We will substitute the use of \"prove\" with more appropriate words such as \"show\" or \"demonstrate\" in the revised version.\n\n> - Despite the novelty of the evaluation dataset, the analyses are based on only ~1500 words from ~150 sentences. It is extremely challenging to get enough data to overcome the sparsity issue for human eye movements as well as for model outputs -- what can we infer from 150 sentences? It would be nice to see this done with other eye-tracking corpora (e.g., Dundee, Provo, etc.) and I see no particular reason why the authors would only work with the dataset they used here.\n\nThank you for the comment. We agree that the corpus size in the Reading Brain dataset is relatively small compared to other commonly used eye-tracking datasets such as Dundee. However, it is comparable to those used in previous work (Schrimpf et al., 2021) on comparing language models and human data such as fMRI (Pereira et al., 2018; Blank et al., 2014) and ECoG (Fedorenko et al., 2016). Those datasets also contain only about a hundred sentences. We use the Reading Brain dataset because it has simultaneously recorded eye-tracking and fMRI data. We already have some preliminary results on the fMRI part, and plan to work on it further in the future. We will also include supplementary analysis based on larger eye-tracking datasets to further support our findings in the revised version.\n\n> - The authors should be aware of other work that has looked at training dataset size, e.g., Oh and Schuler (2023), both the TACL proceedings and the arxiv paper\n\nWe thank the reviewer for pointing out these two exciting papers. These studies give insights about how and why the surprisal derived from larger LMs is less correlated with the human reading time. Although the focus of our paper is model attention (an internal state) instead of surprisal (a reflection of the final prediction), we agree that these two papers are related to the analysis of human resemblance and scaling, and we will add these references in our revised paper.\n\n> - The authors should ideally also relate the work to some understanding of human language processing. I don't know what the connection is but using human psycholinguistic data solely for evaluation is really limiting given the massive industry that is correlational studies to human reading dynamics and LLMs.\n\nThank you for the helpful suggestion. We believe that comparing human look-back saccades with model attention is indeed based on the reading dynamics behind them. For example, prior studies have found that look-backs in reading are related to working memory efficiency (Walczyk & Taylor, 1996), and that scanpaths reflect parsing strategies (von der Malsburg & Vasishth, 2013). We will make this link clearer in the revised version. Additionally, beyond correlational studies, our long-term goal is to extract and compare the reading mechanisms and dynamics of humans and machines at a higher level, and try to improve machine reading by incorporating human-like dynamics.\n\n## Response to the reproducibility score\n\nWe do not understand why the reviewer thinks our results cannot be replicated. We will make parameters and other settings clearer in the next version. All the model weights, codes and intermediate results will also be publicly available on GitHub after acceptance. We would also be very happy to invite the reviewer to check whatever results that they think cannot be replicated.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper uses an uncommon dataset (the Reading Brain data) for evaluation\n* The paper assesses new, cutting edge models that are available in an open capacity (i.e., not closed source)\n* The experiments are extensive and seem appropriate for the authors' research questions", "reasons_to_reject": "* I think the paper is too atheoretical. I do not see a clear motivation for human evaluation -- it is sort of presented as-is. I do not think that this human evaluation helps us to understand how these models differ from each other than any other dataset (that is, not this dataset of reading times, or any dataset of reading times) would do.\n* I do think that looking at eyetracking saccade co-occurrences is interesting but this approach has been done in the literature before -- scanpath work should be cited to motivate the approach since it is not novel (though it is an interesting way of capturing something)\n* I would not refer to these saccade data as \"human attention\" -- I would steer clear of calling *anything* _human attention_ without very carefully situating the analyses in over a century of psychophysical literature and being extremely clear about what cognitive processes are going on. The model \"attention\" is also not \"attention\" in the same way we talk about it colloquially -- something I suspect the authors know.\n* I find the focus on L1/L2 also perplexing. The authors should be extremely clear about what they think differentiates the L1/L2 data and why this is relevant to LLMs. As it is, I think that the authors frame L2 readers as having a kind of \"deficit\" in processing that is not necessarily appropriate. Rather, L2 reading may simply engage different kinds of cognitive skills that L1 readers may use less or not at all.\n* The authors repeatedly state that they have \"proved\" things throughout the manuscript and that is simply not how scientific inquiry goes.\n* Despite the novelty of the evaluation dataset, the analyses are based on only ~1500 words from ~150 sentences. It is extremely challenging to get enough data to overcome the sparsity issue for human eye movements as well as for model outputs -- what can we infer from 150 sentences? It would be nice to see this done with other eyetracking corpora (e.g., Dundee, Provo, etc.) and I see no particular reason why the authors would only work with the dataset they used here.\n* The authors should be aware of other work that has looked at training dataset size, e.g., Oh and Schuler (2023), both the TACL proceedings and the arxiv paper\n* The authors should ideally also relate the work to some understanding of human language processing. I don't know what the connection is but using human psycholinguistic data solely for evaluation is really limiting given the massive industry that is correlational studies to human reading dynamics and LLMs.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper tests the relative importance of model size (\"scaling\") and model objective (instruction-tuned or not) in the fit of attention patterns in autoregressive (next-word prediction) large language models to \"human attention.\" This is quantified by comparing the self-attention patterns in the models to co-occurrences between saccades in human eyetracking data for both first language and second language English sentences. The authors compare several models (Alpaca, LLAMA, and Vicuna) at different model sizes and analyze according to model layer depth as well. Generally speaking, they find that size is more important for matching human attention patterns than instruction training, that the models seem to resemble L2 readers more than L1 readers, and that instruction training weakens the resemblance to human attention patterns. They also present a baseline for degree of similarity that is meant to be trivial.\n\nETA: I have read the author response and appreciate the time the authors put in to making a response to my concerns. I will keep my score the same.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "@article{von2013scanpaths,\n  title={Scanpaths reveal syntactic underspecification and reanalysis strategies},\n  author={Von der Malsburg, Titus and Vasishth, Shravan},\n  journal={Language and Cognitive Processes},\n  volume={28},\n  number={10},\n  pages={1545--1578},\n  year={2013},\n  publisher={Taylor \\& Francis}\n}\n\n@article{von2015determinants,\n  title={Determinants of scanpath regularity in reading},\n  author={von der Malsburg, Titus and Kliegl, Reinhold and Vasishth, Shravan},\n  journal={Cognitive science},\n  volume={39},\n  number={7},\n  pages={1675--1703},\n  year={2015},\n  publisher={Wiley Online Library}\n}\n\n@article{oh2023does,\n  title={Why does surprisal from larger transformer-based language models provide a poorer fit to human reading times?},\n  author={Oh, Byung-Doh and Schuler, William},\n  journal={Transactions of the Association for Computational Linguistics},\n  volume={11},\n  pages={336--350},\n  year={2023},\n  publisher={MIT Press}\n}\n\n@misc{oh2023transformerbased,\n      title={Transformer-Based LM Surprisal Predicts Human Reading Times Best with About Two Billion Training Tokens}, \n      author={Byung-Doh Oh and William Schuler},\n      year={2023},\n      eprint={2304.11389},\n      archivePrefix={arXiv},\n      primaryClass={cs.CL}\n}"}}, {"review_id": "iE2UwN4npZ", "reviewer": "Reviewer_GxRX", "summary": "", "questions": "1. Regarding the experiment described at the end of Section 3.1: how do you account for the fact that the attention distribution will be different because it also flows to the tokens in the instruction itself?\n2. Did you consider aggregating LM attention patterns across multiple layers? For LMs the attention is distributed across layers, so it is not completely 'fair' to compare an individual layer's attention to human attention as a whole.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the very detailed and constructive comments. Below are our answers to the issues, questions and suggestions.\n\n## Replies to the reasons for rejection \n\n> - The paper does not seem to be aware of highly related work on the relation between human 'attention' (i.e. eye-tracking data) and LM 'attention'. In earlier work it has been shown that attribution methods yield a better fit to human attention than raw attention (Beinborn & Hollenstein, 2021), and the paper would benefit from engaging with this literature more (see below for references).\n\nThank you very much for the references. The papers are indeed highly related to our work, and we will add these citations in the revised manuscript. Attribution methods are good for explaining relative word importance (Beinborn & Hollenstein, 2021), but they are designed for showing the effect of the input on the output. Instead, we use attention analysis in this paper because the attention scores can show the interaction between the input tokens, resembling the form of human saccade matrices. Indeed, our results show that raw attention scores of recent LLMs such as LLaMA already correlate with human saccade patterns, suggesting that raw attention scores of large transformer models also contain useful information for interpretation as attribution methods do.\n\n> - The paper also does not seem to be aware of a recent line of work that devises better methodologies of interpreting LM attention (see below for references).\n\nThe papers listed by the reviewers include several new and better methods (e.g., Attention Flow, ALTI, etc.) to interpret attention from transformers other than using the raw attention scores. We will add citations to them in the revised version. However, these attention interpretation methods also have some limitations. For example, ALTI are mainly designed for encoders, and Attention Flow averages over all attention heads to reduce calculation, which loses head-wise information. Instead, we think that raw attention already contains some basic patterns in language perception and is good enough for our analysis. In this regard, we have repeated the divergence analyses using Attention Rollout, and the results are consistent with our findings.\n\n> - The paper is too vague about \"human attention\" and would benefit from engaging more with literature from the cognitive science about attention in reading comprehension.\n\nWe fully agree and will include more literature from cognitive science about human attention in reading comprehension, such as Scanpath (mentioned by reviewer #dzum), reading times and working memory for the background of our research.\n\n> - The paper would benefit from a better restructuring/rewriting, there are a lot of unnecessary subsubsections and 1-sentence paragraphs.\n\nThank you for the very helpful suggestion. We will incorporate it in our revised paper.\n\n> - The paper would benefit from a more detailed analysis of the difference between L1 and L2 speakers, that reaches further than a stronger 'trivial pattern reliance' of L2 speakers.\n\nThank you for your useful suggestion. We provide below some more analyses of the two language groups. The correlation between L1-average and L2-average saccade is 0.80, which suggests that they are different to some degree. The Inter-Subject Correlation is 0.1560(+-0.0073) for L1 and 0.2463(+-0.0090) for L2, suggesting that L1 individuals have more diverse reading patterns, which may also partly explain why the model fits better to L2.  Also, we observed different behavioral patterns of L1 and L2 subjects, such as L1 subjects tend to look back on previous words less than L2, but we are still searching for a good way to quantify them. We will add this information to the revised manuscript.\n\n## Answers to the questions:\n\n> 1. Regarding the experiment described at the end of Section 3.1: how do you account for the fact that the attention distribution will be different because it also flows to the tokens in the instruction itself?\n\nThank you for raising this issue. We agree that the prefix itself would affect the total attention distribution, and reviewer #pjsV also mentioned this issue and suggested comparing plain text and noise+plain text to account for the effect of extra prefix tokens. We have conducted the suggested analysis and the result shows that the divergence between plain and noise+plain text of Alpaca and Vicuna is not significantly higher than LLaMA in most of the layers, which further supports our findings. We will add this new result to the revised version.\n\n> 1. Did you consider aggregating LM attention patterns across multiple layers? For LMs the attention is distributed across layers, so it is not completely 'fair' to compare an individual layer's attention to human attention as a whole.\n\nThank you for the insightful suggestion.  We agree that the model attention is distributed across layers. Our analysis of the layer-wise attention is based on the model mechanism that the representations in higher transformer layers contain more contextual information induced by previous layers, which would reflect the process of contextual language perception. However, we will add the results on attention across layers using methods such as Attention Rollout and Attention Flow to tackle this issue.\n\n## Replies to the suggestions\n\n> - Explain what types of \"human attention\" you use for your analysis, right now in Section 3 you have not explained this at all yet, and only in Section 4.2 this is briefly touched upon. This is important though, as you place a lot of importance on this \"human attention\" in your analysis, so it should be clear to a reader what exactly you mean by it.\n\nThank you for the useful suggestion. The \"human attention\" we refer to in this work is the human saccade patterns, which reflect the participation of contextual words when the subject is perceiving the current word. We will make this clearer in the introduction part of the revised version. \n\n> - Refrain from using the term \"prove\" when there is no theoretical proof involved, you merely \"demonstrate\" something here.\n\nThank you for pointing out this issue. We will substitute the use of \"prove\" with more appropriate words such as \"show\" or \"demonstrate\" in the revised version. \n\n## Style / Typos\n\n> - The sentence around L127 (On the other hand...) is hard to understand.\n\nThank you very much for pointing this out. The correct sentence should be \"On the other hand, we select several trivial attention patterns which correspond to meaningless structures as reference for bad attention, and design the metric of trivial pattern reliance for any given attention.\" We will correct this sentence in our revised version.\n\n## Response to the reproducibility score \n\nWe will make parameters and other settings clearer in the next version. All the model weights, codes and intermediate results will also be publicly available on GitHub after acceptance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- A lot of recent work on instruction tuning has focused on the quality and behavioural change of this procedure. The focus on change in model internals, as presented in this work, is an exciting and fruitful line of work, that will provide better insights into the reasons why instruction tuning is so successful.\n- The authors present an extensive experimental setup that tackles various interesting questions regarding the impact of instruction tuning and scale.", "reasons_to_reject": "- The paper does not seem to be aware of highly related work on the relation between human 'attention' (i.e. eye-tracking data) and LM 'attention'. In earlier work it has been shown that attribution methods yield a better fit to human attention than raw attention (Beinborn & Hollenstein, 2021), and the paper would benefit from engaging with this literature more (see below for references).\n- The paper also does not seem to be aware of a recent line of work that devises better methodologies of interpreting LM attention (see below for references).\n- The paper is too vague about \"human attention\" and would benefit from engaging more with literature from the cognitive science about attention in reading comprehension.\n- The paper would benefit from a better restructuring/rewriting, there are a lot of unnecessary subsubsections and 1-sentence paragraphs.\n- The paper would benefit from a more detailed analysis of the difference between L1 and L2 speakers, that reaches further than a stronger 'trivial pattern reliance' of L2 speakers.", "questions_to_authors": "1. Regarding the experiment described at the end of Section 3.1: how do you account for the fact that the attention distribution will be different because it also flows to the tokens in the instruction itself?\n2. Did you consider aggregating LM attention patterns across multiple layers? For LMs the attention is distributed across layers, so it is not completely 'fair' to compare an individual layer's attention to human attention as a whole.", "paper_topic_and_main_contributions": "The paper investigates the impact of 2 important factors in LLM research, **scaling** and **instruction tuning**, with respect to how they affect Transformer attention and align with 'human attention'. Using the Jensen-Shannon divergence as a metric for measuring the difference of attention distribution between models, and L1/L2 speakers, the authers uncover various surprising findings on the role of scaling and instruction tuning.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "There has been quite some work on the connection between model attention/attribution and human attention/reading importance via eye tracking experiments:\n- \"Relative Importance in Sentence Processing\" - Beinborn & Hollenstein (2021)\n- \"A Cross-lingual Comparison of Human and Model Relative Word Importance\" - Morger et al. (2022)\n- See also the \"CMCL 2021 Shared Task on Eye-Tracking Prediction\" (https://aclanthology.org/2021.cmcl-1.7/)\n\nThe paper also misses a lot of papers that have recently presented better methods of interpreting attention by looking further then just the attention weights and incorporating the content of the value vectors as well.\n- \"Quantifying Attention Flow in Transformers\" - Abnar & Zuidema (2020)\n- \"Attention is Not Only a Weight: Analyzing Transformers with Vector Norms\" - Kobayashi et al. (2020)\n- \"Incorporating Residual and Normalization Layers into Analysis of Masked Language Models\" - Kobayashi et al. (2021)\n- \"Measuring the Mixing of Contextual Information in the Transformer\" - Ferrando et al. (2022)\n- \"Quantifying Context Mixing in Transformers\" - Mohebbi et al. (2023)\n- \"Explaining How Transformers Use Context to Build Predictions\" - Ferrando et al. (2023)", "Typos_Grammar_Style_and_Presentation_Improvements": "## Suggestions\n- Explain what types of \"human attention\" you use for your analysis, right now in Section 3 you have not explained this at all yet, and only in Section 4.2 this is briefly touched upon. This is important though, as you place a lot of importance on this \"human attention\" in your analysis, so it should be clear to a reader what exactly you mean by it.\n- Refrain from using the term \"prove\" when there is no theoretical proof involved, you merely \"demonstrate\" something here.\n\n## Style / Typos\n- The sentence around L127 (On the other hand...) is hard to understand."}}]

Author rebuttal:

Meta-review: The work is about the effects of scaling and instruction tuning on the attention distribution and language modeling performance of large language models (LLMs). The authors use two types of instructions: L1 for general language modeling and L2 for specific tasks. They measure the attention distribution using three metrics: entropy, human resemblance, and trivial attention patterns. They also compare the perplexity scores of different models on various datasets.

Pros: 
1: An interesting and surprising finding that challenges the common assumption that instruction tuning improves the self-attention distribution of transformer language models.
2: Revealed a novel and intriguing phenomenon that the self-attention distribution of large language models is closer to non-native speaker attention distributions than native speakers, which opens up new avenues for research and improvement.

Cons:
1: Uses the term “human attention” loosely and without proper definition or justification, and does not relate it to the model attention or the cognitive processes involved in reading comprehension.
2: Didn't provide a detailed or meaningful analysis of the difference between L1 and L2 speakers, and only mentions a stronger trivial pattern reliance of L2 speakers, which does not explain the underlying causes or consequences of this phenomenon.
Decision: Accept-Findings