Transformer-Based Language Model Surprisal Predicts Human Reading Times Best with About Two Billion Training Tokens
Authors: Byung-Doh Oh
William Schuler
Venue: EMNLP 2023 Findings (2023)
Keywords: surprisal theory
human sentence processing
large language models

Abstract: Recent psycholinguistic studies have drawn conflicting conclusions about the relationship between the quality of a language model and the ability of its surprisal estimates to predict human reading times, which has been speculated to be due to the large gap in both the amount of training data and model capacity across studies. The current work aims to consolidate these findings by evaluating surprisal estimates from Transformer-based language model variants that vary systematically in the amount of training data and model capacity on their ability to predict human reading times. The results show that surprisal estimates from most variants with contemporary model capacities provide the best fit after seeing about two billion training tokens, after which they begin to diverge from humanlike expectations. Additionally, newly-trained smaller model variants reveal a 'tipping point' at convergence, after which the decrease in language model perplexity begins to result in poorer fits to human reading times. These results suggest that the massive amount of training data is mainly responsible for the poorer fit achieved by surprisal from larger pre-trained language models, and that a certain degree of model capacity is necessary for Transformer-based language models to capture humanlike expectations.

Transformer-Based Language Model Surprisal Predicts Human Reading Times Best with About Two Billion Training Tokens
Byung-Doh Oh Department of Linguisties The Ohio State University oh. 53 10ssu. edu
William Schuler
Department of Linguistics
The Ohio State University schuler. 77@.su.7cuucmm edu
Abstract
been evaluated on their ability predict measures of processing difficulty.
Recent psycholinguistic studics have drawn conflicting conclusions about the relationship between the quality of language model and the ability of its surprisal estimates to predict human reading times, which has been specu- lated to be due to the large gap in both the amount of training data and model capacity across studies. The current work aims to consol- idate these findings by evaluating surprisal esti- mates from Transformer-bo based language model variants that vary systematically in the amount training data and model capacity on their ability predict human reading times. The results show that surprisal estimates from most variants with contemporary model capacities provide the best fit after seeing about bil- lion training tokens, after which they begin to diverge from humanlike expectations. Addi- tionally, newly-trained smaller model variants reveal a "tipping point' at convergence, after which the decrease in language model perplex- ity begins to result in poorer fits to human read- ing times. These results suggest that the mas- sive amount of training data is mainly respon- sible for the poorer fit achieved by surprisal from larger pre-trained language models, and that certain degree of model capacity is nec- essary for Transformer- based language models capture humanlike expectations.
Recent studies in computational psycholinguis- tics have provided conflicting evidence with regard to the relationship between LM quality (i.e. next- word prediction accuracy) and goodness-of-frr to human reading times. Earlier work using newly- trained LMs showed negative relationship be- tween LM perplexity and predictive power of sur- prisal estimates (Goodkind and Bicknell, 2018:
Wilcox et al., 2020; Merkx and Frank, 2021), but more recent work using large pre-trained Transformer LMs (c.g. GPT-2; Radford et al., 2019) 9) show a robust positive relationship between the two variables (Oh et al., 2022; Oh and Schuler, 2023). While Oh and Schuler (2023) conjecture that these studies capture two distinct regimes, it remains less clear where the reversal in this rela- tionship happens. The main challenge answering this question lies in the massive difference terms of both the amount of training data and the model capacity of LMs that were studied.
The current study aims to cover this conceptual middle ground by evaluating, on their ability to pre- dict human reading times, surprisal estimates from Transformer-based LM variants that vary system- atically in the amount of training data and model capacity. Results from regression analyses show that surprisal from most LM variants with contem- porary model capacities make the biggest contri- bution to regression model fit after seeing about two billion tokens of training data, after which ad- ditional training data result in surprisal estimates that continue to diverge from humanlike expecta- tions. Additionally, surprisal estimates from newly- trained smaller LM variants reveal a 'tipping point' convergence, after which the decrease in perplex- ity begins result in poorer fits to human read- ing times. Taken together, these results suggest that the vast amount of training data is mainly re- sponsible for the poorer fit achieved by surprisal
1 Introduction
The predictability of upcoming linguistic material has long been considered crucial factor underly- ing difficulty in human sentence processing (Hale, 2001; Levy, 2008), and has received empirical support from numerous studies showing surprisal (Shannon, 1948) to be highly predictive of relevant behavioral and neural measures (c.g. Demberg and Keller, 2008; Smith and Levy, 2013; Hale al., 2018; Shain al., 2020). Since language models (LMs) are trained to estimate conditional proba- bility distribution of a word given its context, sur- prisal estimates calculated from them have often

---

from larger Transtormer-basee pre-trained LMs (Oh al., 2022; Oh and Schuler, 2023), and that a certain degree of model capacity is necessary for Transformer- based LMs to capture humanlike expectations that manifest in reading times.
Model
#L
dannal
#Parameters
Pythia 70M
512
~00
Pythia 160M
12
12
768
~160M
Pythia 410M
24
16
1024
~410M
Pythia IB
16
2048
~B
Pythia 1.4B
24
16
2048
1B
Pythia 2.8B
32
32
2560
~288
Pythia 6.9B
32
32
4096
<6699
2 Experiment 1: Influence of Training Data Size
Pythia 12B
36
40
5120
~122
Table I: Model capacities of Pythia variants whose surprisal estimates were examined in this work. #L, fiH, and dmodal refer number of layers, number of attention heads per layer, and embedding size, respectively.
The first experiment examines the influence of training data size on the predictive power of Transformer-basee LM surprisal by evaluating LM variants at various points in training on self-paced reading times from the Natural Stories Corpus eye-gaze durations
Crucially for this experiment, all eight Pythia variants were trained using identical batches of training examples that were presented in the same order. These training examples come from the Pile (Gao et al., 2020), which is a collection of English language datasets that consist of around 300 bil- lion tokens. Batches of 1,024 examples with a sequence length of 2,048 (i.e. 2,097,152 tokens) were used train the eight variants for 143,000 steps, which amounts to about one epoch of the entire Pile dataset. Model parameters that were saved during early training stages (i.e. after 1, 2, 4, 256, 512 steps) as well as after every 1,000 steps are publicly available.
(Futrell et al., 2021) and gy-pasteee-gaze from the Dundee Corpus (Kennedy et al., 2003).
2.1 Response Data
The Natural Stories Corpus contains reading times from 181 subjects that read 10 naturalistic English stories consisting a total of 10,245 tokens. The data points were filtered to remove those for sentence- initial and final words, those from subjects who answered three or fewer comprehension questions longer
correctly, and those shorter than 100 than 3000 ms, which resulted in a total of 384,905 observations in the exploratory set. The Dundee Corpus contains eye-gaze durations from 10 sub- jects that read 67 newspaper editorials consisting a total of 51,501 tokens. The data points were fil- tered to exclude those for unfixated words, words following saccades longer than four words, and sentence-, screen-, document-, and line-initial and final words, which resulted in total of 98, 15 ob- servations in the exploratory set.' All observations were log-transformed prior to model fitting.
Each article of the Natural Stories Corpus and each article of the Dundee Corpus tokenized by Pythia's byte-pair encoding (BPE; Sennrich et al., 2016) tokenizer and provided as input to each model variant. For each model variant, all publicly available intermediate model weights were used to calculate surprisal estimates on the two corpora. In cases where each story or article was longer than single context window of 2,048 tokens, surprisal estimates for the remaining tokens were calculated by using the second half of the previous context window as the first half of new context window.
2.2 Predictors
This experiment evaluates surprisal estimates from eight variants of Pythia LMs (Biderman et al., 2023), whose intermediate parameters were saved at various points during training. Pythia LMs are decoder-only autoregressive Transformer-base based models? whose variants differ primarily in their ca- pacity. The model capacities of the Pythia variants are summarized in Table 1.
2.3 Regression Modeling
Subsequently, following previous work (Oh et al.. 2022; Oh and Schuler, 2023), a 'baseline' linear mixed-effects (LME) model that contains baseline predictors for low-level cognitive processing, and 'full' LME models that additionally contain each LM surprisal predictor, were fit to self- read- ing times and go-past durations using 1me4 (Bates et al., 2015). These baseline predictors are word length in characters and index of word position in each sentence (Natural Stories and Dundee), as well as saccade length and whether or not the pre-
'The held-out set of each corpus, which have a comparable number of observations, reserved statistical significance testing and therefore was not analyzed this work. 'Technical details such as the parallelization of self- intention/fcedforwee computations and the separation of em- bodding//rojection matrices differentiate Pythia LMs from other large language model families.

---

Natural Stories SPR
Dundee
73M
1200
K00M
700
41OM
1200
1000
300
500
120
800
400
400
300
256
16k
143
143
256
16k
64% 143K
Training Steps
Training Steps
70M
1200
325M
4:0M
700
3.40
2,80
600
328
800
500
400
400
300
2
273
2015
2/0
200
20
21-
2000
2.0
201a
275
Perplexity
Perplexity
302M
2205
1.40
Perplexity
2:90
Perplexity
215.0
320
215.
120
2,0
230
250
25.
16
64
256
18
88
16k
64k
16
6A
256
1%
Ak.
16k
64%
Training Steps
Training Steps
Figure 1: Increase in regression model log- oigliikei due to including each surprisal estimate from Pythia variants as a function of training steps (top) and perplexity (middle; the stars indicate the fully trained versions after 143,000 steps), as well perplexity function of training steps (bottom) on the exploratory of Natural Stories (left) and Dundee data (right).
vious word was fixated (Dundee only). All pre- dictors were centered and scaled, and the LME models included by-subject random slopes for all fixed effects and random intercepts for each subject. addition, a random intercept for each subject- sentence interaction was included for self- cele-daced reading times collected from 181 subjects, and random intercept for each sentence was included for eye-gaze durations collected from smaller number of 10 subjects. Once the regression mod- els were fit, the increase in regression model log- likelihood (ALL) was calculated for each regres- sion model by subtracting the log-liket ligeliooldoood of the baseline regression model from that of full re-
gression model. Finally, the perplexity of each LM variant was calculated on the two corpora.
2.4 Results
The results in Figure show that across both cor- pora, surprisal from most LM variants made the biggest contribution regression model fit after 1,000 training steps (i.e. after about two billion tokens). This seems to represent 'humanlike
*Results from after 2,000 steps were selectively omitted for clarity, as they were consistent with the general trend. As pointed out by reviewer, including frequency-based pre- dictor in the regression models may change the exact location of this peak. However, this work avoids potential confounds introduced by the corpus used for frequency estimation by evaluating surprisal estimates on their own following the pro- tocols of Oh and Schuler (2023).
"Spillover' predictors were not included in the regression models to avoid convergence issues.

---

Natural Stories
Dundee
800
1-144
1-144
750
3-2-120
1100
2-3-192
700
2-3-192
1000
4-2
650
3-4-256
2-4-556
600
900
6-8-512
6-8-512
550
300
500
450
700
256
2K
3k
4k
5k
7k
10k
256
1K
2K
3K
4k
5K
6k
7k
8K
9K
10k
Training Steps
Training Steps
11:04
1::64
1:2-128
750
1100
2:3-352
700
192
1000
244-256
650
2:4-256
600
300
08:512
08:512
800
500
700
2¹
210
202
21
2
216
2
2'
21n
2
2
21
2
2
Perplexity
Perplexity
220.
1-1-44
220.0
1-1-64
2080
1-2-120
2180
1-2-128
2.60
2-3-192
Perplexity
2-4-256
Perplexity
2-4-256
214.
3-4-256
3-4-256
234.
212.
6--512
5-5-512
2170
2
200.
280
250
16
64
256
1k
2k
3k
4k.
5k
6k
7k
8K
5k
10k
16
64
256
1k
2k
36
4k.
5k
6k
7k
8K
96
10k
Training Steps
Training Steps
Figure 2: Increase in regression model log-likelihood due to including each surprisal estimate from newly-trained LM variants as function of training steps (top) and perplexity (middle; the stars indicate the fully trained versions after 10,000 steps), well as perplexity Stories (left) and Dundee data (right). The variants are labeled using their number of layers, number of attention heads per layer, and embedding size, that order.
function of training steps (bottom) the exploratory set of Natural
optimum," after which surprisal estimates begin to diverge from humanlike expectations training continues. At this point in training, there appears be no systematic relationship between model ca- pacity and predictive power surprisal estimates. However, after all 143,000 training steps (i.e. after about 300 billion tokens), the eight model variants show a strictly monotonic and negative relation- ship, which directly replicates the findings of Oh and Schuler (2023) 5 Taken together, these results indicate that the vast amount of training data is re- sponsible for the poorer fit achieved by surprisal
from larger Transformer-basee LMs.
3 Experiment 2: Influence of Model Capacity
The second experiment further examines the re- lationship between model capacity and predic- tive power of surprisal estimates by evaluating Transformer-based LM variants smaller than the Pythia variants at various points in training, follow- ing similar procedures Experiment
3.1 Procedures
Surprisal estimates from eight smaller LM vari- ants were evaluated at various points during train- ing in this experiment. The largest of these vari-
"The best-fitting line between log perplexity and ALL of these variants had a slope significantly greater than 0 at < 0.05 level according to one-tailed f-test on hoth corpora.

---

ants has the same model capacity as the smallest Pythia 70M variant, and the smaller variants were designed to have fewer layers and attention heads, as well as smaller embeddings. These variants were trained closely following the training proce- dures of the Pythia variants, including the size and order of training batches. For computational ef- ficiency, these variants were trained for the first 10,000 training steps, based the observation that ALL on both corpora did not change substantially after 8,000 steps for the smallest Pythia variant. The predictive power of resulting surprisal esti- mates was evaluated following identical procedures as Experiment 1.
helpful for learning e.g. selectional preferences that align well with humanlike prediction and process- ing difficulty. However, as the models see more training data, they are able to achieve 'superhuman' prediction, which makes their surprisal estimates diverge more and more from human reading times as training continues. The words for which predic- tion by LMs improves with massive amounts of training data are likely be open-class words like nouns and adjectives, whose reading times were identifed as being most severely underpredicted by their surprisal estimates (Oh and Schuler, 2023).
Moreover, at the end of training, these model variants show strictly monotonic and negative relationship between perplexity and fit human reading times. This directly replicates the find- ings of Oh et al. (2022) and adds to a growing body of research reporting an inverse correlation be- tween model size and regression model fit (Kurib- ayashi et al., 2022; Shain et al., 2022; de Varda and Marelli, 2023). The current results demonstrate that this relationship emerges with large amounts of training data and becomes stronger as training continues. The bottleneck posed by the limited model capacity of the smaller variants appears to prevent them from learning to make excessively accurate predictions that cause the divergence be- tween surprisal and human reading times. How- ever, newly-traine LM variants that are smaller than those of contemporary standards reveal 'tip- ping point' at convergence, which indicates that certain amount of model capacity is necessary for LMs to correctly learn humanlike expectations. Finally, across both experiments, model capacity does not seem to modulate the relationship between perplexity and fit to human reading times, with data points from different LM variants forming a con- tinuous curve between log perplexity and ALL. This suggests that Transformer-based LMs of dif- ferent capacities share similar inductive bias that initially improves the fit of surprisal estimates to human reading times but begins to have an adverse effect on it with large amounts of training data.
3.2 Results
The results in Figure 2 show that surprisal from the two largest variants made the biggest contribution to regression model fit after 1,000 training steps on both corpora, replicating the results of Experiment 1. In contrast, the smaller variants such as the 2- 2-128 and 2-3-192 variants seem to peak later at around 2,000 training steps and stabilize afterward. After all 10,000 training steps, the model variants show a reversal in the relationship between LM perplexity and fit to reading times; the 2-3-192 variant seems to represent 'tipping point,' after which the decrease in perplexity starts result in poorer fits to human reading times. Additionally, variants that are smaller than this yield surprisal estimates that are less predictive of reading times when sufficiently trained. These results suggest that certain degree of model capacity is necessary for Transformer-bastt based LMs to capture humanlike expectations that manifest in reading times.
4 Discussion and Conclusion
This work aims to consolidate conflicting findings about the relationship between LM quality and the predictive power of its surprisal estimates by systematically manipulating the amount of train- ing data and model capacity. Experimental re- sults show that surprisal from most contemporary Transformer-basse LM variants provide the best fit to human reading times with about two billion training tokens, after which they begin to diverge from humanlike expectations. It is conjectured that early training data up to about two billion tokens is
A Acknowledgments
We thank the reviewers and the area chair for their helpful comments. This work was supported by the National Science Foundation grant #1816891. All views expressed are those of the authors and do not necessarily reflect the views of the National Science Foundation.
"Refer to Appendix A for the model capacities of these variants as well as further details on their training procedures. Code and trained weights are available at https: //gi thub. con/ com/ byungdoh/ /s/m surprisal.

---

Limitations
Richard Futrell, Edward Gibson, Harry Tily, Idan Blank, Anastasia Vishnevetsky, Steven Piantadosi. and Evelina Fedorenko. 2021. The Natural Stories corpus: A reading- time corpus of English texts con- taining rare syntactic constructions. Language Re- sources and Evaluation, 55:63-77.
The connection between conditional probabilities of Transformer-basee language models and human sentence processing drawn in this work is based on language model variants trained English text and data from human subjects that are native speakers of English. Therefore, the connection made this work may not generalize to other languages.
Leo Gao, Stella Biderman, Sid Black, Laurence Gold- ing, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. The Pile: An 800GB dataset diverse text for language modeling. arXiv preprint, arXiv:2101 M0Xx:2211100022 0000270
Ethics Statement
Experiments presented in this work used datasets from previously published research (Futrell et al., 2021; Kennedy et al., 2003), in which the proce- dures for data collection and validation are outlined. As this work focuses on studying the connection between conditional probabilities of language mod- els and human sentence processing, its potential negative impacts on society seem be minimal.
Adam Goodkind and Klinton Bicknell. 2018. Predic- tive power of word surprisal for reading times a linear function of language model quality. In Pro- ceedings of the 8th Workshop on Cognitive Modeling and Computational Linguistics, pages 10-18.
John Hale. 2001. A probabilistic Earley parser as a psy- cholinguistic model. In Proceedings of the Second Meeting of the North American Chapter of the Asso- ciation for Computational Linguistics on Language Technologies, pages 1-8.
References
John Hale, Chris Dyer, Adhiguna Kuncoro, and Jonathan Brennan. 2018. Finding syntax in human encephalography with beam search. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 2727-2733
Alex Andonian, Quentin Anthony, Stella Biderman, Sid Black, Preetham Gali, Leo Gao, Eric Hallahan, Josh Levy-Kramer, Connor Leahy, Lucas Nestler, Kip Parker, Michael Pieler, Shivanshu Purohit, Tri Songz, Phil Wang, and Samuel Weinbach. 2021. GPT-NeoX: Large scale autoregressive language modeling in Py- Torch.
Alan Kennedy, Robin Hill, and Joël Pynte. 2003. The Dundee Corpus. In Proceedings of the 12th Euro- pean Conference on Eye Movement.
Douglas Bates, Martin Mächler, Ben Bolker, and Steve Walker. 2015. Fitting linear mixed-effects models using 1me4, Journal of Statistical Software, 67(1):1- 48.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Rep- resentations, pages 1-15.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O' BrBrie. Eric Hal- lahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. 2023. Pythia: A suite for analyzing large language mod- els across training and scaling. In Proceedings of the 40th International Conference on Machine Learning. volume 202, pages 2397-2430.
Tatsuki Kuribayashi, Yohei Oseki, Ana Brassard, and Kentaro Inui. 2022. Context limitations make neural language models more human-like In Proceedings of the 2022 Conference on Empirical Methods Natural Language Processing, pages 10421-10436.
Roger Levy. 2008. Expectation based syntactic compre- hension. Cognition, 106(3): 1177.
Danny Merkx and Stefan L. Frank. 2021. Human sen- tence processing: Recurrence attention? In Pro- ceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pages 12-22.
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. arXiv preprint, arXiv:2205.1411
Byung-Doh Oh, Christian Clark, and William Schuler, 2022. Comparison structural parsers and neural language models as surprisal estimators. Frontiers in Artificial Intelligence, 5:777963.
Andrea de Varda and Marco Marelli. 2023. Scaling in cognitive modelling: a multilingual approach to human reading times. In Proceedings of the s1 Annual Meeting of the Association for Computational Linguistics, pages 139-149.
Byung- -oong-Dohho Oh and William Schuler. 2023. Why does surprisal from larger Transformer-based language models provide a poorer fit human reading times? Transactions of the Association for Computational Linguistics, 11:3363350.
Vera Demberg and Frank Keller. 2008. Data from eye- tracking corpora as evidence for theories of syntactic processing complexity. Cognition, 109(2):193-210)

---

Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models unsupervised multitask learners. OpenAl Technical Report.
Model
Smatel
#Parameters
Repro 1-1-64
~66
Repro 1-2-128
128
~13
Repro 2-2-128
128
13M
Repro 2-3-192
192
~20
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase. and Yuxiong He. 2020. ZeRO: Memory optimiza- tions toward training trillion parameter models. In Proceedings of the International Conference for High Performance Computing. Networking, Storage and Analysis, pages 1-16.
Repro 2-4-256
256
~27M
Repro 3-4-256
256
~2M
Repro 4-6-384
384
4-46
Repro 6-8-512
512
700
Model capacities of newly-trained LM variants whose surprisal estimates were examined in this work. #I., #H, and dmadel refer to number of layers, number of attention heads, and embedding size, respectively.
Table 2:
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Lin- guistics, pages 1715-1725.
2020) were provided each variant in the same or- der as the Pythia variants. The variants were trained using the Zero Redundancy Optimizer (ZeRO; Ra- jbhandari et al., 2020) implementation of Adam (Kingma and Ba, 2015) with a learning rate of 0.001. The learning rate was warmed up linearly over the first 1% of training steps (i.e. 1,430 steps) and were subsequently lowered to minimum of 0.0001 following cosine annealing schedule over the remainder of the 143,000 training steps. How- ever, for computational efficiency, training was stopped after the first 10,000 training steps. For comparability with the Pythia variants, intermedi- ate parameters were saved during carly training stages (i.e. after 1, 2, 4. ... 256, 512 steps) as well as after every 500 steps from step 1,000 onward.
Cory Shain, Idan Asher Blank, Marten van Schijn- del, William Schuler, and Evelina Fedorenko. 2020. reveals aegguae--peeiihhe predictive coding dur- ing naturalistic sentence comprehension. Neuropsy- chologia, 138: 107307.
Cory Shain, Clara Meister, Tiago Pimentel, Ryan Cot- terell, and Roger Levy. 2022. Large-scale evidence for logarithmic effects of word predictability on read- ing time. PsyArXiv preprint.
Claude Elwood Shannon. 1948. A mathematical theory of communication. Bell System Technical Journal, 47:379-423.
Nathanicl J. Smith and Roger Levy. 2013. The effect of word predictability on reading time is logarithmic. Cognition, 128:302-319.
Ethan Godleb Wilcox, Jon Gauthier, Jennifer Hu, Peng Qian, and Roger P. Levy. 2020. the predictive power of neural language models for human real- time comprehension behavior. In Proceedings of the 42nd Annual Meeting of the Cognitive Science Society, pages 1707-1713.
A Model Capacities and Training Procedures of Smaller LM Variants
The eight LM variants that were trained as part of Experiment 2 are decoder-only autoregressive Transformer-baat based models that share the same ar- chitecture as the Pythia LM variants (Biderman al., 2023). Their model capacities summa- rized in Table 2.
These variants were trained using the GPT-NeoX library (Andonian et al., 2021) closely following the training procedures of the Pythia LM variants.
Identical training batches of 1,024 examples with a sequence length of 2,048 from the Pile (Gao et al..
'The only minor difference that the FlashAttention (Dao et al., 2022) implementation scaled dot-product attention could not be used during training due to mismatch in GPU hardware specifications.


Reviews:
[{"review_id": "Qb3EGGahmB", "reviewer": "Reviewer_nrTc", "summary": "", "questions": "- A: Why does \"the 2-3-192 variant\" represent \"a tipping point\"?\n- B: How are random-effects structures of regression modeling determined for self-paced reading times and eye-gaze durations, respectively?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your critical assessment and constructive feedback on this work, as well as references to earlier work that observed a similar relationship between language model perplexity and fit to human reading times on Japanese data. We will situate our work in relation to these results in a “Discussion” section if the paper is accepted.\n\nAccording to the guidelines for author response, below we address your concerns and questions point by point, presenting new results where appropriate.\n\n- **Reasons to Reject #1:** We are sorry that the description of the results was not clear for the experiment manipulating model capacity. The observation that “the smaller variants seem to peak later at around 2,000 training steps and stabilize afterward” was meant to describe that unlike the two largest variants examined in the experiment (i.e. ‘4-6-384’ and ‘6-8-512’ variants), the smaller variants (e.g.  ‘2-2-128’, ‘2-3-192’, and ‘2-4-256’ variants) have a later peak and do not demonstrate a substantial decrease in ΔLL by training step 10k, suggesting their convergence. This can also be seen in the bottom panels of Figure 2, where perplexity stops decreasing for the smaller variants (resulting in clusters of data points). We will clarify this description if the paper is accepted.\n\n- **Reasons to Reject #2:** We speculate that training data up to two billion tokens is helpful for learning e.g. selectional preferences that align well with humanlike prediction and processing difficulty. However, as the models see more training data, they are able to achieve ‘superhuman’ prediction, which makes their surprisal estimates diverge more and more from human reading times as training continues. This is consistent with the findings of Oh and Schuler (2023, “Why does surprisal from larger Transformer-based language models provide a poorer fit to human reading times?”), and we will add a discussion of these points in a “Discussion” section if the paper is accepted.\n\n- **Question A:** Based on the results of the first experiment, a natural question to ask is how much smaller Transformer language models can become before their surprisal estimates begin to degrade. In this regard, the ‘2-3-192’ variant (the red stars in the bottom panels of Figure 2) represents a ‘tipping point’ in the sense that language models that are smaller in scale begin to yield surprisal estimates that are less predictive of human reading times when sufficiently trained.\n\n- **Question B:** We followed the experimental protocols of Oh and Schuler (2023) to determine the random effects structures on the two corpora of human reading times. Their protocol is consistent with the conventional practice of psycholinguistics of keeping maximal by-subject random effects (Barr et al., 2013, “Random effects structure for confirmatory hypothesis testing: Keep it maximal”).\n\n- **Reproducibility:** With regard to your concern for reproducibility, we would like to point out that the pre-trained Pythia models that were examined in the first experiment as well as the training data for models examined in the second experiment are publicly available (Biderman et al., 2023, “Pythia: A suite for analyzing large language models across training and scaling”), as well as the regression software and the Natural Stories Corpus. We also plan to publicly release weights of all newly-trained models as well as code used for surprisal calculation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- To the best of my knowledge, this paper is the first systematic investigation of the correlation between model quality and psychometric predictive power with respect to training data and model capacity.", "reasons_to_reject": "- While the results of training data are clear, the results of model capacity are hard to interpret. For example, the observation that \"the smaller variants seem to peak later at around 2,000 training steps and stabilize afterward\" cannot be seen from Figure 2.\n- The reason why two billion training tokens are the magical number remains to be discussed.", "questions_to_authors": "- A: Why does \"the 2-3-192 variant\" represent \"a tipping point\"?\n- B: How are random-effects structures of regression modeling determined for self-paced reading times and eye-gaze durations, respectively?", "paper_topic_and_main_contributions": "The main contribution of this paper is to demonstrate that the correlation between model quality and psychometric predictive power does hold beyond two billion training tokens.", "excitement": 3, "reproducibility": 1, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "- Tatsuki Kuribayashi, Yohei Oseki, Takumi Ito, Ryo Yoshida, Masayuki Asahara, and Kentaro Inui. 2021. Lower perplexity is not always human-like. ACL."}, "extra_text": {}}, {"review_id": "Kim3A3DRQA", "reviewer": "Reviewer_vi7K", "summary": "", "questions": "A. It would be nice to see a plot of perplexity by training steps - Acknowledged\nB. I would also like to see R^2 reported as well as significance tests for those sections where fit is assessed -- currently it is only described qualitatively and it would be nice to know that the curves are in fact distinct and/or that there are significant differences as a function of model parameters - Acknowledged", "limitations": "", "rating": 3, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your critical assessment and constructive feedback on this work. Some of the confusion may be due to the fact that the discussion of the results was abrogated due to space constraints. We will include a discussion of the points below in a “Discussion” section in the allotted extra page if the paper is accepted.\n\nAccording to the guidelines for author response, below we address your concerns and questions point by point, presenting new results where appropriate.\n\n- **Reasons to Reject #1:** The main theoretical implication of this work is that the relationship between next-word prediction accuracy (i.e. perplexity) and the predictive power of surprisal estimates is modulated by the amount of training data language models have seen. Until very recently, it was thought that more accurate language models are better approximators of humanlike expectations, and this assumption has even informed recent work on the functional form between surprisal and reading times (Hoover et al., 2023, “The plausibility of sampling as an algorithmic theory of sentence processing”). Our work shows that after about two billion training tokens, more accurate predictions lead to divergence from humanlike expectations.\n\n- **Reasons to Reject #2:** With regard to the peak observed in this work, we speculate that training data up to two billion tokens is helpful for learning e.g. selectional preferences that align well with humanlike expectations and processing difficulty. However, as the models see more training data, the models are able to achieve ‘superhuman’ levels of prediction, which make their surprisal estimates diverge more and more from human reading times as training continues. This is consistent with the results of Oh and Schuler (2023, “Why does surprisal from larger Transformer-based language models provide a poorer fit to human reading times?”), who found that excessively accurate predictions of open-class words like proper nouns and adjectives lead to underpredictions of human reading times.\n\n- **Reasons to Reject #3:** To our knowledge, the Pythia models are the only pre-trained language models that use purely publicly available training data and have publicly available checkpoints at various points during training.\n\n- **Reasons to Reject #4:** While this work evaluates language models under the assumptions of surprisal theory, it is less straightforward to evaluate them under other (e.g. memory-based) theoretical frameworks, as their only training objective is to predict the next word. Additionally, the fact that surprisal from these language models is predictive of human reading times across two corpora and various model sizes (i.e. all ΔLL are positive and substantial) provides support for this framework. The ability of language model surprisal to predict measures of comprehension difficulty has been replicated many times in recent work (Wilcox et al., 2020, “On the predictive power of neural language models for human real-time comprehension behavior”; Shain et al., 2020, “fMRI reveals language-specific predictive coding during naturalistic sentence comprehension”, inter alia).\n\n- **Reasons to Reject #5:** Go-past durations were analyzed in this work as regressions are thought to reflect additional processing difficulty that is incurred by the current word. We additionally conducted regression analyses using the scan-path fixation durations of the Dundee corpus instead, but the overall findings (i.e. the peak in ΔLL at 1k training steps, poorer fit to reading times by surprisal from larger language models at the end of training) did not change.\n\n- **Question A:** Perplexity decreases monotonically as a function of training steps, with the total decrease being larger for the larger language model variants at the end of training. We will include a visualization of these two variables in the Appendix if the paper is accepted.\n\n- **Question B:** We will include a visualization of the regression models’ R$^2$ values. Additionally, we will report the results of a non-parametric permutation test that tests whether the slope between log perplexity and ΔLL is greater than that expected by chance at the end of language model training (the stars in the bottom panels of Figure 1; *p* < 0.001 on both corpora).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Paper is easy to read\n* Experiments are well-described and soundly conducted\n* Evaluation is fair and conducted over multiple reading time datasets", "reasons_to_reject": "* It is not clear what this answers with respect to any cognitive questions - Author response indicates a restatement of the introduction which does not answer a cognitive question (rather an empirical one); restatement could be clearer in the body of the text if this is truly a sufficient response to my question.\n* The specific mechanisms about the psychometric fit and where it peaks are not well-understood - The authors do not address the theoretical questions outlined by \"why is it these specific values.\" The speculations in the text are critical to their scientific question -- it is not trivial to raise one's hands up about the theory.\n* Results are limited to the Pythia models - The author response is sufficient. This is however a limitation about the generality of these effects that could be made clear earlier.\n* Surprisal is assumed to be the linking function which does not fully address the \"psychometric fit\" issue - The authors make an earnest attempt at a response. However, my concern is that we are too focused on using surprisal to explain reading times. If one disagrees with the premise, that makes this experiment superfluous. Thus, the authors' response to my concern also belongs in the text.\n* Eyetracking data selection/filtering is a bit perplexing -- it looks like everything is analyzed (e.g., regressions to earlier parts of the sentence) with some exclusions. - Thank you for the clarification. Please place this information more prominently in the text.\n\nI am particularly worried that the work is a bit \"atheoretical\" with respect to psycholinguistic phenomena. I don't really understand why the peak in psychometric fit occurs where it does and the paper's explanations do not address our understanding of language modeling (memorization, the source of deviations, etc.), our understanding of surprisal, or relationships between context and language comprehension.", "questions_to_authors": "A. It would be nice to see a plot of perplexity by training steps - Acknowledged\nB. I would also like to see R^2 reported as well as significance tests for those sections where fit is assessed -- currently it is only described qualitatively and it would be nice to know that the curves are in fact distinct and/or that there are significant differences as a function of model parameters - Acknowledged", "paper_topic_and_main_contributions": "This paper presents several experiments to account for a dissociation in the prediction of human behavioral data -- some models improve in their psychometric fit as they get better at modeling linguistic regularities, while others deviate. The paper parametrically manipulates the size of the training data and demonstrates that there is a consistent peak around 2 billion tokens in being able to account for reading times. It suggests that the apparent discrepancy in earlier modeling work is driven by training dataset size.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "* \"LM\" in the title should be spelled out to state \"Language Model\""}, "extra_text": {}}, {"review_id": "XAVn10AxfE", "reviewer": "Reviewer_AxD9", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your critical assessment and constructive feedback on this work, as well as references to earlier work that observe a similar relationship between language model perplexity and fit to human reading times. We will situate our work in relation to these results in a “Discussion” section if the paper is accepted.\n\nAccording to the guidelines for author response, below we address your concerns and questions point by point, presenting new results where appropriate.\n\n- **Reasons to Reject #1:** We are sorry that the interpretation of the results was not clear for the two experiments. The first experiment shows that surprisal from larger language models begins to demonstrate a poorer fit to reading times only after seeing a certain amount of training data (i.e. ~2 billion tokens), which suggests that larger language models can learn to achieve ‘superhuman’ prediction that deviates from humanlike processing difficulty (Oh & Schuler, 2023, “Why does surprisal from larger Transformer-based language models provide a poorer fit to human reading times?”) given the same amount of training data as the smaller language models. However, the second experiment shows that models that are too small are not able to learn enough from the training data, with models that are smaller than the ‘2-3-192’ variant yielding surprisal estimates that are less predictive of human reading times when sufficiently trained. We will add a discussion of these points in a “Discussion” section if the paper is accepted.\n\n- **Reasons to Reject #2:** We will include additional details of the regression analysis. In our experiments, spillover was not controlled for, following the protocols of Oh and Schuler (2023) in order to avoid issues with regression model identifiability.\n\n- **Reasons to Reject #2:** We additionally conducted regression analyses on two more eye-tracking datasets, namely the Ghent Eye-tracking Corpus (Cop et al., 2017, “Presenting GECO: An eyetracking corpus of monolingual and bilingual sentence reading”) and the Provo Corpus (Luke & Christianson, 2018, “The Provo Corpus: A large eye-tracking corpus with predictability norms”), following the same procedures as the first experiment. The two main results (i.e. the peak in ΔLL at 1k training steps, poorer fit to reading times by surprisal from larger language models at the end of training) were robustly replicated. We will include these results in the Appendix if the paper is accepted.\n\n- **Reasons to Reject #2:** Generally, model capacity does not seem to modulate the relationship between perplexity and ΔLL; the bottom panels of Figure 1 show that data points from models with different capacities form one continuous curve between log perplexity and ΔLL. We will additionally include a description of this observation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The consistency of the results is striking given the different nature of the datasets; the datasets have very different genres of text and were collected under very different circumstances. This increases my credence that the result is robust.\n\n- The paper makes a nice contribution to the literature and support Oh and Schuler’s idea that the reason very large language models don’t model human reading times well is that they are “too good” at prediction, and know too many things such as low-frequency entity names that humans do not know. In some ways the paper raises more questions than it answers (why the inflection point at 1k?) but this is good as it will inspire future work.", "reasons_to_reject": "- The paper claims that “a certain degree of model capacity is necessary for Transformer-based LMs to capture humanlike expectations that manifest in reading times.” It’s not clear to me from the text, nor from the figures, what exactly the claim is here. Is it that models need to have low capacity to model human reading times, and too-high capacity is deleterious? If so would that support accounts like Futrell, Gibson & Levy’s (2020) lossy-context surprisal? Or is it that a model with too high capacity can learn too much from the training data? Or is the claim that models need a certain capacity and beyond this there are diminishing returns? It’s not clear to me what the specific claim here is.\n\n- I also think the paper could be improved. First, some details of the analysis aren’t clear. Was spillover accounted for and how? It would also be nice to move beyond the Dundee and NS datasets. Also, in terms of the analysis, it would be interesting to see plots comparing Delta LL for the different model architectures in Figure 2 but matched in perplexity (by using earlier checkpoints). This could answer questions about the real influence of model capacity.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper aims to empirically investigate the relationship between language model surprisals and human reading times and in particular the relationship between model capacity and training set size and model performance in predicting RT. The paper aims to elucidate the mysterious findings that a positive relationship between LM quality and RT predictive capacity seems to reverse for very large models. The paper finds that the reversal happens at around 1k training steps, consistently for two datasets in two different modalities (eye tracking vs. self-paced reading).", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "https://aclanthology.org/2021.acl-long.405/\nhttps://aclanthology.org/2022.emnlp-main.712/"}, "extra_text": {}}]

Author rebuttal:

Meta-review: # Overview

Prior work in psycholinguistics has shown that as language models become better (in terms of perplexity/cross-entropy) their surprisal estimate’s ability to predict reading times (RTs) improves (Goodkind et al. 2018; Wilcox et al. 2020).
A few recent papers, however, show this trend reverses after a certain “language model quality threshold” (i.e., GPT-2 small’s surprisals have better predictive power than either GPT-2 XL or GPT-3; Oh et al. 2022; Oh and Schuller 2023; Shain et al. 2022).
This paper investigates when this trend-reversal occurs, by analysing how a model’s psychometric predictive power changes over training—using a number of language models with different sizes.
They find clear results suggesting that (under their experimental conditions) a language model’s psychometric predictive power peaks when they are exposed to roughly two billion training tokens.

# Meta review

The reviewers generally agree that this paper is well-written, sound and makes an important contribution to the literature. Further, the fact that experiments are consistent across a range of datasets and RT measurements is both striking and reassuring. Personally, I believe that if these results are confirmed this could be a very impactful paper (but see my personal review below).

The reviewers also point out some issues with the paper. In particular, the paper does not try to address the question of “why two billion training tokens?”---i.e., it does not try to address the cognitive questions behind this observed trend. Further, the impact of model capacity on the observed trends seems unclear. Finally, the authors claim they did not include spillover effects in their analysis due to issues with regression model identifiability, but: (i) including spillover effects is common practice and these effects can be stronger than the main effects depending on the dataset (in e.g., Brown); (ii) I do not see why identifiability should be relevant to this particular study which focuses on delta log-likelihoods (a longer form justification could be useful). In my opinion, though, these issues are either addressable for camera ready, or not large enough to warrant rejection.

# My Review

First, I'd like to say that I agree with the positive points about the paper summarised above.
However, I personally believe that this paper has a significant experimental flaw: it **does not seem to control for unigram log-probabilities** in its analyses.

This may be particularly problematic because:

* It is well-known that unigram log-probabilities have a strong predictive power over reading times.
Beyond the most recent confirmation of the effect (in Shain, 2023, which was released after EMNLP’s CFP), most prior work in this vein controlled for frequency effects (Goodkind et al. 2018; Wilcox et al. 2020, 2023; Oh et al. 2022; Shain et al. 2022).
In particular, a close look at Oh et al.’s (2022) results (comparing fig. 5 to 7 or fig. 6 to 8) shows that the predictability of unigram log-probability over RTs is larger than that of surprisal for these experimental conditions.

* Figure 5 in Chang and Bergen (2022) shows that a language model’s distribution (and by proxy its surprisals) approximate the unigram distribution most closely after about 1000 training updates (also replicated in Meister et al. 2023).

Together, these results suggest that the effect demonstrated in this paper is that: transformers approximate unigram at roughly 1k training steps; and unigram log-probabilities predict RTs better than language model’s surprisals.
In conclusion, by not controlling for frequency effects this paper might be confounding these two trends.

Furthermore, prior work (Goodkind et al. 2018; Wilcox et al. 2020) which showed that *improving language models improves their psychometric predictive power over reading times* has always controlled for frequency in their analysis.
More recent work which shows this trend stops when models get larger than GPT-2 small (Oh et al. 2022; Shain et al. 2022) also controlled for frequency.
To me, by not controlling for frequency, this paper might be investigating a different phenomenon.
In case this paper is accepted, I would recommend adding new experiments addressing this issue (or at least a disclaimer about this alternative interpretation of the results).

# References

Shain et al. 2022. Large-Scale Evidence for Logarithmic Effects of Word Predictability on Reading Time

Shain. 2023. Word Frequency and Predictability Dissociate in Naturalistic Reading

Goodkind et al. 2018. Predictive power of word surprisal for reading times is a linear function of language model quality

Wilcox et al. 2020. On the Predictive Power of Neural Language Models for Human Real-TimeComprehension Behavior 

Wilcox et al. 2023. Testing the Predictions of Surprisal Theory in 11 Languages

Chang and Bergen. 2022. Word Acquisition in Neural Language Models

Meister et al. 2023. A Natural Bias for Language Generation Models
Decision: Accept-Findings