Language Model Quality Correlates with Psychometric Predictive Power in Multiple Languages
Authors: Ethan Wilcox
Clara Meister
Ryan Cotterell
Tiago Pimentel
Venue: EMNLP 2023 Main (2023)
Keywords: Cognitive Modeling
Language Models
Eye Tracking Data
Cross-linguistic Analysis
TL;DR: We find that better language models produce more powerful estimates of human reading data in multiple languages.

Abstract: Surprisal theory (Hale, 2001; Levy, 2008) posits that a word’s reading time is proportional to its surprisal (i.e., to its negative log probability given the proceeding context). Since we are unable to access a word’s ground-truth probability, surprisal theory has been empirically tested using surprisal estimates from language models (LMs). Under the premise that surprisal theory holds, we would expect that higher quality language models provide more powerful predictors of human reading behavior---a conjecture we dub the quality--power (QP) hypothesis. Unfortunately, empirical support for the QP hypothesis is mixed. Some studies in English have found correlations between LM quality and predictive power, but other studies using Japanese data, as well as using larger English LMs, find no such correlations. In this work, we conduct a systematic crosslinguistic assessment of the QP hypothesis. We train LMs from scratch on small- and medium-sized datasets from 13 languages (across five language families) and assess their ability to predict eye tracking data. We find correlations between LM quality and power in eleven of these thirteen languages, suggesting that, within the range of model classes and sizes tested, better language models are indeed better predictors of human language processing behaviors.

Language Model Quality Correlates with Psychometric Predictive Power in Multiple Languages
100
Ryan Cotterell 100
Tiago Pimentel 6
Ethan Gotlieb Wilcox
Clara Meister
100
ETH Zürich
University of Cambridge
(ethan. wilcox, clara. meister, ryan. ryan. cotterell, tiago. pimentel @@inf. ethz. ch
Abstract
et al., 2020, 2023; Shain et al., 2022; de Varda and Marelli, 2022, 2023), Beyond these empirical findings, the relationship between predictabilityy and reading time has received formal treatment, forming the basis of surprisal theory (Hale, 2001; Levy, 2008). In surprisal theory, a word's predictability is operationalized terms of its surprisal, or contextual negative log-likelihodd: 8/(we Wee) = log (wr wer). practice, however, we do not know the contextual probability of words according to the data- generating distribution pv. Thus, it is common to estimate surprisal using the distribution over words from language model Po instead, i.e., Sg so(we w<rr - log w<r).
Surprisal theory (Hale, 2001; Levy, 2008) posits that a word's reading time is propor- tional its surprisal (i.e., to its negative log probability given the proceeding context). It has been empirically tested using surprisal estimates from language models (LMs). Under the premise that surprisal theory holds, we would expect that higher quality language models, whose predictions are more accurate, provide more powerful predictors of human reading behavior-a conjecture we dub the quality-power (QP) hypothesis. Unfortunately, empirical support for the QP hypothesis is mixed. Some studies in English have found correlations between LM quality and psychometrie predictive power, but other studies using Japanese data, well using larger English LMs, find no such correlations. this work, we conduct systematic crosslin- guistic assessment of the QP hypothesis. We train LMs from scratch on small- and medium-sized datasets from 13 languages (across five language families) and assess their ability to predict eye tracking data. find correlations between LM quality and psycho- metric predictive power in eleven of these thirteen languages, suggesting that, within the range of model classes and sizes tested, better language models provide better predictors of human language processing behaviors.
Recent advances in language modeling have drastically increased the quality of surprisal estimates from language models (Hoffmann et al., 2022; Chowdhery et al., 2022; OpenAI, 2023). The topic of interest of this work is whether this increased quality also Icads to an increase in a language model's psychometric predictive power (Frank and Bod, 2011; Goodkind and Bicknell, 2018), i.e., how well its surprisal estimates can be used to predict RTs. To put it simply, we ask: Are better language models more powerful predictors of human behavior? If surprisal theory holds, we would expect this power to correlate positively with language models' quality. We will refer this conjecture as the quality-power hypothesis, abbreviated as the QP hypothesis.
https://githie wtw/ rycoab// quallty-powe--hhottheei
1 Introduction
Empirical evidence for the QP hypothesis is mixed. When tested English corpora, some studies have found that models that place higher probability on held-out data do tend be better at predicting the pattern of human reading times (Goodkind and Bicknell, 2018; Wilcox et al., 2020). More recent work, however, has discovered two em-
The relationship between word's predictability and its reading time (RT) is an important object of study in psycholinguistics because it allows us to draw insights about how humans process sentences (Smith and Levy, 2013; Kuperberg and Jaeger, 2016; Shain et al., 2022). Extensive prior research across many datasets, populations, reading-time measurement methodologies and even languages- has found that the more predictable a word is, the faster it will be to process (Kuribayashi et al., 2021; Meister et al., 2021: Smith and Levy, 2013; Wilcox
'Empirical results mostly support the existence of linear relationship between surprisal and reading times (Smith and Levy, 2013: Shain et al., 2022: Wilcox et al., 2023): They find that surprisal is better psychometric predictor than raw probabilities, and that non lincar functions of surprisal are not hetter than lincar ones.

---

pirical exceptions: The first empirical shortcoming was presented in Kuribayashi et al. (2021) the only² non-English study (to the best of our knowledge) investigating the saality-power rela- tionship. Specifically, Kuribayashi et al. find that the QP hypothesis does not hold in Japanese. The second shortcoming was identified by Shain et al. (2022) and Oh and Schuler (2023), who observe that the quality-power relationship does not hold for the best (and most recent) language models, suggesting state-of-the-e language models might not be aligned with human incremental predictions.
fore time) a reader must spend to process word is a monotonically increasing function of its sur- prisal, where surprisal a measure of a word's information content (Shannon, 1948; Cover and Thomas, 2006). Surprisal theory has attracted lots of attention over the years, by both the natural language processing and cognitive science com- munities (Hale, 2001, 2003, 2016; Keller, 2004; van Schijndel and Schuler, 2016; van Schijndel and Linzen, 2018; Shain, 2019, 2021; Shain and Schuler, 2021, 2022; Wilcox et al., 2020; Meister
et al., 2021, 2022; Hoover et al., 2022; Oh et al., 2021; Oh and Schuler, 2022, 2023; Kuribayashi et al.. 2021, 2022, inter alia).
In this work we present two empirical studies that investigate these shortcomings. First, we conduct a systematic crosslinguistic analysis of the QP hypothesis, using small- and medium-sized LMs.* We train 104 language models from scratch on data from 13 languages, using subsamples of the Wiki40b dataset (Guo et al., 2020). Within each language, we operationalize LMs' qualities as their average per-word cross entropy on a test set. We then use the surprisals under these models to predict human reading times on multilingual corpus of eye tracking data. We find that, in eleven out of thirteen languages, there is significant correlation between language model's quality and its surprisals' psychometric predictive power.
One way surprisal theory has been evaluated is by quantifying surprisal's predictive power. If word's surprisal influences reading times, we should see evidence in naturalistic reading data, Psycholinguists thus train regressors P& to predict reading times: r(w) ~ Pø(r I)), where Pø 's input is defined as a vector of baseline features Xt. comparing the regressors' performances when sur- prisal is included and when it is not. Specifically, the predictive power of surprisal is quantified as the difference in the log- oggliielihood (Ilh) of reading time data under the two different regressors:
Am = (p x)) - [1 xx)) (1)
To investigate the second shortcoming, we note that (potential) issue with the findings presented in Oh and Schuler (2023) and Shain et al. (2022) is that the training data for these models is not publicly available, and might plausibly includes the reading time corpora against which their psychometric predictive power was evaluated. We discuss why this might present an issue, and perform an investigation of the relationship between data leakage and model's predictive power in $5, albeit with null results.
Note that this is an ideal Ath. which assumes we have access to surprisals from the data-generating distribution, which write sy (as shorthand for ( wx+)))
2.1 Estimated Surprisal
As noted in $1, do not have access p, and thus, we cannot compute sy(w; w<ec) directly. We must therefore estimate it, which we do with the use of a language model po. Formally, the language models pa that we discuss here are autoregressive probability distributions over V dM V U {LOS}, de- fined the conditional distribution Po w w<e).
2 Surprisal vs Reading Times
Originally proposed by Hale (2001), surprisal the- ory posits that the amount of effort (and there-
A language model's parameters are typically cho- sen with the objective of minimizing its cross en- tropy with the truc distribution:"
'Concureent to this study, de Varda and Marelli (2023) investigate the QP hypothesis in a crosslinguistic setup. Un- like our work, they rely on pre-trained multilingual models whereas we make use of no-house trained monolingual models. This allows them to investigate larger models than us, hut also it introduces biases inherent the use of multilingual models.
H>Lo(W)
Σ (v) WCS
Σ
solur wee) (2)
'See Blasi et al. (2022) for survey on the importance testing cognitive hypotheses beyond English.
'In practice, as do not have access to pv. this is done using Monte Carlo estimator and training dataset Drm (w (") N In=i, where (") are assumed to be sampled i.i.d.
"Note that our best in-house trained language models are still of far lower quality than the large language models ana- lyzed by Shain et al. (2022) and Oh and Schuler (2023).
from pv: (W) x MN |(win)) so(w!"" ((())

---

where S dd V* {EOS}. A language model's 's train- ing, then, pushes it towards being better surprisal estimator. This can be shown clearly when we rewrite the cross entropy as the sum of p/'s 's en- tropy and the Kullback-Leible (KL) divergence between p. and Po:
just Rényi entropy as additional predictors are presented in App. A and B. To quantify the power of the language model, we report 4uh (as laid out in eq. (4)) measured across ten folds of held-out data. A positive Åth means that including surprisal and entropy as predictors increases predictive power over reading times. A negative value of Am is also possible and it implies overfitting.
H)-o( (W) + KL(p/ |I po)
(3)
Eye Tracking Data. We measure Åm on MECO (Siegelman al., 2022), a multilingual corpus of eye tracking data collected on simple Wikipedia-style articles from thirteen different languages. Breaking the languages into their respective families, we have eight Indo-European languages (Dutch, English, German, Greek, Italian, Norwegian, Russian and Spanish), two Uralic languages (Finnish, Estonian), one Afro-Asiatic language (Hebrew), one Koreanic language (Korean) and one Turkic language (Turkish). Articles in the MECO dataset undergo a multi-step translation process to ensure that meaning was preserved across languages. Following previous studies (Smith and Levy, 2013; Wilcox et al., 2020), use cach word's gaze duration measure of reading time.? For dependent variables in our regressors, we use cross-participant averages, and treat skipped words as having a reading time of zero (as previously done by Rayner et al., 2011; Pimentel et al., 2023; Wilcox et al., 2023).
where the KL measures a statistical "distance" be- tween two distributions, being zero only when Pe = P/. Importantly, language models are not perfect estimates of pv, and so they do not achieve this minimum. Intuitively, Hy-o (W) should then give us a sense for how similar Po is to p/, and, thus, how similar 8o is to 8y. These surprisal esti- mates 8a are then in turn used to estimate surprisal's predictive power
Am = -Ilh(p((r) X2, 86)) - (r x()) (4)
We can now state the QP hypothesis in terms of these more formal definitions: the QP hypothesis predicts that as the cross entropy HV-e(W) decreases, a model's surprisal values should become better predictors of reading times, leading to larger Am.
3 Experimental Setup
Measuring Predictive Power As discussed in $2, and following previous work in this area (Goodkind and Bicknell, 2018; Wilcox et al., 2020), we quantify a language model's predictive power as its delta log likelihood (Am)). When predicting the reading time of word wi in context, we will use features associated with w2 and its two preceding words t0--1, and Wh-2 to account for spillover effects.6 6 We will refer to this combined set of three words as our target words. The vector of variables our baseline regressor X; include the log unigram frequency and the length (in characters) of our target words. The variables for our comparison regressor include x; plus two additional variables: the surprisal and the Rényi entropy of our target words. Rényi entropy is gen- eralization of Shannon entropy which measures the expected surprisal of a word, given its context, and has been shown previously to impact reading times above and beyond surprisal (Pimentel et al., 2023). Results from regressors that use just surprisal, or
Language Models. We train language models on different language sections of the Wiki40B dataset (Guo al., 2020). The data for each language is pre-split into a training, validation and test in Wiki40B. Before training, we a UnigramLMM tokenizer with a vocabulary size 32k to each lan- guage's ' entire training set. This tokenizer is then shared across all LMs within that language. We then subsample each language's training set into sets with 1M, 3M, 10M, 30M, 100M, 300M and IB tokens. We train models the entire training set available for that language. All models are trained using fairseq (Ott et al., 2019), following their rec- ommended language modeling
of this subsets, well independently each
We use a standard decoder-only transformer with 6 layers, context window size of 5 512 tokens, and shared
embeddings. We train our models using Adam (Kingma and Ba. 2015), with
Gaaze duration is the amount of time between readers first fixation on word and the first time their gaze leaves the word.
"Spillover effects, common in reading, are when properties of a word impact the reading times of subsequent words.

---

Dulch
Estonian
Engi ish
Finnish
German
0,50,
6180
E- pAll
0.55
0.55
01000
0.55
Dalta Log Likchhood (aranage per wezee
0355
41005
0:31
0.51
41000
1015
0.00
emt
Greek
Hobrow
Italian
Korean
Norwegian
0.30
00005
pdt.02
Dw00.00 pdt.00
0.55
E:00
enss
ama
9/000
oses
0/00
0.0
eree
01000
0.50
400
Russian
Spanish
Turkish
6
8
10
6
10
12
12000
0101
p-l1001
1-1130 p40.001
0.53
6152
0.51
Log Training
Tokens
10
10
12
10
Cross Entropy (on MECO)
Figure I: Results. Error bars are 95% confidence intervals on heldout data. Log training tokens is given in base 10. Fits are linear lines of hest-fit and 95% confidence intervals. Blue labels show the r- and p-value of a Pearson correlation test hetween cross entropy and delta log- likelikeliood. We find negative correlation in 11 out of 13 languages tested.
a learning rate of 5e 4000 warm-up updates, and dropout of 0.1. We evaluate our models after each full epoch on their respective validation sets, using early stopping with a patience of 3 epochs. As our measure of language model quality, we report their cross entropy on the MECO dataset.
test this trend statistically, we fit linear regression models with Åm as the sole dependent variable and cross entropy as predictor. We find significant effect of cross entropy on Åm in 11 out of our 13 analyzed languages." 8 We do not find a significant effect of cross entropy in Finnish or German. Results are shown in Fig. 1, where we also report the Pearson correlation coefficient between the cross entropy and mean Åuh of each language model. The high value of quantitatively confirms the visual trends seen in Fig. 1.
4 Results of the Crosslinguistic Analysis
The results for our crosslinguistic analysis can be seen in Fig. 1, with the languages in the different facets, H on the r-axis and Auhh on the y-axis. First, we note that the majority of language models are able to achieve Am. above zero, indicating that surprisal and entropy values from the models are helpful in predicting reading times regardless of their size. This is in line with previous work that has found that surprisal and entropy have psycho- metric predictive power. That being said, the sur- prisal and entropy from our worse models (which were trained on only I million words) typically do not lead to positive Âjh, which is not unexpected from models that are such poor estimates of P. (as evinced by their large cross entropy).
Finally, we also run regression model on all our data, pooled across languages. Here, in addition to the fixed effect of H. :0, include random slopes and intercepts for each language." 9 In this regres- sion we find a significant effect of H on Alh (3 = -0.0026,7 p < 0.01). Together, these results indicate that across a range of languages and model sizes, there is consistent relationship between a language model's 's quality and its predictive power.
"Dutch (8 =000077, P < 0.001). English
Turning to the relationship between cross
(8 00001, p < 0.001), Greek =00.0028 p <
entropy and Åmm. visually, it is quite clear that the majority of languages exhibit a positive relationship between language model quality and psychometric predictive power, i.c., a negative relationship between Åm. and cross entropy. To
0.001). Hebrew (3 =0.00113, p < 0.001). Italian
(0 -0.0088, p 0.001), Norwegian (B =0.00020
p < 0.001), Russian (8 =0000037, p < 0.001). Turk-
ish (3 000003,, p < 0.001), Spanish (8 =0.000,,
p < 0.01), Estonian (3 0.0008, p < 0.05) and Korean
(3 0.0009, p < 0.05).
"Our Imer call was dll ~ x-ent (x-ent lang)

---

5 Analysis of Data Leakage
0.01758
0.0172
0.01826
As mentioned in the introduction, prior studies have found that the relationship between LM quality and psychometric predictive power does hold for large, contemporary LMs (Oh and Schuler, 2023;
Delta Log Libeellhoo per perward))
0.020
0.010
Shain al., 2022). One potential concern for both of these studies is that they used models whose
0.005
training data is not publicly available, and is so large in scale, that it quite fcasibly suffers from data leakage. That is, as human reading time datasets were publicly available, models might have been trained and tested on the same materials.
0.000
0% Percent MECO Corpus in Training Data
100%
50%
Figure 2: Results of the Data Leakage Experiment: Eiror bars 95% confidence intervals averaged across languages. Average values Am. in each category are printed above the bars.
Motivation. There are number of ways in which model's surprisal estimates for leaked data could provide a poorer fit to reading times. Importantly, models are likely to underestimate surprisal on training data, and there could feasibly be difference in the degree to which high surprisal vs. low surprisal values are underestimated. Such a difference could hurt the ability to use surprisal to predict RT's when restricted to linear regressors, which are the main focus of many RT analyses since the relationship between RT and surprisal has been observed to be linear in nature (Smith and Levy, 2008, 2013; Shain al., 2022). Explicitly, since the aforemennioned underestimation might disproportionately skew a subset of surprisal estimates, linear regressors would be unable to model this altered relationship.
and then fit a mixed-effects regression model with these di, as the response variable and single categorical fixed effect indicating whether the ōm, was derived from data-leaked model or not. We run separate tests comparing between 0% and 50% and comparing between 0% and 100% leakage. We included random slopes and intercepts for each language. The degree of data leakage was not a sigmificant predictor of Em (p >> 0.05). Although our results can not be taken show, definitively, that the breakdown in the QP relationship observed by Shain et al. (2022) and Oh and Schuler (2023) is or is not due to data leakage, they suggest that this likely not the primary cause. We thus encourage researchers to explore other hypotheses about why these larger models appear to be poorer predictors of human reading behavior.
Experimental Setup. To assess whether leakage could be a confounding factors in previous evidence against the QP hypothesis, we train models with different amounts of leaked data and investigate its impact Ajh-- To do so, create versions of our full (i.e., not subsampled) training datasets in each language that include 50% or 100% of the MECO materials for that language. We then evaluate these models on MECO as before.
Conclusion
This paper investigates the QP hypothesis, i.c., the relationship between a language model's quality and its psychometric predictive power. It looks specifically at settings in which there is some evidence against the hypotheses by providing (i) a crosslinguistic assessment of its predictions and (ii) an investigation of the role of data leakage in model predictive power. Although our results demonstrate that, at small- and medium-sized data scales, the QP hypothesis holds in the large majority of lan- guages tested, our results do raise some questions. Perhaps the most pressing of these has do with our results for Finnish and German, which showed negative (albeit non-significant) correlations be- tween quality and power. Further testing in these two languages, thus, is an important next step for future research.
Results. Results for this experiment can be seen in Fig. 2, which shows the average Ajlh across languages. The average Am, for each amount of leakage is also printed above the bars. Although the version of MECO with 0% leaked data does achieve the highest Åmp, the confidence intervals overlap with each other. To confirm this statisti- cally, we first compute delta log likelihood esti- mated for each word-Ro RT pair in our corpus:
6ph lo X1, 80)) (o x:)
(5)

---

Limitations
Zongwei Zhou, Xuezhi Wang, Brennan Sacta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. PaLM: Scaling language modeling with pathways. CoRR, abs/2204.02311
One empirical limitation of this study is that we did not assess the QP hypothesis in Japanese (which is not in the MECO dataset). Negative results in that language were one of the primary motivations for this work. Assuming that the negative Japanese re- sults still hold, what should we make of this? One possibility is that the results from Japanese might be due to its writing system, which combines syl- labaries (the kana) and logosyllabic characters (the kanji). All of the scripts investigated here either use alphabets or, in the case of Korean, blends between alphabets and syllabaries. Our results strongly sug- gest that the QP hypothesis does hold across lan- guages that share common approach to writing. A next logical step would be test this hypothesis across writing systems in a more systematic way.
Thomas M. Cover and Joy A. Thomas. 2006. Ele- ments of Information Theory, second edition. Wiley- Interscience.
Andrea de Varda and Marco Marelli. 2022. The effects of surprisal across languages: Results from native and pon-native reading. In Findings of the Associa- rion for Computational Linguistics: AACI-IICNILP 2022, pages 138-144, Online only. Association for Computational Linguistics.
Andrea de Varda and Marco Marelli. 2023. Scaling in cognitive modelling: multilingual approach to human reading times. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 139- 149, Toronto, Canada. Association for Computational Linguistics.
Ethics Statement
Stefan L.. Frank and Rens Bod. 2011. Insensitivity of the human sentence- propeessiigcccigggggeee system to hierarchical structure. Psychological Science, 22(6):829-88) -344.
The MECO dataset required human participation. We refer the reader to the associated publication for their ethical guidelines in subject interactions. We do not foresee any ethical issues with our work.
Adam Goodkind and Klinton Bicknell. 2018. Predictive power of word surprisal for reading times is a linear function of language model quality. In Proceedings of the 8th Workshop on Cognitive Modeling and Com- putational Linguistics (CMCL 2018), pages 10-18. Salt Lake City, Utah. Association for Computational Linguistics.
Acknowledgements
EGW would like to acknowledge support from an ETH Postdoctoral Fellowship. CM is funded by Google PhD Fellowship. TP is funded by a Face- book PhD Fellowship.
Mandy Guo, Zihang Dai, Denny Vrandečié, and Rami Al-Rfou. 2020. Wiki-40B: Multilingual language model dataset. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 2440- 2452, Marseille, France. European Language Re- sources Association.
References
Damián E. Blasi, Joseph Henrich, Evangelia Adamou, David Kemmerer, and Asifa Majid. 2022. Over- reliance on English hinders cognitive science. Trends in Cognitive Sciences, 26(12): 153-1170.
John Hale. 2001. A probabilistic Earley parser as a psy- cholinguistic model. In Second Meeting of the North American Chapter of the Association for Computa- tional Linguistics, pages 1-8.
Aakanksha Chowdhery, Sharan Narang. Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko. Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du Ben Hutchinson, Reiner Pope. James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontack Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai, Thanumalayan Sankaranarayana Pil- lai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee,
John Hale. 2003. The information conveyed by words in sentences. Journal of Psycholinguistic Research. 3012):3
John Hale. 2016. Information -therettiat complex- ity metrics. Language and Linguistics Compass, 10(9):397-412.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford. Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark. Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Si- monyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. 2022. Training compute-optimal large language models. arXiv preprint arXiv:2203. 15556.

---

Jacob Louis Hoover, Morgan Sonderegger. Steven T. Piantadosi, and Timothy J. O'Donnell. 2022. The plausibility of sampling an algorithmic theory of sentence processing. PsyArXiv preprint.
Byung-Doh Oh and William Schuler. 2022. Entropy- and distance- based predictors from GPT-2 attention patterns predict reading times over and above GPT-2 surprisal. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9324-9334, Abu Dhabi, United Arab Emirates. Association for Computational Linguisties.
Frank Keller. 2004. The entropy rate principle as a predictor of processing effort: An evaluation against eye-tracking data. In Proceedings of the 2004 Con- ference Empirical Methods in Natural Language Processing, pages 317-324, Barcelona, Spain. Asso- ciation for Computational Linguistics.
Byung-Doh Oh and William Schuler. 2023, Why does surprisal from larger transformerrbased language models provide a poorer fit to human reading times? Transactions of the Association for Computational Linguistics, 11:36-3550.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations.
OpenAL 2023. GPT-4 technical report. 0bs//303.08774,
CORR.
Gina R. Kuperberg and T. Florian Jaeger. 2016. What do we mean by prediction in language comprchension? Language, Cognition and Neuroscience, 31(1):32- 59.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng. David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Con- ference the North American Chapter of the Associa- tion for Compulational Linguisties (Demonstrations), pages 48 18-53, Minneapolis, Minnesota. Association for Computational Linguistics.
Tatsuki Kuribayashi, Yohei Oseki, Ana Brassard, and Kentaro Inui. 2022. Context limitations make neural language models more human-like. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages (0421-10436, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Tiago Pimentel, Clara Meister, Ethan G. Wilcox, Roger Levy, and Ryan Cotterell. 2023. On the effect of anticipation on reading times. Transactions of the Association for Computational Linguistics.
Tatsuki Kuribayashi, Yohei Oseki, Takumi Ito, Ryo
Keith Rayner, Timothy Slattery, Denis Drieghe, and Simon P. Liversedge. 2011. Eye movements and word skipping during reading: Effects of word length and predictability. Journal of Experimental Psychol- ogy: Human Perception and Performance,3700 337():5514 114 528.
Yoshida, Masayuki Asahara, and Kentaro Tnui. 2021. Lower perplexity not always human like. In Pro- ceedings of the 59th Annual Meeting of the Associa- tion for Computational Linguistics and the 1 1ith Inier- national Joint Conference on Natural Language Pro- cessing (Volume 1: Long Papers), pages Online. Association for Computational Linguisties.
Cory Shain. 2019. A large-scale study of the effects of word frequency and predictability in naturalistic reading. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 1886,4094,00 Minncapolis, Minnesota. Association for Computational Linguistics.
Roger Levy. 2008 Expectation- based syntactic compre- hension. Cognition, 106(3):1 1 127-
Clara Meister, Tiago Pimentel, Thomas Clark, Ryan Cotterell, and Roger Levy. 2022. Analyzing wrap- up effects through an information-theortticc lens. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics ( Volume 2: Short Papers). pages 20-28. 28, Dublin, Ireland. Associ- ation for Computational Linguistics.
Cory Shain. 2021. CDRNN: Discovering complex dy- namics in human language processing. In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 1th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3718-3734, Online. Association for Computational Linguistics.
Clara Meister, Tiago Pimentel, Patrick Haller, Lena Jäger, Ryan Couterell, and Roger Levy. 2021. Revis- iting the uniform information density hypothesis. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, Online. Association for Computational Linguistics.
Cory Shain, Clara Meister, Tiago Pimentel, Ryan Cot- terell, and Roger Levy. 2022. Large-scale evidence for logarithmic effects of word predictability on read- ing time. PsyArXiv preprint.
Byung-Doh Oh, Christian Clark, and William Schuler. 2021. Surprisal estimators for human reading times need character models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1t International Joint Confer- ence on Natural Language Processing (Volume I: Long Papers), pages 3746-3757, Online, Association for Computational Linguistics.
Cory Shain and William Schuler. 2021. Continuous- time deconvolutional regression for psycholinguistic modeling. Cognition, 215:10
Cory Shain and William Schuler. 2022. A deep learn- ing approach to analyzing continuous-time systems. arXiv preprint 220.

---

Claude E. Shannon. 1948. mathematical theory of communication. The Bell System Technical Journal, 27(3):379-423.
statistically, we fit linear regression models with Åm, as the response variable and H/ >0 as the sole predictor variable. We find a significant effect of Hy-+ on Amh in Dutch, Estonian, Greek, Italian, Norwegian, Russian, Turkish (p < 0.001), English, Hebrew and Spanish (p < 0.05), but not in Finnish or Korean. In German, we find a positive effect of Åm on Hi- (p 0.05). As before, we also run mixed-effects regression model that tests the ef- feet cross-linguistically with random intercepts and slopes for language and log-number of training to- kens. We find a significant effect of Am on Hy-0 (p < 0.05). Overall, these results are consistent with the results in the main body of the paper.
Noam Siegelman, Sascha Schroeder, Cengiz Acartürk, Hee-Don Ahn, Svetlana Alexeeva, Simona Amenta, Raymond Bertram, Rolando Bonandrini, Marc Brys- baert, Daria Chernova, Sara Maria Da Fonseca, Nico- las Dirix, Wouter Duyck, Argyro Fella, Ram Frost, Carolina A. Gattei, Areti Kalaitzi. Nayoung Kwon, Kaidi Loo, Marco Marelli, Timothy C. Papadopou- los, Athanassios Protopapas. Satu Savo, Diego E. Shalom. Natalia Slioussar. Roni Stein. Longjiao Sui, Analí Taboh, Veronica Tønnesen, Kerem Alp Usal, and Victor Kuperman. 2022. Expanding horizons of cross-linguistoe research on reading: The multi- lingual eye-movement corpus (MECO). Behavior Research Methods, 54:2843-2863.
B Results with Entropy
Nathaniel J. Smith and Roger Levy. 2008. Optimal processing times in reading: formal model and em- pirical investigation. In Proceedings of the Cognitive Science Society, volume 30, pages 595-600 600.
Additionally, we conduct a version of our analy- sis that includes only (Rényi) entropy as a non- baseline predictor. Results are presented in Fig. 4. Again, we find that the majority of languages show negative relationship between H/-9 and Âmm. Conducting the same statistical analysis as dis- cussed in the main body of the paper, we find a sig- nificant effect of Aub on o in Dutch, Korean, Russian, Turkish (p < 0.001), English, German, Greek, Italian (p < 0.01), Norwegian, Hebrew and Spanish (p- < 0.05), but no effect in Finnish. We find a positive effect in Estonian (p < 0.001). We also run mixed-effects regression model that tests the effect cross-linguistically with random inter- cepts and slopes for language and log-number of training tokens. We find significant effect of Á on H (p < 0.05).
Nathaniel J. Smith and Roger Levy. 2013. The effect of word predictability on reading time is logarithmic. Cognition, 128(3):002-319.
Marten van Schijndel and Tal Linzen. 2018. A neural model of adaptation in reading. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4704-7716 10, Brussels, Belgium. Association for Computational Linguistics.
Marten van Schijndel and William Schuler. 2016. Ad- dressing surprisal deficiencies in reading time models. In Proceedings of the Workshop Computational Linguistics for Linguistic Complexity (CLALC), pages 32-37, Osaka, Japan. The COLING 2016 Organizing Committee.
Ethan G. Wilcox, Tiago Pimentel, Clara Meister, Ryan Cotterell, and Roger Levy. 2023. Testing the predic- tions of surprisal theory in 11 languages. Transac- tions of the Association for Computational Linguis- tics.
Ethan Gotlieb Wilcox, Jon Gauthier, Jennifer Hu, Peng Qian, and Roger Levy. 2020. On the predictive power of neural language models for human real-time com- prehension behavior. In Proceedings of the Cognitive Science Society.
A Results with Surprisal
In order to facilitate a closer comparison with pre- vious analyses, which have not looked at the effect of entropy (c.g., Goodkind and Bicknell, 2018), conduct version of our analysis with surprisal as the sole non-baseline predictor. Results are pre- sented in Fig. 3. As with the results in the main section, we find a negative relationship between Amh and o across languages. To test this trend

---

Dutch
Estonian
English
Finnish
German
055 pointor
m-- p-116
01035
Sl, polnts
uolk
polnt
025
0000
0.01
DUDSD
4100
4006
015
rook
Hebrow
Italion
K Korean
Norwegian
Detta Log Likeffood (aweroyy
oats
0w-1.07, p=0.000
Pord 71, pa0.05
2e-0 p-00001
b-dl 66, pH0d3
0.38 pe0.001
eoso
esse
eath
con
coto
eat
0o00
-ore
600
10
6
10
12
Russian
Spanish
Turkish
0.0
0.000
17..00005
010ts
Log Training
Tokens
090
000
7
8
10
10
12
6
12
Cross Entropy (on MECO)
Figure 3: Results with Surprisal Only: Error hars are 95% confidence intervals on heldnut data. Fits are lincar lines best-fit and 95% confidence intervals, Blue labels show the r- and p-value of a Pearson correlation test between cross entropy and delta log- eigllikeihhood. We find a negative correlation out of 13 languages tested.
0.90
Dutch
Estonian
English
01204
Finnish
Gemman
[- 50, p4100
96 pRInT
000080 p4000
m0,6 proco
m-B pato
0.00
Brees
H0.05
Gmp
Gire
Creek
Hebrew
Italian
Korean
Norwegian
Dolla Log Like hood (1w0s0ag
per 3.5 poloo
taara
0.07 poll
12, polnto
0.74 pol.t
tikia
1355
bosaa
eree
Broa
1251
bass
8
10
6
10
12
Sponish
Turkish
p-l0007
2.70 petli
0.015
E--0157, p=t0001
Grilo
G.00
Log Training Tokens
62m
7
40005
12
Cross Entropy (on MECO)
Figure 4: Results for Entropy Only: Target models include Rényi entropy with a 0.5 as an additional predictor. We find a negative correlation in 11 out of 13 languages tested.


Reviews:
[{"review_id": "14rSavLIB6", "reviewer": "Reviewer_Z8nG", "summary": "", "questions": "- A: Does the addition of Renyi entropy break the formal proof because the proof does not assume Renyi entropy?\n- B: How are the equations (2, 4, 6) derived?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for your review!\n\n> The reason why the correlation does not hold in Japanese, Finnish, and German remains to be discussed.\n\nWe did not have a longer discussion about language differences due to space, and because, as a short paper, investigating these crosslinguistic differences in detail was not our focus. We will, however, add a longer discussion for CR.\n\n> The terms of the equation (1) seem to be reversed.\n\nThanks for pointing this out. We had been working with negative LLH in an earlier draft of the article, and did not properly update this equation to take into account the reversed sign. We will change this for CR. Inverting these terms both in eq. 1 and 6 does not affect our proof.\n\n> Does the addition of Renyi entropy break the formal proof because the proof does not assume Renyi entropy?\n\nSee General Comment (b) above about entropy in the formal argument. \n\n> How are the equations (2, 4, 6) derived?\n\n(Eq2) is the language modeling loss, written out to highlight how it is equivalent to cross-entropy; (Eq4) is an assumption we make, namely, that the language model’s distribution is a noisy version of the true distribution; (Eq6) uses the definition of mutual information in the first line, of the delta log-likelihood on the second, and the equivalence between the entropy and negative log-likelihood (in this case $\\mathrm{H}(R | X)$ and $-\\mathrm{llh}(f_{\\phi} (\\boldsymbol{x}))$) that we would get if we could fit an optimal reading time predictor $\\mathrm{rt} = f_{\\phi} (\\boldsymbol{x})$. We will provide clearer explanations for each of these equations with the added space in the CR.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- To the best of my knowledge, this paper is the first attempt to investigate whether the correlation between model quality and psychometric predictive power (QP hypothesis) does hold cross-linguistically.", "reasons_to_reject": "- The reason why the correlation does not hold in Japanese, Finnish, and German remains to be discussed.\n- The terms of the equation (1) seem to be reversed, where llh of the nested model (baseline only) is usually subtracted from llh of the nesting model (baseline + surprisal), which may in turn break the formal proof.", "questions_to_authors": "- A: Does the addition of Renyi entropy break the formal proof because the proof does not assume Renyi entropy?\n- B: How are the equations (2, 4, 6) derived?", "paper_topic_and_main_contributions": "The main contribution of this paper is to investigate whether the correlation between model quality and psychometric predictive power (QP hypothesis) does hold cross-linguistically.", "excitement": 4, "reproducibility": 1, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- l.144: It should, instead be -> It should, instead, be\n- l.261: each languages -> each language\n- l.296: cross linguistically -> cross-linguistically"}, "extra_text": {}}, {"review_id": "CtLQHWu4fV", "reviewer": "Reviewer_6sX4", "summary": "", "questions": "Does the proof sketch apply to cases like bigram and n-gram models described above?  If so, how?  If not, why not?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review!\n\n> The paper seems motivated as a response to related work showing an inverted, positive relationship between perplexity and quality of fit, but does not test at data sizes (and presumably model sizes) where this inversion was found. My scores for soundness and excitement are based on this assumed motivation and this assumption about models sizes.\n\nSee general response (a) above.\n\n> Human sentence processing is usually thought to operate under constraints such as memory limitations which large language models do not operate under. The proof sketch either does not cover the case where some models have such constraints, or does not make it clear how these cases would be covered. For example, if H is a bigram model and theta is an n>2-gram model, then it seems to me H could with enough training have sufficient statistics for all its parameters and converge while theta would improve as n increases, thus theta would seem to provide a poorer fit to r generated from H as theta's number of parameters is increased. It would be nice to have an example like this to explain whether and how the case of memory limitations would be covered.\n\nSee general response (c) above.\n\n\n> The paper should specify the model sizes used in the experiments.\n\nWe will add this in for camera ready.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Cross-linguistic psycholinguistic experimentation is essential to testing general theories.\n\nI did not notice any issues with the methodology of the experiments.", "reasons_to_reject": "The paper seems motivated as a response to related work showing an inverted, positive relationship between perplexity and quality of fit, but does not test at data sizes (and presumably model sizes) where this inversion was found.  My scores for soundness and excitement are based on this assumed motivation and this assumption about models sizes.\n\nThe paper should specify the model sizes used in the experiments.\n\nI also was not satisfied with the proof sketch in Section 3.1.  Human sentence processing is usually thought to operate under constraints such as memory limitations which large language models do not operate under.  The proof sketch either does not cover the case where some models have such constraints, or does not make it clear how these cases would be covered.  For example, if H is a bigram model and theta is an n>2-gram model, then it seems to me H could with enough training have sufficient statistics for all its parameters and converge while theta would improve as n increases, thus theta would seem to provide a poorer fit to r generated from H as theta's number of parameters is increased.  It would be nice to have an example like this to explain whether and how the case of memory limitations would be covered.\n", "questions_to_authors": "Does the proof sketch apply to cases like bigram and n-gram models described above?  If so, how?  If not, why not?", "paper_topic_and_main_contributions": "This paper studies the relationship between surprisal from transformer models trained from scratch and fixation durations from eye-tracking data in multiple languages.  Results show a negative relationship between perplexity and quality of fit.", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tLCHlQ1d5X", "reviewer": "Reviewer_fGt4", "summary": "", "questions": "Footnote 2: I am not sure I understand your point. If all surprisal values are underestimated (say lower mean and lower variance), then we would have larger ms/bit estimates, but not necessarily worse fit to human reading times. See also work by Merlin & Toneva, 2022 (\"Language models and brain alignment: beyond word-level semantics and prediction\", https://arxiv.org/pdf/2212.00596.pdf) showing that fine-funing a model on test data increase LM-brain alignment.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review!\n\n> The paper seems motivated as a response to related work showing an inverted, positive relationship between perplexity and quality of fit, but does not test at data sizes (and presumably model sizes) where this inversion was found. My scores for soundness and excitement are based on this assumed motivation and this assumption about models sizes.\n\nSee general response (a) above.\n\n> Human sentence processing is usually thought to operate under constraints such as memory limitations which large language models do not operate under. The proof sketch either does not cover the case where some models have such constraints, or does not make it clear how these cases would be covered. For example, if H is a bigram model and theta is an n>2-gram model, then it seems to me H could with enough training have sufficient statistics for all its parameters and converge while theta would improve as n increases, thus theta would seem to provide a poorer fit to r generated from H as theta's number of parameters is increased. It would be nice to have an example like this to explain whether and how the case of memory limitations would be covered.\n\nSee general response (c) above.\n\n\n> The paper should specify the model sizes used in the experiments.\n\nWe will add this in for camera ready.\n\nResponse to Review #3:\n\nThank you for your review!\n\n> **Major [1]* My main concern regarding this paper is that it considers very small models and reduced training data scales (6 layer transformers with at most 1B training tokens); the problem with this is that the findings in the literature that are in contrast with the QP hypothesis are reported at completely different scales, and I think that everyone would agree that up until GPT-2 (small) larger/lower-perplexity models are better psychometric models. You cite Shain (2023) and Oh & Schuler (2023), but the smallest models Oh & Schuler consider are GPT-2 small / GPT-Neo 125M / OPT 125M, and Shain found that GPT-2 was better than GPT-3. So, you are not really testing whether these results hold cross-linguistically without data leakage, because you're testing at a different scale. I see that training multiple larger transformer models can be very demanding and I don't think it is necessary for this kind of study, but the framing should be adjusted and this difference with previous literature should be acknowledged.\n\nSee general response (a) above.\n\n> **Minor [1]** It would be interesting to see the results for entropy and surprisal tested separately in an appendix.\n\nSee general response (b) above.\n\n> **Minor [2]** ll 63-65 \"If Surprisal Theory holds, we would expect this power to correlate positively with language models’ quality.\". Not necessarily: this is true only (a) if surprisal theory holds, and (b) if your causal graphs on page 3 are correct (i.e., if there isn't any relationship between sθ(w_t) and sH(w_t). As far as I know, surprisal theory per se does not make any strong claim with respect to (b).\n\nWe agree with this comment. We will clarify the role played by surprisal theory in this claim for CR.\n\n> **Minor [4]** ll 289-293 Reporting only p-values without the regression coefficients and standard error is not ideal (also in line 301).\n\nThe p-values are reported from linear regression models, testing the effect of cross entropy on delta log likelihood. For CR we are happy to report the estimate of the cross entropy effect, as well as the coefficient from a correlation analysis.\n\n> **Questions Footnote 2:** I am not sure I understand your point. If all surprisal values are underestimated (say lower mean and lower variance), then we would have larger ms/bit estimates, but not necessarily worse fit to human reading times.\n\nThank you for this question. To explain our reasoning, there are a few ways in which a model trained on leaked data could provide a poorer fit to human reading times: (i) First, models could likely underestimate RTs if we fit the linking function (i.e., $f_\\phi$ in our paper’s notation) between surprisal values and RTs on non-leaked data (which is unseen and potentially higher surprisal). We agree however that in this case, if we also train the linking function on leaked data then it could be possible to recover a good model RT fit. (ii) Second, the underestimation of surprisal may not be uniform across the board—high surprisal words may be “more underestimated” than lower surprisal words—which might hurt delta log likelihood if we use a linear linking function to predict RTs. (iii) Perhaps most importantly, though, evaluating surprisals on data that wasn’t held-out from training may not just underestimate the surprisals, but also lead to hard-to-predict biases—e.g. if we estimated surprisals with an ngram model and no smoothing, the errors in estimates should be proportional to how frequent a context is, which would make it harder to learn a reading time regressor $f_\\phi$. Regarding Merlin & Toneva (2022), thanks for pointing this work to us. If we understood their paper correctly, however, (and please do correct us if we are wrong) they finetuned their languages models only on a held out portion of their data (Section 2.4, “We achieve this by finetuning the baseline model with the language modeling objective on a portion of the stimulus text that the brain recordings correspond to.”), but still run their brain analysis on words which were not seen during this finetuning process. If we are right, this would be akin to van Schijndel et al.’s (2018, https://aclanthology.org/D18-1499/) adaptive surprisal models, which are finetuned on a sentence before being used to predict the following one. We will add a discussion of this into the CR. \n\n> There are a couple of very recent papers that tackle the same exact problem you are addressing, on the same dataset. They are very recent, so it is of course reasonable that you did not include them in your previous draft, but they should definitely be cited in the amended version of the article.\n\nWe will add these in, thanks for referring us to them.\n\n> **Presentation Improvements [1]** I'd say that there is evidence that the probabilistic information that humans deploy in real time is systematically biased with respect to the \"true\" linguistic distribution. For instance, Goodkind & Bicknell (2021) (\"Local word statistics affect reading times independently of surprisal\") show that humans are sensitive to local statistics (bigram, trigrams) above and beyond surprisal estimated with language models. This increased sensitivity to local statistics could be interpreted as a systematic deviation of sH(w_t) from s(w_t), which could be reflected in sθ(w_t).\n\nSee general response (c) above.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "[1] The cross-linguistic approach is most welcome! (side note: you might want to take a look at Blasi et al. 2022 \"Over-reliance on English hinders cognitive science\" (https://www.cell.com/trends/cognitive-sciences/fulltext/S1364-6613(22)00236-4) if you want to make a stronger case on why it is important to have non-English languages in your sample).\n\n[2] I really enjoyed how elegantly the formal proof and the empirical results were combined.\n\n[3] The paper is both technically sound and theoretically rigorous. \n\n[4] The authors train their own transformer models to avoid potential effects of data leakage.", "reasons_to_reject": "Major:\n\n[1] My main concern regarding this paper is that it considers very small models and reduced training data scales (6 layer transformers with at most 1B training tokens); the problem with this is that the findings in the literature that are in contrast with the QP hypothesis are reported at completely different scales, and I think that everyone would agree that up until GPT-2 (small) larger/lower-perplexity models are better psychometric models. You cite Shain (2023) and Oh & Schuler (2023), but the *smallest* models Oh & Schuler consider are GPT-2 small / GPT-Neo 125M / OPT 125M, and Shain found that GPT-2 was better than GPT-3. So, you are not really testing whether these results hold cross-linguistically without data leakage, because you're testing at a different scale. I see that training multiple larger transformer models can be very demanding and I don't think it is necessary for this kind of study, but the framing should be adjusted and this difference with previous literature should be acknowledged. \n\nMinor:\n\n[1] It would be interesting to see the results for entropy and surprisal tested separately in an appendix. Does the negative relationship between cross-entropy and DeltaLogLik hold when considering surprisal alone? This would facilitate the dialogue with previous research on the relationship between perplexity and DeltaLogLik, which is usually focused on surprisal only.\n\n[2] ll 63-65 \"If Surprisal Theory holds, we would expect this power to correlate positively with language models’ quality.\". Not necessarily: this is true only (a) if surprisal theory holds, and (b) if your causal graphs on page 3 are correct (i.e., if there isn't any relationship between sθ(w_t) and sH(w_t). As far as I know, surprisal theory per se does not make any strong claim with respect to (b).\n\n[4] ll 289-293 Reporting only p-values without the regression coefficients and standard error is not ideal (also in line 301).", "questions_to_authors": "Footnote 2: I am not sure I understand your point. If all surprisal values are underestimated (say lower mean and lower variance), then we would have larger ms/bit estimates, but not necessarily worse fit to human reading times. See also work by Merlin & Toneva, 2022 (\"Language models and brain alignment: beyond word-level semantics and prediction\", https://arxiv.org/pdf/2212.00596.pdf) showing that fine-funing a model on test data increase LM-brain alignment.", "paper_topic_and_main_contributions": "This short paper addresses a current open issue in computational psycholinguistics, namely the relationship between the linguistic (ability to predict the next word) and the psychological accuracy (fit of surprisal / entropy values to human reading time data) of a language model. This problem is addressed by comparing the model fit of surprisal values from different transformer models trained on corpora of varying size on a multilingual eye-tracking dataset. The authors report that better language models are indeed better predictors of human reading data.", "excitement": 5, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "There are a couple of very recent papers that tackle the same exact problem you are addressing, on the same dataset. They are very recent, so it is of course reasonable that you did not include them in your previous draft, but they should definitely be cited in the amended version of the article. \n\n[1] Wilcox et al. (2023) \"Testing the Predictions of Surprisal Theory in 11 Languages.\" https://arxiv.org/abs/2307.03667\n\n    - In particular §4.3, Figure 4. They found a negative correlation (although non-significant) between perplexity and DeltaLogLik across languages.\n\n[2] de Varda & Marelli (2023) \"Scaling in Cognitive Modelling: a Multilingual Approach to Human Reading Times.\" https://aclanthology.org/2023.acl-short.14/\n\n    - They use XGLM models of different size, and found a negative relationship between parameter size and DeltaLogLik for early fixation measurements, and more nouanced results for other fixation measurements. \n\nAnother citation that I think is relevant is [3] which I think nicely complements Hao et al. (2020) (see ll 148 - 153). Not only language models that correlate more strongly with cloze estimates have stronger psychometric predictive power, but training a language model to mirror the empirical human cloze distribution while auto-regressing leads to language models that have higher psychological accuracy.\n\n[3] Eisape, Zaslavsky, & Levy (2020) \"Cloze Distillation: Improving Neural Language Models with Human Next-Word Prediction\" https://aclanthology.org/2020.conll-1.49/", "Typos_Grammar_Style_and_Presentation_Improvements": "[1] I'd say that there is evidence that the probabilistic information that humans deploy in real time is systematically biased with respect to the \"true\" linguistic distribution. For instance, Goodkind & Bicknell (2021) (\"Local word statistics affect reading times independently of surprisal\") show that humans are sensitive to local statistics (bigram, trigrams) above and beyond surprisal estimated with language models. This increased sensitivity to local statistics could be interpreted as a systematic deviation of sH(w_t) from s(w_t), which could be reflected in sθ(w_t).\n\n[2] I would at least mention that the dependent variable being fitted are gaze duration values in the main body of the text."}}]

Author rebuttal:

Meta-review: This paper conducts a large multilingual analysis of the relationship between large language model quality and its ability to predict human reading behavior. They study a diverse set of languages and find a correlation between LLM quality and ability to predict reading times (measured in an eye tracking corpus) in most languages. For a short paper, this paper makes a large contribution, and all reviewers agree that the topic is exciting. Reviewer 2 had a concern about the proof sketch; I think the authors addressed this well during the discussion period. The other reviewers flag some smaller issues, which also appear to have been addressed.
Decision: Accept-Main