The Linearity of the Effect of Surprisal on Reading Times across Languages
Authors: Weijie Xu
Jason Sejin Chon
Tianran Liu
Richard Futrell
Venue: EMNLP 2023 Findings (2023)
Keywords: psycholinguistics
surprisal theory
linearity
reading time
cross-linguistic

Abstract: In psycholinguistics, surprisal theory posits that the amount of online processing effort expended by a human comprehender per word positively correlates with the surprisal of that word given its preceding context. In addition to this overall correlation, more importantly, the specific quantitative form taken by the processing effort as a function of surprisal offers insights into the underlying cognitive mechanisms of language processing. Focusing on English, previous studies have looked into the linearity of surprisal on reading times. Here, we extend the investigation by examining eyetracking corpora of seven languages: Danish, Dutch, English, German, Japanese, Mandarin, and Russian. We find evidence for superlinearity in some languages, but the results are highly sensitive to which language model is used to estimate surprisal.

The Linearity of the Effect of Surprisal on Reading Times across Languages
Weijie Xu
Jason S. Chon
University of California, Irvine weijie. xu@uci.
Stanford University
j jchonS@stanford.ede
Tianran Liu
Richard Futrell
University of Washington tianranl@uw.c edu
University of California, Irvine rfutrell@uci. edu
Abstract
and the result is highly sensitive which language model is used to estimate surprisal.
In psycholinguisties, surprisal theory posits that the amount of online processing effort ex- pended by a human comprehender per word positively correlates with the surprisal of that word given its preceding context, In addition this overall correlation, more importantly, the specific quantitative form taken by the pro- cessing effort as a function of surprisal offers insights into the underlying cognitive mecha- nisms of language processing. Focusing on English, previous studies have looked into the linearity of surprisal on reading times. Here, we extend the investigation by examining eye- tracking corpora of seven languages: Danish, Dutch, English, German, Japanese, Mandarin, and Russian. We find evidence for superlinear- ity in some languages, but results are highly sensitive to which language model is used to estimate surprisal.
2 Background
In the field of psycholinguisties, Surprisal Theory (Hale, 2001; Levy, 2008) holds that processing difficulty is cettmmatiin-thhoreele in nature. Letting Wy be the word that a comprehender is perceiving at time t, and wet be the sequence of all previous words, the surprisal Sy measures the amount of information contained in wt in context:
S. - log p (wy weet).
(1)
In general, Surprisal Theory holds that the amount of processing difficulty (for our purposes, the read- ing time, or RT) experienced by a comprehender for word Wy in context wez is given by f(Si), where f is a monotonically increasing linking function converting bits of information into milliseconds of RT.
1 Introduction
A great deal of insight into human language pro- cessing can be gleaned by studying the word-by- word processing difficulty experienced by a com- prehender who is listening to or reading language. Consequently, the timecourse of incremental lan- guage processing difficulty has formed one of the major objects of study in psycholinguistics (Se- divy, 2014, Ch. 8). Here take up the question of the functional form of the effect of surprisal (the negative log probability of each word given preceding context) on word-by-word processing difficulty, as indexed by reading time. Whereas pre- vious empirical work has addressed this question in English, we study the question in the existing reading time databases of Danish, Dutch, English, German, Japanese, Mandarin Chinese, and Russian. To preview the results, find consistent evidence for superlinearity for German and Russian; for all other languages except Mandarin, we find linear effect of surprisal, but the analysis is inconclusive about the presence additional superlinear effects
Surprisal Theory well-supported empirically by findings of a correlation between surprisal and RT, and the ability of surprisal to explain RT effeets previously attributed to other processing mecha- nisms (Levy, 2008), although it may not explain the full range of human processing difficulty (van Schijndel and Linzen, 2021; Arehalli et al., 2022). The correlation between surprisal and RT gets stronger when probabilities lower-perplexity language models (Goodkind and Bicknell, 2018; Hao al., 2020; Wilcox al., 2020), although this relationship may not uni- versal across languages (Kuribayashi et al., 2021), and breaks down for recent large models (Oh et al., 2022; Oh and Schuler, 2023), whose ability to pre- dict words is super-human (Buck al., 2022).
calculated using
The goal of this work is empirically inves- tigate the form of the linking function f across languages. Although early work did not commit to a specific form for the linking function f (Hale,

---

Language
Corpus
Genre
Tokens
Subjects
Danish
Copenhagen Corpus (Hollenstein et al., 2022)
Speeches
26,454
57
Dutch
Ghent Byetracking Corpus (GECO; Cop et al., 2017)
Novel
58,302
18
English
Dundee Corpus (Kennedy et al., 2003)
Editorials
24,679
10
German
Potsdam Sentence Corpus (Kliegl et al., 2006)
Artificial
557
273
Japanese
BCCWI-EyeTracc (Asahara et al., 2016)
News
970
24
Mandarin
Beijing Sentence Corpus (Yan et al., 2010)
Artificial
1,521
30
Russian
Russian Sentence Corpus (Laurinavichyute et al., 2019)
Artificial 892
96
Table I: Eyetracking corpora used as datasets. 'Artificial' genre refers sentences written by experimenters and presented in isolation. 'Tokens' indicates the number of text tokens (as defined using the tokenization provided by the eyetracking corpus) for which there reading time data from at least one subject after all data exclusions.
2001; Levy, 2005), subsequent theoretical and em- pirical developments have suggested that the link- ing function might be simple proportionality, that is f())
3 Methods
3.1 Reading-time data
S,, where k is scalar conver- sion factor from bits of information to milliseconds of RT (Levy, 2008; Smith and Levy, 2013). Sev- eral studies have found evidence for linearity of the linking function ff using both n-gram models and modern language models (Smith and Levy, 2013; Goodkind and Bicknell, 2018; Merkx and Frank, 2021; Aurnhammer and Frank, 2019; Wilcox et al., 2020; Shain et al., 2022).
We use the reading-time datasets described in Ta- ble There are two classes of reading-time corpus. The first is corpora of large amounts of naturalistic connected text, such as the English Dundee corpus, the Dutch GECO corpus, the Danish CopCo cor- pus, and the Japanese BCCWJ-ByeT these cor- pora contain many different tokens in different con- texts. The second class is the 'Sentence Corpora' of Mandarin, German, and Russian; these corpora contain only few hundred trificially-construce sentences which are read in isolation, but they have data from many participants.
However, the picture is complicated by theoret- ical considerations and recent empirical evidence. Theoretically, a superlinear function is favored by model of incremental language processing in which an incremental parse tree is derived from sen- tence prefix by sampling (Hoover et al., 2023), and the possibility of a superlinear linking function is related to the observation that speakers tend to communicate information at a relatively constant rate per unit time (Fenk and Fenk, 1980; Levy and Jaeger, 2007; Jaeger, 2010; Clark et al., 2023): such effects can be predicted from Surprisal Theory if processing cost is superlinear in surprisal (Smith and Levy, 2013; Levy, 2005, 2018). Empirically, Meister al. (2021) find evidence for a small su- perlinear effect of surprisal on reading times the level of sentences, and Hoover al. (2023) find superlinear effects at the level of words.
Following Smith and Levy (2013), we exam- ine first-pass gaze durations per word when this is possible based on the available eyetracking data; otherwise we examine first fixation duration. First- pass gaze duration, also called first-pass reading time and first-run dwell time, is defined the sum of the duration of all fixations on word w, starting with the first fixation on w and ending with any fixation on any other word.' !
Data exclusions. Following Smith and Levy (2013), we exclude the following datapoints from our analysis: words at the beginning or end of a line (when this could be determined words attached to punctuation and words that were skipped or have an RT of 0 ms. We also ex-
corpus);
Our work addresses the fact that croad-coverage psycholinguistic studies of surprisal have for the most part used reading time corpora of English only, and the functional form of the effect of sur- prisal on word RT in particular has only been in- vestigated outside of English one other study (Wilcox et al., 2023, to be discussed below in Sec- tion 5) to our knowledge.
For the English Dundee corpus, used the sum of FDUR values in the first pass. For Dutch, we used the WORD_FIRS: Btt__FRRSSFIXAIFII FIXATION_TIME field. For German, we the field. Mandarin, we used the fixation_dur dotooondddiioonnoo field. For Danish, we used the hord_first__pr srssst__ss_ddd field, For Japanese, we used the fpt field. For Russian, we used the IA__IRRS_DWWLL TIME field.
"This exclusion does not affect the Sentence Corpora, where words were presented without punctuation.

---

Language
Train Size
Dev. PPL
the other languages. Table 2 shows training set size and final language model perplexities.
Danish
42M
46.3
Dutch
166M
32.9
3.3 Statistical Analysis
English
1,856M
32.6
We analyze the RT data by fitting linear mixed- effects models (Baayen et al., 2008) to predict RT as a function of surprisal, controlling for word length. In order to test for a linear effect of RT, we fit sequence of models:
German
831M
31.8
Japanese 292M
52.2
Mandarin 804M
3.1
Russian
451M
32.6
Table 2: Summary of the training size and the dev set perplexity of the monolingual surprisal models trained in Wikipedia text. Training set sizes are per-token ac- cording
Mo: A regression including word length and log-tran te-trannffrmeed word frequency from Speer (2022) as a control predictor, with random in- tereepts by subject. This is the maximal con- sistently converging random effect structure (Barr et al., 2013).
trained tokenizers. The training size and dev set perplexity for Mandarin are italicized to indicate that these are per-byte, rather than per-word, reflecting the byte-level tokenization of our Mandarin language model.
My: Mo plus a linear effect of surprisal.
3. MrQ: Mz plus quadratic effect of surprisal, testing for superlinear effect beyond the lin- car effect.
clude datapoints with surprisal value greater than 20 nats or RT greater than 2000 ms.
3.2 Language models
The existence of a linear effect was determined by likelihood ratio test comparing Mo to Mr. The existence of an additional superlinear effect was tested by comparing Mr to MLQ-
We use two sets of language models to derive sur- prisal estimates. First, we use the existing mGPT model (Shliazhko et al., 2022), large multilin- gual model trained on Wikipedia text also used by Wilcox et al. (2023). Second, we train new autore- gressive language models using the GPT-2 archi- tecture (Radford et al., 2019) on text from Wiki40b dataset (Guo et al., 2020), removing title and sec- tion headers and splitting the remaining content by paragraph. We use the provided training/dev set splits. For all languages except Japanese and Mandarin, train the default GPT-2 tokenizer on the training set. For Japanese, the GPT-2 tokenizer uses excessive memory, so we use the XLNet tok- enizer instead (Yang et al., 2019). For Mandarin, we tokenize the text by byte (Xue et al., 2022), be- cause the GPT-2 tokenizer splits up the sentences in the eyetracking corpus very differently from the provided tokenization.
Spillover: In reading time data, processing slow- down due to factors such as surprisal is often de- layed, appearing on following words (Erlich and Rayner, 1983). In order to control for this effect, we adopt the standard practice in psycholinguistics: for all the variables in our models, we use not only the value of the current word, but also the value of the K previous words, where the value of the spillover window size K is determined by the fol- lowing procedure: first a control model is fit to the reading time data using only word length as a pre- dictor, and then a second model is fit using the same variables, incrementing the spillover window size by one. If the incremented model is a significantly better fit to the data than the control model by a likelihood ratio test, then the size of the spillover window is expanded. The resulting spillover win- dow sizes for the four corpora are shown in Table 3. If token does not have valid surprisal values for all previous tokens within its spillover window, it is excluded from statistical analysis. Results fol- lowing an alternative procedure where the spillover window is fixed at 2 for all corpora are shown in Appendix A.
We train language models to minimize cross- entropy loss on the training set using the huggingface library (Wolf et al., 2020). For lan- guages other than Mandarin, we continue training until either dev set loss reaches 3.5 nats, or 7 epochs through the training data have been completed. For Mandarin, we continue training until the decrease in dev set loss is flat by visual inspection; the spe- cial treatment of Mandarin is because its byte-level loss is not comparable to the token-level loss for
"The trained LMM are available at https::/hugginggaaat co/rfutrell

---

mGPT
Monolingual
Language
Spillover
ET PPL
Linear
Quadratic
ET PPL.
Linear
Quadratic
Danish
3
41.1
+
0.127
76.7
+
Dutch
3
124.5
+
295.7
+
English
2
52.0
+
p 0.935
74.9
+
German
3
37.1
+
+
186.9
+
Japanese
0
16.1
+
p - 0.354
215.9
+
p = 0.326
Mandarin
0
21.4
0.275 +
= 0.487 +
30.7
p =
0.284
p = 0.081
Russian
3
27.1
220.6
+
Table Summary of results using mGPT language model and the monolingual models trained Wikipedia text to estimate surprisal. 'Spillover' is the spillover window size determined by the procedure in Section 3.3, 'ET PPL' is the perplexity each language model evaluated on the eyetracking corpus text. Note that the perplexities across models are not directly comparable because of different tokenizations. The monolingual perplexity Mandarin is italicized to indicate that these are per-hyte, rather than per-word, reflecting the byte-level tokenization of our Mandarin language model. 'Linear' the sign of any significant (at p < 0.05) linear effect of surprisal on RT for the current token, and 'Quadratic" is the sign of any significant quadratic effect for the current token; p values are presented if the effect is not significant.
4 Results
effect: the quadratic effect of surprisal varies in significance and sign depending on the language model used.
Figure 1 shows average reading time as a func- tion of surprisal, along with linear fits, using the mGPT surprisal estimates. A strong positive re- lationship always obtains. Visually, the effect ap- pears be largely linear in the larger corpora of Danish, Dutch, and English, but the shape of the ef- fect is harder to discern for the the smaller corpora (German, Japanese, Mandarin, and Russian).
It is hard to discern a systematic difference be- tween languages showing evidence for only lin- effect and those showing possible deviations from linearity-2 consistent superlinear effect is found only in German and Russian, the two small- est datasets. The evidence for superlinearity does not seem to follow a typological pattern.
Regression results for both surprisal models are schematized in Table 3, which gives the sign of the coefficient for (lincar quadratic) surprisal in the fitted regression predicting reading time. As expected, the sign of the linear effect is always pos- itive, meaning that increasing surprisal corresponds to increasing reading time; the effect significant in all languages except Mandarin. For the mGPT model, the quadratic effect is significantly negative in Dutch, indicating sub-linearity and significantly positive in German and Russian, indicating super- linearity. The quadratic effect is not significant elsewhere. For the monolingual models, we find positive quadratic effects in all languages except Japanese and Mandarin.
Concurrent work
Concurrent to our work, Wilcox et al. (2023) have examined the linearity of the effect of surprisal on RT in 11 languages of the MECO cyctracking corpus (Siegelman et al., 2022), finding consis- tent evidence for linearity and a lack of evidence for superlinearity across all languages studied. 4 Our work complementary with this other work, differing in datasets, languages considered, and statistical methodology. Regarding data, Wilcox et al. (2023) use the MECO corpus, consisting of parallel translated text, which is more similar in size to the 'Sentence Corpora' that use for German, Mandarin, and Russian. In contrast, we use larger corpora of naturalistic Dutch, English, and Japanese. We also investigate reading times in Japanese and Mandarin, which
The full regression code and results are available at https://gi com/wei jieii--ee--e jiixu-charliee charlie/ RT.
for Danish,
4.1 Discussion
Overall, the results provide evidence for positive linear effect of surprisal on reading time across lan- guages, and inconsistent evidence for superlinear
"Another related work is de Varda and Marelli (2022. 2023). who study the existence of a surprisal effect across languages; however, they do not test the functional form of surprisal.

---

Danich
Dutch
Englah
Gorman
220
322
215
300
210
275
240
250
205
Reading Timee (mo)
250-
200
200
200
16
[apanese
Menderi
Russien
500
350
600-
400
300
400
300
300
250
200
10
20
10
20
10
20
Surprisal (bits)
Figure I: Average RT as function of the surprisal from mGPT. Surprisal is binned into 25 categories, and the mean RT within each category is shown in black with 95% confidence interval in blue. A linear fit these points is shown in red.
are not present in Wilcox et al. (2023). Reading times in these languages are especially interest- ing because they are written using orthographies that do not mark word boundaries; thus reading in these languages involves some arguably differ- ent cognitive processes than the other languages, word/morpheme segmentation must be inferred from the text.
and depends on the language model used. Deter- mining the functional form of the effect in further languages will require reading time corpora large amounts of naturalistic text and strong language models.
Limitations
The generality of our conclusions is limited by factors related to statistical methodology, the nature of the corpora used, and the nature of the language models trained.
Our work also differs from Wilcox et al. (2023) the statistical methodology. Following Smith and Levy (2013) and Wilcox et al. (2020), Wilcox et al. (2023) compare fits of linear and non-linear Generalized Additive Models when predicting read- ing times from surprisal after controlling for word length and frequency, with fixed spillover win- dow across languages. In contrast, we adopt sim- pler approach: we compare mixed-effects linear model against quadratic model, with a variable spillover window across languages.
Statistical Methodology Our analysis is limited because we adopt relatively simple statistical methodology, in three ways.
First, our handling of spillover effects, while standard in the psycholinguistic literature, does not account for the full shape of the timecourse of the effect of surprisal, which may lead to lack of power to detect effects (Shain and Schuler, 2018). More sophisticated methods for handling spillover effects, such as deconvolutional time series meth- ods (Shain, 2019; Shain et al., 2022), may yield different results. The procedure of setting fixed spillover window also limits statistical power be- cause token can only be kept in the regression if it has valid surprisal values for all previous tokens within the spillover window; the result is that many tokens must be thrown out.
Resolving the apparent tension between our re- sults, which join Hoover et al. (2023) in finding possible evidence for superlincarity, against the rc- sults suggesting pure linearity (Shain et al., 2022; Wilcox al., 2023, for example) should be focus in future research.
6 Conclusion
We conclude that the effect of surprisal on reading times across languages may be slightly superlinear, but the evidence for this superlinearity is unstable
Second, our test for superlinear effects of sur-

---

for varying a, allowing them to quantify the shape of the superlinearity (they find best overall fits with values between a 1 and 1.5). Our method can detect the existence of superlinear effect but cannot find specific exponent to characterize prisal on RT is relatively simple: we compare model with only a linear effect against a model with linear + effect. In contrast, Smith and Levy (2013) search for linearity using a method involving cubic splines, and Meister et al. (2021) fit models predicting RT as
different languages may have different inherent en- tropy rates (Bentz al., 2017).
Ethics Statement
We do not foresee ethical issues with this work.
Acknowledgements
RT = k
(2)
We thank Masayuke Asahara, Nora Hollenstein, Reinhold Kliegl, Anna Laurinavichyute, and Ming Yan for generously sharing data and answering our questions. This work was supported by NSF Grant #1947307 and an NVIDIA GPU Grant and to R.F.
Third, we use linear regression to predict read- ing times, because we want to study a linear effect of surprisal on RT in particular. However, linear regression assumes a Gaussian error distribution, which does not generally hold for human reaction times: human reaction times cannot be negative and tend to be skewed to the right, and have been char- acterized using distributions such as Log-Normal (Baayen and Milin, 2010) and Ex-Gaussian (Luce, 1986). The use of a more realistic error distribu- tion may lead to different results, especially in our critical datapoints where RT values are higher than expected from linear Surprisal Theory.
References
Suhas Arehalli, Brian Dillon, and Tal Linzen. 2022. Syntactic surprisal from neural models predicts, but underestimates, human processing difficulty from syntactic ambiguities. In Proceedings of the 26th Conference on Computational Natural Language Learning (CoNLL). pages 301-313, Abu Dhabi, United Arab Emirates (Hybrid). Association for Com- putational Linguistics.
Masayuki Asahara. Hajime Ono, and Edson T. Miyamoto. 2016. Reading-time annotations for "Bal- anced Corpus of Contemporary Written Japanese". In Proceedings of COLING 2016, the 26th Inter- national Conference Computational Linguistics: Technical Papers, pages 684-694, Osaka, Japan. The COLING 2016 Organizing Committee.
Eyetracking Corpora We use publically- available eyetracking datasets which are not matched across languages, because the datasets were developed by different researchers for different purposes at different times. As result, the corpora differ not only in language but also genre.
Christoph Aurnhammer Stefan Frank. 2019. Eval- uating information-thcoretii measures of word pre- diction in naturalistic sentence reading, Neuropsy- chologia, 134:107198.
R. Harald Baayen, D.J. Davidson, and Douglas M. Bates. 2008. Mixed-effects modeling with crossed random effects for subjects and items. Journal of Memory and Language, (444)3390-412
Language Models We only use one language model architecture (GPT), which is potential source of inaccuracy in our surprisal values: other language model architectures may deliver surprisal values that better reflect human expectations. Fur- thermore, the ET corpus texts differ in genre from the Wikipedia text corpora used to train the lan- guage models. This discrepancy may result in sur- prisal values for the ET corpus texts that do not re- flect human expectations. This factor may account for the discrepancy where we find superlinear effect in English whereas some previous work has not (for example, Wilcox al., 2020).
R. Harald Baayen and Petar Milin. 2010. Analyzing re- action times. International Journal of Psychologicul Research, 3(2):22-28.
Dale J. Barr, Roger P. Levy. Christoph Scheepers, and Harry J. Tily. 2013. Random effects structure for confirmatory hypothesis testing: Keep it maximal. Journal of Memory and Language, 68(3):255 2778
Christian Bentz, Dimitrios Alikaniotis, Michael Cysouw, and Ramon Ferrer-i-Cancho 2017. The en- tropy of w Learnability and expressivity across more than 1000 languages. Entropy, 19:275-307.
Another potential limitation is that we train all the monolingual language models the same tar- get perplexity, which may not reflect the same level of language model quality per language because
Buck, Fabien Roger, and LawrenceC. 2022. Language models seem be much better than humans next- token prediction. Less Wrong.

---

Thomas Hikaru Clark, Clara Meister, Tiago Pimentel, Michael Hahn, Ryan Cotterell, Richard Futrell, and Roger P. Levy. 2023. A Croos-LLinguistir Pressure for Uniform Information Density in Word Order. Trans- actions of the Association for Computational Linguis- tics, 1 11:1048-1000
Nora Hollenstein, Maria Barrett, and Marina Bjömsdót- tir. 2022. The Copenhagen Corpus of eye tracking recordings from natural reading of Danish texts. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 1712-1720, Mar- seille, France. European Language Resources Asso- ciation.
Uschi Cop. Nicolas Dirix, Denis Drieghe, and Wouter Duyck. 2017. Presenting GECO: An eyetracking corpus of monolingual and bilingual sentence reading. Behavior Research Methods, 49|2) 615.
Jacob Louis Hoover, Morgan Sonderegger, Steven T. Piantadosi, and Timothy J. O' D'onnelll 2023. The Plausibility of Sampling as an Algorithmic Theory of Sentence Processing. Open Mind, 7:350-391.
Andrea de Varda and Marco Marelli. 2022. The effects of surprisal across languages: Results from native and non-native reading, In Findings of the Associa- tionfot for Computational Linguistics: AACL-IJCNLP 2022, pages 138-144, Online only. Association for Computational Linguistics.
Florian Jaeger. 2010. Redundancy and reduction: Speakers manage syntactic information density. Cog- nitive Psychology, 61(1):23 62.
Alan Kennedy, Robin Hill, and Joël Pynte. 2003, The Dundee corpus. Poster presented at the 12th Euro- pean Conference on Eye Movement.
Andrea de Varda and Marco Marelli. 2023. Scaling in cognitive modelling: a multilingual approach to human reading times. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 139- 149, Toronto, Canada. Association for Computational Linguistics.
Reinhold Kliegl, Antje Nuthmann, and Ralf Engbert. 2006. Tracking the mind during reading: The influ- ence of past, present, and future words on fixation durations. Journal of Experimental Psychology: Gen- eral, 135(1):12.
Tatsuki Kuribayashi, Yohei Oseki, Takumi Ito, Ryo Yoshida, Masayuki Asahara, and Kentaro Inui. 2021. Lower perplexity is not always human-like, In Pro- ceedings the 59th Annual Meeting of the Associa- tion for Computational Linguistics and the 11th Inter- national Joini Conference on Natural Language Pro- cessing (Volume Long Papers). pages 5203-5217, Online. Association for Computational Linguistics.
Kate Erlich and Keith Rayner. 1983. Pronoun assign- ment and semantic integration during reading: Eye movements and immediacy of processing. Journal of Verbal Learning & Verbal Behavior, 22:75-87.
August Fenk and Gertrud Fenk. 1980. Konstanz im Burzzeitgediichnni--Knnaaza Konstanz im sprachlichen In- formationsflus. Zeitschrift für experimentelle and angewandte Psychologie, 400444400-
Anna K. Laurinavichyute, Irina A. Sekerina, Svetlana Alexeeva, Kristine Bagdasaryan, and Reinhold Kliegl. 2019. Russian sentence corpus: Benchmark mea- sures of eye movements in reading in Russian. Be- havior Research Methods, 51)) 1661-11887
Adam Goodkind and Klinton Bicknell. 2018. Predictive power of word surprisal for reading times a linear function of language model quality. In Proceedings of the 8th Workshop on Cognitive Modeling and Com- putational Linguistics (CMCL 2018), pages 10-18. Salt Lake City, Utah. Association for Computational Linguisties.
Roger P. Levy. 2005. Probabilistic Models of Word Or- der and Syntactic Discontinuity. Ph.D. thesis, Stan- ford University, Stanford, CA.
Mandy Guo, Zihang Dai, Denny Vrandečić, and Rami Al-Rfou. 2020. Wiki-40B: Multilingual language model dataset. In Proceedings of the Twelfth Lan- guage Resources and Evaluation Conference, pages 2440-2452, Marseille, France. European Language Resources Association.
Roger P. Levy. 2008. Expectation-based syntactic com- prehension. Cognition, 106(3): 1177.
Roger P. Levy. 2018 Communicative efficiency, uni- form information density, and the Rational Speech Act theory. In Proceedings of the 40th Annual Meet- ing of the Cognitive Science Society, page 684-689.
John Hale. 2001. prohabilistic Earley parser as psy- cholinguistic model. In Second Meeting of the North American Chapter of the Association for Computa- tional Linguistics.
Roger P. Levy and T. Florian Jaeger. 2007. Speakers optimize information density through syntactic reduc- tion. In Advances in Neural Information Processing Systems, volume 19, pages 849-856.
Yiding Hao, Simon Mendelsohn, Rachel Sterneck, Randi Martinez, and Robert Frank. 2020. Probabilis- tie predictions of people perusing: Evaluating metries of language model performance for psycholinguis- tic modeling. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pages 75-86, Online. Association for Computational Linguistics.
R. Duncan Luce. 1986. Response Times. Oxford Uni- versity Press, New York.
Clara Meister, Tiago Pimentel, Patrick Haller, Lena Jäger, Ryan Cotterell, and Roger Levy. 2021. Revisit- ing the Uniform Information Density hypothesis. Proceedings of the 2021 Conference on Empirical

---

Methods in Natural Language Processing, pages 963- 980, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Robyn Speer. 2022, rspeer/wordfreq: v3.0.
Marten van Schijndel and Tal Linzen. 2021. Single- stage prediction models do not explain the magnitude of syntactic disambiguation difficulty. Cognitive Sci- ence, 45(6):e 12988.
Danny Merkx and Stefan L. Frank. 2021. Human sen- tence processing: Recurrence or attention? In Pro- ceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pages 12-22, Online. Association for Computational Linguistics.
Ethan Gotlieb Wilcox, Jon Gauthier, Jennifer Hu, Peng Qian, and Roger P. Levy. 2020, On the predictive power of neural language models for human real- time comprehension behavior. In Proceedings of the 42nd Annual Meeting of the Cognitive Science Sociery, page 1707-1713.
Byung-Doh Oh, Christian Clark, and William Schuler. 2022. Comparison of structural parsers and neural language models as surprisal estimators. Frontiers in Artificial Intelligence, 5.
Eahan Gotlieb Wilcox, Tiago Pimentel, Clara Meister, Ryan Cotterell, and Roger P. Levy. 2023. Testing the predictions of surprisal theory in 11 languages. arXiv preprint arXiv:2307.03667.
Byung-Doh Oh and William Schuler. 2023, Why Does Surprisal From Larger Transformer-Basse Language Models Provide Poorer Fit to Human Reading Times? Transactions of the Association for Com- putational Linguistics, 11:36-350..
Thomas Wolf, Lysandre Dehut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger. Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-art natural language processing, In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 38-45, Online, Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl Blog, 1(8y:9.
Julie Sedivy. 2014. Language in Mind: An Introduction to Psycholinguistic. Oxford University Press.
Cory Shain. 2019. A large-scale study of the effects of word frequency and predictability naturalistic reading. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4086-4094, Minneapolis, Minnesota. Association for Computational Linguistics.
Linting Xue, Aditya Barua, Noah Constant, Rami Al- Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2022. ByT5: Towards a token-free future with pre-trained byte-to-byte models. Transac- rions of the Association for Computational Linguis- tics, 10:291-308
Cory Shain, Clara Meister, Tiago Pimentel, Ryan Cot- terell, and Roger Philip Levy. 2022. Large-scale evi- dence for logarithmic effects of word predictability on reading time. PsyArXiv.
Ming Yan, Reinhold Kliegl, Eike M. Richter. Antje Nuthmann, and Hua Shu. 2010. Flexible saccade- target selection in Chinese reading. The Quarterly Journal of Experimental Psychology, 6344)7005-725.
Cory Shain and William Schuler. 2018 Deconvolu- tional Time Series Regression: A technique for mod- eling temporally diffuse effects. In Proceedings of the 2018 Conference on Empirical Methods in Nat- ural Language Processing, pages 2679-2689, Brus- sels, Belgium. Association for Computational Lin- guistics.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Car- bonell, Russ Salakhutdinov, and Quoc V Le. 2019. XLNNe:: Generalized autoregressive pretraining for language understanding. In Advances in Neural In- formation Processing Systems.
Oleh Shliazhko, Alena Fenogenova, Maria Tikhonova, Vladislav Mikhailov, Anastasia Kozlova, and Tatiana Shavrina. 2022. mgpt: Few-shot learners go multilin- gual. arXiv preprint arXiv:2204.07580.
A Results with Fixed Spillover Window
Table 4 shows our main results but using a fixed spillover window of size 2 for all languages, for the mGPT and monolingual surprisal models. When using mGPT, fixing the spillover window yields significant positive linear effect of surprisal in Mandarin, eliminates the evidence for the apparent sublinear effect in Dutch, and gives evidence for superlinear effect in Japanese. When using the monolingual models, the results are the same as in the main text, but now with an apparent positive quadratic effect in Japanese.
Noam Siegelman, Sascha Schroeder, Cengiz Acartürk, Hee-Don Ahn, Svetlana Alexeeva, Simona Amenta, Raymond Bertram, Rolando Bonandrini, Marc Brys- baert, Daria Chernova, et al. 2022. Expanding hori- zons of cross-linguistic research on reading: The multilingual eye-movement corpus (meco). Behav- ior Research Methods, 54(6):2843-288 -266..3.....
Nathaniel J. Smith and Roger P. Levy. 2013. The effect of word predictability on reading time logarithmic. Cognition, 120(3)3002-39.

---

Linear
Quadratic
Language
mGPT
Monoling
mGPT
Monoling
Danish
+
n8. 0.289
+
Dutch
+
ns, P 0.101
+
English
+
ns, p 0.935
German
+
Japanese
+
Mandarin
ns, p = 0.058
ns, = 0.855
IIS = 0.424
Russian
+
Table Main results as in Table using the mGPT surprisal model with afixed spillover window of size
B Additional Figures
Figure shows reading times function of sur- prisal based the monolingual surprisal models. Figures 3 and show residual reading times after controlling for spilled-over length and frequency, as a function of surprisal, for the mGPT and mono- lingual language models respectively.

---

Darich
Dutch
Engleh
German
240
230
400
240
220
300
300
230
220
230
Reading Timee (mx)
200
200-
10
15
Japanese
Mandarie
Russiar
500-
300
400
400
270
300
240
200
230
10
15
10
15
10
15
Surprisal
Figure 2: As in Figure 1, average RT function of the surprisal from monolingual L.M.
Dutth
Engiish
Gorman
20
10
Resitial RT (ms)
-30
1b
Japanese
Manderin
Russien
100
200
100
100
474
10
10
20
30
Surprisal (bits)
Figure 3: Average residual RT controlled by word length and word frequency as a function of the surprisal from mGPT.

---

Darish
Dutch
Englen
50-
50-
23-
400
Recicial RT (mS)
-25
-10
Japanese
Mansadn
Russian
130
100
20
100
-50
10
15
10
15
10
15
Srprisal (bits)
Figure 4: Average residual RT controlled by word length and word frequency as function of the surprisal from monolingual LM.


Reviews:
[{"review_id": "V7LguMTFB9", "reviewer": "Reviewer_1TwT", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. We agree that it is possible that the \"by-language\" variation is actually by-corpus variation, although we note that we trained separate models on the same genre of text (Wikipedia) for each language. We will rewrite to de-emphasize any claim that language is the main factor influencing the shape of the surprisal effect, and to emphasize the claim of a universal surprisal effect which is linear in many cases and maybe superlinear in others.\n2. We will remove any conclusion that functional form meaningfully differs as a function of language.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The question is of interest for cognitive modeling\n- The principle cross-linguistic contribution is important and necessary for this subfield to progress\n- Data analysis is well-motivated and connects well with previous work in this domain", "reasons_to_reject": "- A key limitation concerns the different corpora that were available for the different languages. These corpora differ substantially in size and in genre (e.g. naturalistic text vs constructed sentences). These differences in genre plausibly affect how adequate GPT2 -- given the idiosyncrasies of its own training set -- serves as a cognitive estimator.  The analysis across languages also differ in other ways that may impact the results, including the specific measures available (first-pass gaze duration vs first fixation duration) and the size of the pre-target context that was used to sum \"spill-over\" surprisal values. \n\nTaken together, I found it difficult to understand where differences between \"linear\" and \"non-linear\" effects might reflect cross-linguistic differences, as opposed to idiosyncratic differences of the available corpora (like genre) or other analysis choices.  I think for this sort of work to be maximally impactful, it would be great to see a robustness analysis varying some of these parameters (e.g. corpus size and genre via sub-sampling the available materials; spill-over size etc.)\n\nA final point is that the main conclusions: that languages differ in whether the effect of surprisal is linear or super-linear, is grounded in comparing a set of \"significant\" model comparison findings against \"non-significant\" model comparisons. As presented, I don't think this inference goes through (\"the difference between significant and non-significant is not itself statistically significant\" - German & Stern 2006).  I think this could be addressed by testing for a statistical interaction of language and model-type. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper examines the shape of the function linking next-word prediction with sentence processing load as measured with reading times. The key contribution is an evaluation of this function across seven languages, where previous work has primarily focused on English. This contribution is relevant because it helps to narrow down the precise linking function between a dominant framework for cognitive modeling (surprisal theory) and human behavior.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fnIToUVGUJ", "reviewer": "Reviewer_JsrJ", "summary": "", "questions": "A) Do you think any of your results would be different if you tested off-the-shelf language models instead of training your own?\n\nB) How did you decide on the stopping criteria in Section 3.2, and why is Mandarin treated differently from the other languages?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. We will adjust the presentation in the opening section regarding Surprisal Theory. Linear surprisal was posited by Levy (2008) and supported by Smith & Levy (2013), but Hale (2001) does not specify a linking function to RT and Levy's (2005) thesis specifies a monotonically increasing linking function which may be linear or superlinear, which is the presentation we follow here. We will make the citations to the literature more precise to reflect this.\n2. Initially we experimented with using GPT-3 as the surprisal model for English, finding superlinearity. But we decided against using such models because they introduce uncontrolled and unknowable variation among the languages, in terms of how much and what kind of training data is included for each language. We trained our own models so we could control the perplexity and training data quantity and quality across languages.\n3. The stopping criteria in Section 3.2 were decided based on compute time available to us. Mandarin is treated separately because, as a byte-level model, its loss is not comparable to the other languages. A token-level model for Mandarin was not feasible due to massive disparities between the LM's tokenization and the RT corpus's tokenization.\n4. For Wilcox et al. (2023) to appear in TACL and released on the arXiv, in fact, we were not aware of this work, which was posted after the EMNLP submission deadline. Our work differs from that work in several ways and we believe the two papers are complementary:\n    - Wilcox et al. 2023 test on small \"sentence corpus\"-style datasets of all of their languages, while we use larger corpora of naturalistic text for English, Dutch, and Danish.\n    - Wilcox et al. 2023 do not include analyses of Japanese and Mandarin, which are crucially different from all the other languages in that their orthographies do not have spaces between words. Reading Japanese and Mandarin thus may involve different cognitive processes from the other languages (requiring a segmentation step), so it is interesting that we find linearity for the corpora of these languages.\n    - Wilcox et al. 2023 differ from our work in the statistical methods: their work has a fixed spillover window for all language, whereas ours is variable; their work uses a GAM as the nonlinear model, whereas we use a quadratic regression; and their work controls for frequency whereas our doesn't.\n    - As for the source of the difference in findings (they find linearity for all languages), we believe this will require extensive investigation comparing the two works which goes beyond our current submission. In the camera-ready, we will include a section specifically to compare to Wilcox et al. 2023 to make all the points above.\n5. Before the camera-ready, we will undertake some additional analysis to make our results more compatible with Wilcox et al. (2023). These will be presented in addition to the current set of results. In particular, we will fit models based on the monolingual LM's from Wilcox et al. (2023) and models controlling for frequency.\n6. We will fix all grammar and writing suggestions", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper contributes useful crosslinguistic data to an ongoing debate about the connection between surprisal and reading time. The methods seem sound and the results are clearly presented.", "reasons_to_reject": "The opening sections of the paper seem to slightly mischaracterize Surprisal Theory. As presented in Levy 2008, Surprisal Theory posits that comprehension difficulty (e.g. reading time) is proportional to surprisal -- meaning that the linking function is linear. This would mean that theories that posit e.g. a superlinear linking function do not fall under Surprisal Theory per se.", "questions_to_authors": "A) Do you think any of your results would be different if you tested off-the-shelf language models instead of training your own?\n\nB) How did you decide on the stopping criteria in Section 3.2, and why is Mandarin treated differently from the other languages?", "paper_topic_and_main_contributions": "This paper tests the relationship between surprisal and reading time on seven languages. It is well established that surprisal and reading time are correlated, but recent work has debated whether the linking function is sublinear, linear, or superlinear. Much of the debate has focused on English data; this study expands the investigation to several other languages. Results suggest a slightly superlinear relationship between surprisal and RT.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Andrea de Varda and Marco Marelli. 2022. The Effects of Surprisal across Languages: Results from Native and Non-native Reading. In Findings of the Association for Computational Linguistics: AACL-IJCNLP 2022, pages 138–144, Online only. Association for Computational Linguistics.", "Typos_Grammar_Style_and_Presentation_Improvements": "A) Wilcox et al. have a very recently submitted preprint which addresses similar questions: https://arxiv.org/abs/2307.03667\n\nB) The Section 2 citation to Hikaru Clark et al. 2023 should just be Clark et al. 2023"}}, {"review_id": "7hdjUtyBDe", "reviewer": "Reviewer_fm4s", "summary": "", "questions": "(a) (135) It would be very helpful to see which languages are first fixation and which are gaze durations. It’s a little difficult to tell from Footnote 1. Relatedly, was there a relationship between FF / GD in the linear / non-linearity of the results? How do the authors think the different metrics changed or impacted their results? I would appreciate more discussion about this.\n\n(b) (147) Could the authors provide a justification for this? Is this something that has been done in the previous literature?\n\n(c) (Fig 1) One concern I have about the figure is that plotting the raw surprisal vs. RT data does not take into account things like the effect of word length on reading times. In previous studies such as Smith and Levy (2013) and Hoover et al., (2022) figures were generated by first fitting a model with both surprisal and baseline predictors, and then sampling from that model to show the isolated effect of surprisal alone. I think doing something like this could help make the implications of the data clearer.\n\n(d) (257) I don’t follow this logic. Shouldn’t this mean that the non-linear effects of reanalysis are more apparent in smaller datasets, where a single non-linearity can skew the overall pattern in the data even more?\n", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. We believe that controlling for frequency in these studies is not fully justified: there is good evidence that frequency effects are due only to the correlation between frequency and surprisal (see for example Shain, 2019, NAACL). Nevertheless, in the camera-ready we will include results (in addition to the current results) that include log token frequency as a control.\n2. Comparing log likelihoods on test data is standard practice in experimental psychology, because a held-out dataset would usually be too small to get a good estimate of the log likelihood. Nevertheless, we will include a heldout analysis (in addition to the current analysis and an analysis using the Wilcox et al 2023 language models). \n3. For question (a)(135), we will add this information to Table 2. We use first-pass gaze duration whenever possible. All languages except Dutch and German are first-pass gaze duration. Dutch and German are first fixation.\n4. For question (b)(147), the data exclusions follow previous literature.\n5. For question (c)(Fig 1), in addition to our current figure, we will include another figure showing residual RT after controlling for word length (and word length spillover). We believe there is value in the current figure, which shows only RT as a function of surprisal, because it is a very simple representation of the data that shows a clearly visually near-linear effect.\n6. For question (d)(257), we will remove this argument from the camera-ready.\n7. See our response to Reviewer 2 with respect to Wilcox (2023).\n8. We will fix all grammar and writing suggestions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(a) The authors address a timely topic that has been the focus of recent discussion in the literature\n\n(b) The authors go beyond previous monolingual analyses, which is a valuable contribution\n\n(c) The paper is well written and clear\n\n(d) Although the authors present their handling of spillover has a limitation (I see their point) their method allows for variable spillover between datasets. This is better than many recent papers which set a fixed sized spillover window regardless of dataset.\n", "reasons_to_reject": "(a) Previous studies have demonstrated an effect of unigram surprisal (i.e. frequency) on reading times. Frequency is absent from this paper and not used in the baseline model M0. I’m worried that the overall picture of results could change if frequency was added in as an additional predictor. (The authors do mention that word length with random by-participant intercepts was the maximal consistently random effects structure – does this mean that they tried random slopes, or that they tried random slopes plus also other main effects, such as frequency?)\n\n(b) The likelihood ratio tests were computed on the log likelihoods of the training data, correct? I think the paper could be made much stronger if the tests were conducted on a held-out portion of the dataset. Especially as models grow more complex, testing on held-out data grows crucial to avoid charges of overfitting. (My apologies if you are already doing this – it was not clear to me from the text!)\n", "questions_to_authors": "(a) (135) It would be very helpful to see which languages are first fixation and which are gaze durations. It’s a little difficult to tell from Footnote 1. Relatedly, was there a relationship between FF / GD in the linear / non-linearity of the results? How do the authors think the different metrics changed or impacted their results? I would appreciate more discussion about this.\n\n(b) (147) Could the authors provide a justification for this? Is this something that has been done in the previous literature?\n\n(c) (Fig 1) One concern I have about the figure is that plotting the raw surprisal vs. RT data does not take into account things like the effect of word length on reading times. In previous studies such as Smith and Levy (2013) and Hoover et al., (2022) figures were generated by first fitting a model with both surprisal and baseline predictors, and then sampling from that model to show the isolated effect of surprisal alone. I think doing something like this could help make the implications of the data clearer.\n\n(d) (257) I don’t follow this logic. Shouldn’t this mean that the non-linear effects of reanalysis are more apparent in smaller datasets, where a single non-linearity can skew the overall pattern in the data even more?\n", "paper_topic_and_main_contributions": "Summary: The authors investigate the functional form of the surprisal / reading time relationship across languages, honing in on whether the link is linear or super linear, which has been a topic of some debate in recent literature. Previous work has addressed this question either through visual inspection of the surprisal / RT function or through linear vs. nonlinear model comparison. The authors employ the latter, albeit a slightly different method than has been used in previous studies. They find that about half the languages tested demonstrate superlinear surprisal / RT effects, and that these tend to be for languages with larger datasets. Overall, I think this paper contributes to the current discussion and that the community will benefit from its inclusion in EMNLP. However, I do have some concerns about the statistical tests used.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "(a) I’m sure the authors are aware of Wilcox et al., “Testing the Predictions of Surprisal Theory in 11 Languages” but just in case you are not, then this paper was arXived recently. It reaches many of the same conclusions, but some different ones. Discussion of the differences – both in terms of methods and conclusions – will be essential in the final version of this paper.\n\n(b) De Varda, Andrea, and Marco Marelli. \"Scaling in Cognitive Modelling: a Multilingual Approach to Human Reading Times.\" Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 2023.\n\n(c) Andrea de Varda and Marco Marelli. 2022. The effects of surprisal across languages: Results from native and non-native reading. In Findings of the Association for Computational Linguistics: AACL-IJCNLP 2022, pages 138–144, Online only. Association for Computational Linguistics.\n", "Typos_Grammar_Style_and_Presentation_Improvements": "(a) (37) “Whereas previous empirical work has addressed this question in English” → I would soften this claim a bit given the recent multilingual work that’s been appearing in the literature. This paper does include new languages, though, so that is worth pointing out up top!\n\n(b) (96) I would mention what a sampling-based model is, briefly, here.\n"}}]

Author rebuttal:

Meta-review: # Overview

This paper investigates the functional form of the surprisal–RT relationship in seven languages. Specifically, it uses likelihood ratio tests to compare linear mixed effect models with: only baseline predictors, baseline + surprisal, baseline + surprisal + surprisal squared. It finds that surprisal helps predict RTs in the 7 studied languages. It also finds that surprisal squared further helps predict RTs in 3 of these languages.

# Meta-review

The three reviewers and I agree this paper tackles an important question, and that its cross-linguistic contribution is important and timely. Further, its methods and results are clearly presented.

The reviewers, however, point out two important limitations in this paper. First, the used crosslinguistic corpora differ substantially in size, genre, and used eyetracking measure; this limits the extent to which conclusions can be drawn. Second, the analysis did not control for frequency effects. (I highlight two more methodological issues below.) 

The authors already promised to run an extra analysis with frequency effects for CR (if accepted), and to de-emphasize any claim that language is the main factor influencing the shape of the surprisal effect. I would also suggest they extend the "Eyetracking corpora" paragraph in their limitations section to highlight the issues pointed out by reviewer 1TwT.

# Detailed Comments

While I enjoyed reading this paper, and mostly agree with the three reviewers about its positive aspects, I highlight a few methodological issues below. I believe fixing those could help strengthen this paper.

* **Spillover effects**: While one of the reviewers highlights the way in which this effect was computed as a reason to accept, I think it's actually negative. Specifically, the authors seem to choose spillover effect windows based on likelihood ratio tests using models which only had access to word length. If a spillover effect is significant for surprisal, but not for word length, it will thus not be included in the model. This might have impacted the analysis on Japanese, which at the moment uses a spillover window of 0. There seems to be a large enough number of tokens in most analysed languages for spillover effects to be set to zero by the model fitting procedure itself (if they are insignificant), so I would recommend rerunning experiments (at least for Japanese) with a fixed spillover effect window.

* **Model training**: Models were trained for 7 epochs or until a validation loss of 3.5 nats was reached. Given that not all languages are "equally hard" to language-model (Cotterell et al., 2018; Mielke et al., 2019), stopping the training early (instead of training until convergence) may unfairly harm models in some languages.


* **Frequency effects**: This paper does not control for frequency effects as a baseline feature, using Shain (2019) as a justification. A more recent work by the same author (Shain, 2023) reaches the opposite conclusion. This new paper, however, only came out *after* the EMNLP submission deadline.

The authors did already promise to add an analysis including frequency effects for camera ready, in case the paper is accepted. I would personally also appreciate seeing an analysis using constant spillover effect windows (specially for Japanese), and, if possible, models trained to convergence.

As an aside, in one of their responses, the authors state that Wilcox et al. (2023) tested on small "sentence corpus"-style datasets. I believe Wilcox et al. (2023) used the MECO dataset (similarly to de Varda et al. 2022; 2023), which has paragraph-level entries.

# References

Cotterell et al. (2018). Are All Languages Equally Hard to Language-Model?

Mielke et al. (2019). What Kind of Language Is Hard to Language-Model?

Shain (2019). A large-scale study of the effects of word frequency and predictability in naturalistic reading

Shain (2023). Word Frequency and Predictability Dissociate in Naturalistic Reading
Decision: Accept-Findings