Scaling in Cognitive Modelling: a Multilingual Approach to Human Reading Times
Year: 2023

Abstract:
Neural language models are increasingly valued in computational psycholinguistics, due to their ability to provide conditional probability distributions over the lexicon that are predictive of human processing times. Given the vast array of available models, it is of both theoretical and methodological importance to assess what features of a model influence its psychometric quality. In this work we focus on parameter size, showing that larger Transformerbased language models generate probabilistic estimates that are less predictive of early eyetracking measurements reflecting lexical access and early semantic integration. However, relatively bigger models show an advantage in capturing late eye-tracking measurements that reflect the full semantic and syntactic integration of a word into the current language context. Our results are supported by eye movement data in ten languages and consider four models, spanning from 564M to 4.5B parameters.

heading: 1 Introduction
text: The role of context-dependent statistical information in human language processing has received considerable attention in cognitive modelling. A solid empirical finding that has emerged from this research line is that speakers actively anticipate the upcoming linguistic material (Huettig, 2015; Staub, 2015). Indeed, behavioral and neural patterns that are diagnostic of reduced cognitive cost have been reported in response to predictable words; these emerged from the analysis of eye movements (Staub, 2015; Ehrlich and Rayner, 1981), changes in pupil size (Frank and Thompson, 2012), selfpaced reading times, (Frank and Hoeks, 2019; Fernandez Monsalve et al., 2012), ERP responses (DeLong et al., 2005; Van Berkum et al., 2005; Kwon et al., 2017), frontotemporal blood oxygenation levels (Baumgaertner et al., 2002; Dien et al., 2008), and MEG data (Takahashi et al., 2021).
Inferential theories of language comprehension argue that prediction must be an intrinsic feature of an incremental probabilistic cognitive processor (Levy, 2008; Shain et al., 2022). These accounts contend that the Kullback-Leibler (KL) divergence (i.e., relative entropy) between the probabilistic state of the processor before and after observing a given word is the cause of the processing difficulty associated with that word. It has been demonstrated that the KL divergence associated with this probability shift is mathematically equivalent to the surprisal of that word, i.e., the negative logarithm of its probability conditioned by the preceding sentence context (surprisal(wi) = − logP (wi|w1, w2 . . . wi−1); Levy, 2008). Inferential theories, which predict a logarithmic linking function between contextual predictability and cognitive cost, are supported by extensive experimental evidence in the computational psycholinguistics literature (Smith and Levy, 2008, 2013; Wilcox et al., 2020; Shain et al., 2022, but see Hoover et al., 2022; Brothers and Kuperberg, 2020).
Statistical language models developed in NLP research have been of paramount importance in the evolution of inferential theories of language comprehension. Indeed, language models are usually trained to predict the upcoming word in a corpus of naturalistic text, and thus define a conditional probability distribution that can be employed to compute word surprisal. Modern computationallyderived estimates of word predictability have been shown to perform on par (Shain et al., 2022) or even better (Hofmann et al., 2022; Michaelov et al., 2022) than predictability estimates obtained with expensive human annotation (although they fail to account for the processing demands of some specific linguistic patterns, see Arehalli et al., 2022; Van Schijndel and Linzen, 2021; Hahn et al., 2022). However, given that language models display a great amount of variation in their architectures and performances, various studies have investigated
139
which models are better suited to characterize the behavioral correlates of human sentence comprehension. Seminal work has shown that the “linguistic accuracy” of a model (i.e., its ability to accurately predict the next word) is positively related to its “psychological accuracy” (namely, the capability of a surprisal estimate to explain variance in human responses, as captured by the increase in fit in a corresponding statistical model; Goodkind and Bicknell, 2018; Wilcox et al., 2020; Merkx and Frank, 2021, but see Hao et al., 2020; Kuribayashi et al., 2021).
A recent incidental finding by Shain et al. (2022) shed doubt on such conclusion. The authors reported that the GPT-2small model substantially outperformed GPT-3 in predicting self-paced reading times and fixation patterns while having a parameter size smaller by three degrees of magnitude and displaying higher perplexity values in next-word prediction. The result, which suggests that the correlations between the linguistic and psychological accuracy of language models might not hold for very deep transformer-based architectures, has been promptly replicated with different GPT-2 variants (Oh et al., 2022; Oh and Schuler, 2022). This observation is at odds with the empirical scaling laws for neural language models (Kaplan et al., 2020), which show that the quality of a language model (both in terms of test loss and downstream performance, Hernandez et al., 2021) increases monotonically as the number of parameters increases (although see Lin et al., 2022).

heading: 2 Related work and motivation
text: Research in computational psycholinguistics has largely followed the progressive switch to the Transformer architecture that has characterized the NLP literature in the last years, with Transformerbased surprisal estimates being evaluated as predictors of processing difficulty (Wilcox et al., 2020; Hao et al., 2020; Merkx and Frank, 2021). While early studies within this research line have documented a positive relationship between the linguistic and the psychological accuracy of a model (Goodkind and Bicknell, 2018; Wilcox et al., 2020; Merkx and Frank, 2021), recent findings with decoder-only large language models have documented an opposite pattern, with larger and betterperforming pre-trained Transformers providing worse psychometric estimates than their smaller counterparts (Oh et al., 2022; Oh and Schuler,
2022).
The possibility that cognitive modelling might constitute an exception to scaling laws is intriguing, but further examination is needed to warrant such claims. All the evidence in support of this view has come from the English language alone (except from Kuribayashi et al., 2021), leaving an open question as to the cross-lingual generalizability of these findings. The English-centric approach to this problem is not surprising, since inferential approaches to language processing have been primarily supported by experimental evidence in English (Aurnhammer and Frank, 2019; Frank and Bod, 2011; Frank et al., 2015; Fernandez Monsalve et al., 2012; Wilcox et al., 2020; Goodkind and Bicknell, 2018; Smith and Levy, 2013), Dutch (Frank and Hoeks, 2019; Brouwer et al., 2010) and German (Boston et al., 2008; Brouwer et al., 2021), while empirical support from non-Germanic languages is far more limited (although see Fan and Reilly, 2020; Kuribayashi et al., 2021). To the best of our knowledge, there is only one study that provided large-scale cross-lingual evidence in support of surprisal theory (de Varda and Marelli, 2022). Indeed, both NLP (Joshi et al., 2020) and cognitive science research (Blasi et al., 2022) have long overrelied on the English language to develop language processing systems and test theories of language and cognition. This tendency can lead to hasty claims of generality, and must be mitigated with cross-linguistic research efforts challenging the universality of English-specific findings.
Another potential shortcoming of the studies that reported the inverse scaling trend is that they only considered a single eye-tracking measurement as an index of processing cost (Oh et al., 2022; Oh and Schuler, 2022). This choice reflects a common tendency within the inferential language processing framework (Aurnhammer and Frank, 2019; Goodkind and Bicknell, 2018; Smith and Levy, 2013; Wilcox et al., 2020); however, natural reading is an ability composed of multiple sub-processes characterized by different levels of complexity (see for instance Plaut et al., 1996; Coltheart et al., 2001). In principle, it is reasonable to assume that different processing stages, characterized by different degrees of complexity, might be better captured by models with varying parameter sizes, with shallow processes better modelled by (relatively) simpler networks, and complex integrative operations better characterized by more complex architectures.

heading: 3 Aims
text: The current work aims at inspecting the relationship between the linguistic and the psychological accuracy of a neural language model across languages, testing whether previous observations on inverse scaling in cognitive modelling hold across a sample of ten languages belonging to four different families. Furthermore, our study considers different eye-tracking measures that are thought to reflect different processing stages, to examine the possibility that the relationship between the psychological and linguistic accuracy of a model might vary as a function of the computational complexity of the cognitive operations being studied.

heading: 4 Methods and materials

heading: 4.1 Data
text: In this study, we considered the eye movement data from the MECO-L1 corpus (Siegelman et al., 2022), a large-scale repository of eye-tracking records covering 13 languages. Participants engaged in a naturalistic reading task, and were presented with 12 texts consisting of encyclopedic entries on a handful of topics; five of the twelve original texts were translated from English to the target languages, while the other seven were nontranslated texts on the same topics and with the same writing styles, comparable length, and similar difficulty. Data points that showed either very short first fixation durations (< 80 ms) or very long total fixation times (top 1% of the participant-specific distribution) were discarded. We analyzed three measures of eye movement behavior for each word wi, which are thought to reflect early, intermediate, and late stages of processing:
1. First fixation (FF): the time elapsed during the first fixation on wi. This measure is often assumed to reflect low-level oculomotor processes, early lexical access, and predictive processing (Demberg and Keller, 2008; Staub, 2015). 2. Gaze duration (GD): the sum of the fixations landing on wi before the gaze leaves the word for the first time. This measure is thought to be indicative of lexical access, and possibly of early syntactic and semantic integration (Inhoff and Radach, 1998; Rayner, 1998).
3. Total reading time (TT): the total amount of time spent looking at wi, including fixations returning to the word after having left it. This measure is thought to reflect full semantic integration (Radach
and Kennedy, 2013) and syntactic integration and reanalysis (Meseguer et al., 2002).

heading: 4.2 Models
text: In this study, we employed the XGLM family of auto-regressive language models (Lin et al., 2021). XGLMs are Transformer-based, decoder-only language models inspired by GPT-3 (Brown et al., 2020). We considered four pre-trained models, with 564M, 1.7B, 2.9B, and 4.5B parameters, and extracted word-by-word surprisal estimates from each of them. In the case of multi-token words, we summed the log probabilities assigned to the sub-word tokens, following the chain rule.

heading: 4.3 Analyses
text: Of the 13 languages included in the MECO dataset we had to exclude the Hebrew, Dutch, and Norwegian data, since these languages were not included in the XGLM pre-training data. Thus, our analyses were conducted in ten languages belonging to four language families (see Appendix A). On average, there were 65,450.8 available data points for each language (SD = 19,712.2). We fit 120 linear1 mixed-effects regression models (10 languages × 4 models × 3 fixation measurements), with random intercepts for participants and items. We included as linear covariates length, log-frequency, and their interaction relative to wi, wi−1, and wi−2, to account for spillover effects. Our models also included a main effect of surprisal relative to wi, wi−1, and wi−2. All the variables were standardized before being entered into the mixed-effects regression models.
To evaluate the increase in the goodness of fit due to the inclusion of surprisal as a fixed effect, we compared each model with a corresponding baseline model, which was identical except for the absence of the fixed effects of surprisal. As common practice in the literature, we calculated the difference in the log likelihood between the baseline and the experimental model (∆LogLik; Goodkind and Bicknell, 2018; Wilcox et al., 2020; Kuribayashi et al., 2021; Oh and Schuler, 2022). In the literature we have reviewed in §1, a common approach was to correlate the perplexity of a language model with the ∆LogLik obtained by adding the surprisal terms; however, perplexity values can
1Our choice of fitting linear models is supported by ample evidence showing that the functional form of the effects of log-probabilities on reading times is indeed linear (see Smith and Levy, 2008, 2013; Wilcox et al., 2020; Shain et al., 2022)
be properly compared only in the context of a fixed reference vocabulary (Wilcox et al., 2020). Technically, XGLM models produce a conditional probability distribution over the same whole vocabulary, regardless of the language of the specific text they are processing. However, the models have received strong evidence during pre-training that some subportions of the vocabulary (e.g. Cyrillic tokens) should be essentially ignored while processing text in some languages (e.g. English), thus reducing their actual reference vocabulary. Hence, while we report the perplexity-based results in Appendix B, we focused on the link between the linguistic and psychological accuracy of the models by observing how the ∆LogLik was affected by the parameter size of the model. The choice of employing parameter size as a proxy of linguistic accuracy is supported by the results in the original XGLM paper, where the authors reported better results in almost all downstream tasks with the bigger versions of the XGLM model family (Lin et al., 2021).
The code employed in this study is publicly available2.

heading: 5 Results
text: The first main finding of our study is that surprisal is a solid predictor of reading times across the languages considered, confirming the previous observation that context-dependent probabilistic processing generalizes beyond the Germanic language sample typically considered in the literature (de Varda and Marelli, 2022). The XGLM-based surprisal estimates were statistically significant in all cases when considering GD and TT, and in the vast majority of the cases when considering FF (see
2https://github.com/Andrea-de-Varda/ surprisal-across-languages
Appendix A). The increase in goodness of fit that could be attributed to surprisal is displayed in Figure 1, grouped by model type and fixation measure. Concerning FF (1a), we reported a general decrease in ∆LogLik when increasing the number of parameters, with the smallest XGLM564M variant outperforming the bigger models in terms of psychological accuracy. A similar trend can be observed in GD (1b), although the difference in psychological accuracy between XGLM564M and XGLM1.7B appears to be rather small3. The results are different when considering TT as the dependent variable (1c), as in this case the model that provided the highest average increase in goodness of fit was XGLM1.7B4.

heading: 6 Discussion
text: In this experiment, we showed that large multilingual Transformer-based models were outperformed by their smaller variants in predicting early eye movement measurements of processing difficulty. These measurements are thought to reflect predictive processes, lexical access, and early semantic integration. This result corroborates the previous claims that cognitive modelling might constitute an exception to empirical scaling laws in NLP (Oh and Schuler, 2022). However, predictability estimates computed by relatively larger variants of the same architecture – but not the largest – provided surprisal estimates that better captured late
3An anonymous reviewer rightfully noted that crosslingual variation increased in GD and TT with respect to FF; we provide a tentative explanation for this phenomenon in Appendix C.
4Note that a coherent pattern was observed when employing perplexity as an index of the language model’s linguistic accuracy (Appendix B).
eye-tracking measurements, which are thought to reflect the full semantic and syntactic integration of a word into the phrasal context. This dissociation is in line with the observation that it is not appropriate to adopt a “one-size-fits-all” approach when studying how linguistic distributional knowledge explains different cognitive processes (Wingfield and Connell, 2022). Instead, context-dependent probabilistic information derived from different neural architectures might be more apt to model certain cognitive mechanisms, depending on the computational complexity of the processes being considered.
Limitations
This work complemented previous analyses on the link between the linguistic and psychological accuracy of a neural language model by expanding the language sample to ten typologically distinct languages. However, our sample of neural language models was limited with respect to the literature focusing exclusively on English (Oh et al., 2022; Oh and Schuler, 2022; Shain et al., 2022). This problem cannot be overcome at the present state of affairs, since there are very few available massively multilingual auto-regressive language models, and the only one with sufficient coverage of our language sample was XGLM. This problem is an expression of a general difficulty in NLP to conduct experimental research on low-resource languages, due to the extreme skewness in the distribution of available resources (Joshi et al., 2020). However, we are confident that future developments in natural language engineering will support an additional test of our hypotheses with a more representative sample of models.

heading: A Effects of surprisal by language and model type
text: We report in Table 1 the regression coefficients of surprisal (as computed on the target word wi), the t statistic and the associated p-value, divided by language, number of parameters, and fixation measure considered. The surprisal estimates obtained from the four XGLM models were statistically significant predictors of processing times in all the language × model combinations when considering GD and TT, and in the vast majority of the cases when considering FF as the dependent variable. These result are overall more solid than the ones obtained by de Varda and Marelli (2022), who did not report significant partial effects of surprisal on FF and GD in some of the languages considered. The authors derived their probabilistic estimates employing mBERT, a bidirectional encoder. This finding highlights the importance of employing standard left-to-right causal language models when studying the effects of predictability on incremental sentence processing.
B Relationship between perplexity and ∆LogLik
The perplexity of a model (Eq. 1) is commonly considered as an intrinsic measure of a language model’s linguistic accuracy. The employment of perplexity as an evaluation of a multilingual language model is not free of concerns (see §4), but for completeness and consistency with the literature we also report the relationship between perplexity and ∆LogLik.
exp [ − 1 N N∑
i=1
logP (wi|w1...i−1) ]
(1)
We analyzed the relationship between perplexity and ∆LogLik by fitting three generalized additive
mixed models (GAMMs; one for each eye-tracking measure considered), with random slopes and intercepts for language. Note that the presence of by-language random effects mitigates the problem of comparing perplexity values with potentially different employed vocabularies.
The results are graphically depicted in Figure 2. In the case of FF (2a), we found a significant relationship between perplexity and ∆LogLik (EDF = 6.093, F = 3.623, p = 0.0095), which appears to be positive and (near)-linear from graphical inspection. In the case of GD (2b), we still found a significant partial effect of perplexity (EDF = 6.760, F = 4.466, p = 0.0019); however, the functional form of this relationship is far from linearity in this case, and is characterized by an initial growth in ∆LogLik with increasing perplexity, a local plateau, and an inversion of the trend in the 400-550 perplexity range. There is then a second inversion of the trend in the 500-600 perplexity range, although with high partial residuals. In the case of TT (2c), the relationship is clearly quadratic from graphical inspection, although the partial effect of perplexity is not statistically significant (EDF = 2.016, F = 2.152, p = 0.123).
Taken together, these results corroborate our observation that there is a negative relationship between the linguistic and the psychological accuracy of a model when considering the earliest fixation measurement, namely FF (§5); this relationship is less clear-cut when considering GD, and nonsignificant when considering TT. The very absence of a significant relationship between perplexity and ∆LogLik in this latter case demonstrates that the finding that smaller models outperform their overparametrized counterparts in cognitive modelling critically depends on the computational complexity of the mental processes being analyzed.

heading: C Cross-lingual variation in later measurements
text: The cross-lingual variation of our results increased with gaze duration and total reading time, in particular when considering XGLM564M; our tentative explanation for this pattern is motivated by the fact that late eye-tracking measures subsume the early ones (FF < GD < TT). XGLM564M is very effective at capturing early eye movement measurements (Figure 1a); some of the later measures are de facto equivalent to the earlier ones in some cases (e.g., if a word is only fixated once, FF, GD, and TT
will have the same value). XGLM564M might be more effective in modelling late eye tracking data in languages where these cases are more common, and less effective in languages where it is more common to refixate. This hypothesis relies on the observation in the MECO paper that refixations are more common in some languages than others (e.g., Estonian, see Siegelman et al., 2022).
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section "Limitations" (unnumbered, page 5)
7 A2. Did you discuss any potential risks of your work? There are no reasonable risks in our work.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract (unnumbered), Introduction (§1), Related work and motivation (§2), Aims (§3)
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? §4.1, §4.2
3 B1. Did you cite the creators of artifacts you used? §4.1, §4.2
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No, due to space restrictions. However, the artifacts that we employed were publicly released for research purposes.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No; however, the artifacts were employed in accordance with their intended use.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No, but the authors of the artifacts did, and we provided a reference to the original article.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? §4.1
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. §4.3
C 3 Did you run computational experiments? §4, §5
7 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? We did report the number of parameters (§4.2) but not the computational budget or the computing infrastructure as we did not train the models ourselves.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Not applicable. Left blank.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? §5
7 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? We used the defaults parameters of the transformers library.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: However, we are confident that future developments in natural language engineering will support an additional test of our hypotheses with a more representative sample of models