Language models emulate certain cognitive profiles: An investigation of how predictability measures interact with individual differences (2024)

Abstract: To date, most investigations on surprisal and entropy effects in reading have been conducted on the group level, disregarding individual differences. In this work, we revisit the predictive power of surprisal and entropy measures estimated from a range of language models (LMs) on data of human reading times as a measure of processing effort by incorporating information of language users’ cognitive capacities. To do so, we assess the predictive power of surprisal and entropy estimated from generative LMs on reading data obtained from individuals who also completed a wide range of psychometric tests. Specifically, we investigate if modulating surprisal and entropy relative to cognitive scores increases prediction accuracy of reading times, and we examine whether LMs exhibit systematic biases in the prediction of reading times for cognitively highor low-performing groups, revealing what type of psycholinguistic subject a given LM emulates. Our study finds that in most cases, incorporating cognitive capacities increases predictive power of surprisal and entropy on reading times, and that generally, high performance in the psychometric tests is associated with lower sensitivity to predictability effects. Finally, our results suggest that the analyzed LMs emulate readers with lower verbal intelligence, suggesting that for a given target group (i.e., individuals with high verbal intelligence), these LMs provide less accurate predictability estimates.1

heading: 1 Introduction
text: Human language comprehension and, by extension, human reading is incremental in nature: humans process words sequentially (Rayner and Clifton Jr, 2009), and different words in varying contexts impose different amounts of cognitive processing effort (Rayner, 1998). Similarly, language models’ conditional probability distributions assign differ-
1Code is available at https://github.com/DiLi-Lab/ LM-cog-profiles
ent probabilities for potential continuations for a given prefix. The relationship between cognitive effort and predictability measures derived from LMs’ probability distribution was operationalized by surprisal theory (Hale, 2001; Levy, 2008). Since then, a large body of research has investigated the exact nature of the relationship between surprisal and human processing effort, such as determining appropriate linking functions (Meister et al., 2021; Shain et al., 2024), or its manifestation in different languages (Wilcox et al., 2023a; Jäger et al., 2015; Kuribayashi et al., 2021, i.a.). Moreover, it has been shown repeatedly that both the quality of a model from which surprisal is extracted as well as the amount of data a model is trained on correlate with the model’s psychometric predictive power2, i.e., its ability to predict human behavioral processing data (Frank and Bod, 2011; Fossum and Levy, 2012; Goodkind and Bicknell, 2018a), albeit
2This hypothesis has been referred to as quality-power hypothesis by Wilcox et al. (2023a), cf. their introduction for a comprehensive summary.
7878
only to a certain extent (Shain et al., 2024; Oh and Schuler, 2023b,a). So far, most studies have tested these predictions on the group-level, neglecting individual cognitive differences that might influence readers’ capacities to make predictions about upcoming material. However, an increasing body of research has demonstrated that an individual’s characteristics do play an important role in human language processing (e.g., Estes, 1956; Daneman and Carpenter, 1980; Levinson, 2012; Van Dyke et al., 2014). Apart from surprisal, it has been suggested that reading processes are not purely responsive, which is mirrored in the surprisal effect of a word on a reader, but also anticipatory: readers make implicit assumptions about future words and allocate time to process it in advance. This anticipatory effect, which is reflected in the reading behavior, is induced by a reader’s expectation about a word’s surprisal and is operationalized as a word’s contextual entropy (Linzen and Jaeger, 2016; van Schijndel and Schuler, 2017; Cevoli et al., 2022; Pimentel et al., 2023).
In this work, we revisit the relationship between predictability measures (surprisal and contextual entropy) and data of human processing effort in consideration of language users’ individual cognitive differences. More specifically, we assess the predictive power (PP) of entropy, surprisal, as well as their interactions with cognitive measures, as predictors of human reading times in linear-mixed models. After establishing the baseline predictive power of surprisal and entropy on our German reading data (HB), we investigate the following novel hypotheses:
H1: Modulating surprisal and entropy effects relative to individual cognitive capacities improves their predictive power on reading times on unseen data. H2: Individuals with higher cognitive capacity rely less on predictive processing strategies, and hence exhibit lower surprisal or entropy effects. H3: LMs are better at predicting reading times for certain cognitive profiles.
To address these hypotheses, we utilize the Individual Differences Corpus (InDiCo; Haller et al., 2023), which contains both reading data and scores of a comprehensive psychometric assessment targeting various cognitive capacities, including verbal and non-verbal working memory, verbal and non-verbal cognitive control, verbal and non-verbal
intelligence and reading fluency. We deploy five pre-trained generative LMs from three languagefamilies—GPT-2 base and large, Llama 2 7B and 13B, and Mixtral—to estimate both surprisal and contextual entropy and quantify their predictive power by including them as predictors in linear regressors, which are fitted to predict by-word reading times from InDiCo. If the regressors’ loglikelihood improves after including these predictors and their interaction with the psychometric scores, we take this as corroboration of their predictive power.
We find that adding interaction terms between predictability measures (surprisal and entropy) and most cognitive scores significantly improves the quality of reading time predictions, and that in general, individuals with higher cognitive capacities exhibit smaller predictability effects. Lastly, there is evidence that LMs’ abilities to predict reading times vary between high- and low-performing individuals within certain cognitive capacities. Specifically, all tested models emulate the processing behaviour of individuals with low verbal intelligence.
Our work is a first step towards investigating i) the differences in surprisal and entropy effects across different cognitive profiles, and ii) what type of cognitive biases might be inherent in the way LMs process language.

heading: 2 Related work

heading: 2.1 Surprisal and predictive power
text: Surprisal is a measure of predictability of a word in its context and has shown to be proportional to cognitive effort in human sentence processing (Hale, 2001; Levy, 2008). It is quantified as the negative log probability of a word given its preceding context. Since the formalization of surprisal theory, many studies have corroborated its linear relationship with reading times (Demberg and Keller, 2008; Shain, 2021; Hoover et al., 2023; Pimentel et al., 2023), not just in English but also across languages (Pimentel et al., 2021; Wilcox et al., 2023a,b; de Varda and Marelli, 2022; Jäger et al., 2015; Kuribayashi et al., 2021). Moreover, researchers have investigated the degree to which surprisal is predictive of human reading times (i.e., assessing the predictive power (PP) of surprisal on human reading times) deploying different LMs. Wilcox et al. (2020) found that the better an LM’s next-word expectation (i.e., the lower its perplexity), the higher its PP. Along the same line, Wilcox
et al. (2023a) demonstrated that an increasing LM quality, quantified by decreasing cross-entropy during training, leads to surprisal values that better predict RTs. Similarly, Goodkind and Bicknell (2018b) showed that the PP of surprisal increases linearly with the quality of LMs. This finding has since been refuted by Oh and Schuler (2023b) who revealed that large models, despite lower perplexity, provide worse PP of RTs. Oh and Schuler (2023a) further demonstrated that LMs provide the best fit to RTs when trained on around 2 billion tokens; beyond that point, additional training data causes the PP to decrease again.

heading: 2.2 Contextual entropy and predictive power
text: Linzen and Jaeger (2016) examined how sentence processing is affected by readers’ uncertainty about the predictions they make during processing. They found that what they term single-step entropy (contextual entropy, cf.§3) does not affect RTs. However, they computed single-step entropy only over upcoming constituents based on verb subcategorization frames; it was later shown that entropy is indeed predictive of RTs (van Schijndel and Schuler, 2017). Cevoli et al. (2022) looked at the interaction of surprisal and entropy when predicting RTs: the impact of surprisal on reading behavior should vary as a function of entropy, such that surprising words inflict particularly high processing load when entropy is low. Wilcox et al. (2023b) examined whether contextual entropy is predictive of reading times and discovered that adding entropy as additional predictor (while keeping surprisal) increases PP, while replacing surprisal with entropy leads to a decrease in PP. However, Pimentel et al. (2023) also showed that using contextual entropy as a predictor in a linear-mixed model can be as good as surprisal when analyzing anticipatory effects reflected in word skipping rates, as opposed to responsive effects captured by gaze duration, for instance.

heading: 2.3 Individual differences in sentence processing
text: Theories of sentence processing generally assume that the cognitive mechanisms involved in language processing are qualitatively identical across speakers. However, this perspective has been challenged, with evidence emerging that differences in cognitive abilities among language users do indeed have a significant impact on processing (Vuong and Martin, 2014; Nicenboim et al., 2015; Farmer et al.,
2017, i.a.). For instance, Kuperman and Van Dyke (2011) demonstrated that measures related to cognitive control interact with word length and lexical frequency effects on fixation times, and Nicenboim et al. (2015) showed that readers ranking lower in working-memory tests exhibit more regressive saccades in regions with high memory load.
Several studies have also investigated individual differences in surprisal effects, in particular in the realm of native and non-native reading (Berzak and Levy, 2023; Schneider et al., 2023). For instance, Berzak and Levy (2023) demonstrated that higher L2 proficiency is associated with increased sensitivity to a word’s predictability in context (surprisal). Moreover, Škrjanec et al. (2023) showed that specialized surprisal from domain-adapted LMs improves reading-time predictions for expert readers.

heading: 3 Methods
text: Surprisal. Given a vocabulary Σ and an augmented vocabulary Σ̄ = Σ∪{EOS}, which contains a special EOS (end-of-sentence) token, the surprisal (Shannon, 1948) of a given sequence is defined as
s(un) def = − log p(un | u<n), (1)
where p(· | u<n) is the true distribution over words u ∈ Σ̄ in context u<n. In other words, surprisal of a word is the negative log-probability conditioned on its left context.
Contextual entropy. The contextual entropy of a Σ̄-valued random variable Un at index n is the expected value of its surprisal, formalized as
H(Un | U<n = u<n) def= Eu∼p(·|u<n) [sn(u)] = − ∑
u∈Σ̄ p(u|u<n) log2 p(u|u<n). (2)
It is a specific version of the Shannon entropy H(U) def = −∑u∈U p(u) log p(u) that is conditioned on the left context of U (Shannon, 1948). As we do not have access to the true distribution p(· | u<n), we approximate both measures using an auto-regressive language model pθ.

heading: 3.1 Assessing predictive power
text: We utilize linear-mixed models (LMMs) M to predict a reading time measure yij , obtained from a subject j on word i, from a set of standardized word-level and subject-level predictors xij , i.e., M : xij 7→ yij .
For our analyses, we want to quantify the predictive power of a given predictor of interest xq
(e.g., surprisal). To do so, we first define a baseline model Mb : xbij 7→ yij that includes a set of baseline predictors xbij , and a target model Mt : xbij ⊕ xqij 7→ yij that additionally includes the predictor of interest xqij , where ⊕ represents the concatenation of two sets of predictors. Following previous work (Wilcox et al., 2020; Meister et al., 2021; Wilcox et al., 2023a; Pimentel et al., 2023, i.a.), we operationalize the predictive power as the mean difference in log-likelihood (∆LL) between the target and the baseline model, i.e.,
∆LL = 1
IJ
[ I∑
i=1
J∑
j=1
logMt(yij | xbij ⊕ xqij)
− I∑
i=1
J∑
j=1
logMb(yij | xbij) ] , (3)
where I is the number of words and J is the number of subjects. To avoid overfitting, we perform 10- fold cross validation. A positive ∆LL indicates a better fit of the target model to the data.

heading: 4 Experiments
text: Data
We employ German reading time data from InDiCo (Haller et al., 2023). This corpus contains eye-tracking-while-reading and self-paced-reading data from 61 native German speakers, collected across four experimental sessions, alongside a comprehensive battery of individual psychometric scores in four cognitive domains: cognitive control, working memory, intelligence, and reading fluency.3 For our analyses, we use the standardized scores of 13 psychometric tests. Following previous work (Wilcox et al., 2023b, i.a.), we employ first-pass reading time (FPRT), also referred to as gaze duration: the sum of all fixations on a word when fixating it for the first time–as a proxy for processing load. Whereas total fixation duration can incorporate words from the right context due to regressive saccades, FPRT most strongly reflects the initial processing difficulty.4 Given that in our study, we only deploy auto-regressive LMs (cf.§4), FPRTs are also more in line with the fact that these models only have access to a word’s left context.
3For a detailed description of the tests, see Appendix B. 4Contrary to previous work, we do not set the reading time for words that were skipped to zero, but instead exclude them from the analysis. See limitations and caveats in Pimentel et al. (2023) on the influence of skipped words on surprisal estimation.
Predictors Word-level predictors. To extract surprisal and contextual entropy estimates, we deploy the German versions of five pretrained transformer-based LMs of different families and sizes, namely GPT-2 base and large (Radford et al., 2019), Llama 2 7B and 13B (Touvron et al., 2023), and Mixtral (Jiang et al., 2024). For details, see Appendix A.1. Crucially, we only consider auto-regressive LMs, as they most closely align with the incremental nature of human language comprehension (Hale, 2006; Rayner and Clifton Jr, 2009).
Since LMs employ tokenizers which split whitespace separated words into sub-word tokens (Sennrich et al., 2016; Song et al., 2021), word-level surprisal is computed by summing up the surprisal values of the sub-word tokens, which is equivalent to computing the surprisal of the joint distribution of sub-word tokens. Similarly, to obtain the wordlevel contextual entropy, we use the sum of the sub-word token-level contextual entropy values as proxy for the joint entropy of the sub-word tokens’ distributions.5
We further include lexical frequency and word length in our analyses since they are known to have an impact on human reading behavior. Lemma frequencies were extracted from dlexDB (Heister et al., 2011), based on the reference corpus underlying the Digital Dictionary of the German Language (DWDS; Berlin-Brandenburgische Akademie der Wissenschaften, 2016). Word length is defined as the number of characters including punctuation. Henceforth, we denote the word-level predictors surprisal si, contextual entropy hi, log-lemma frequency fi, and word length li for a word i.
Psychometric scores. The psychometric assessment in InDiCo includes a total of 13 tests targeting different cognitive domains such as verbal and nonverbal working memory, cognitive control and intelligence, as well as reading fluency. A list of tests and their abbreviations can be found in Appendix B. For all test scores, higher scores originally indicate higher performance except for the Stroop Reaction Time Effect (Stroop) and the Simon Reaction Time Effect (Simon). In order to facilitate interpretability, we take the negative values of these scores such that a high value indicates high cognitive control. We standardize all scores in order to facilitate comparisons between tests. We denote the score of a
5For details on pooling of surprisal and entropy, see Appendix A.2.
given psychometric test c for subject j.

heading: 4.1 Baseline analyses (HB)
text: To corroborate results from previous work, we first assess the predictive power of entropy and surprisal in general, not taking into account individual psychometric scores. We define a baseline model Mb0 with predictors xb0i including the word-level predictors word length li, log-lemma frequency fi, a global intercept β0, and an additional random bysubject intercept β0j , i.e.,
Mb0 : yij ∼ β0 + β0j + β1 li + β2 fi, (4)
where yij refers to the log-transformed first-pass reading time6 of subject j for the ith word in the stimulus corpus across all texts and following a log-normal distribution. The target models Mts0 and Mth0 solely include an additional surprisal or entropy term, i.e., si or hi.
Results. As depicted in Figure 1, surprisal and contextual entropy exhibit significant predictive power (PP), albeit consistently lower for entropy. For GPT-2 base and large, adding both surprisal
6First-pass reading time denotes the sum of all fixation durations on a given word during its first pass (i.e., when reading it for the first time).
and contextual entropy as predictors increases the PP; for the other models, the combined version yields the same PP as using surprisal alone. Across models, GPT-2 base has the highest PP, with PP decreasing as model size increases.

heading: 4.2 Assessing the predictive power of interactions between surprisal/entropy and psychometric scores (H1)
text: To examine whether an interaction between cognitive scores and surprisal or entropy leads to an increase in predictive power on reading times, we define a baseline model Mb1 with predictors xb1ij including the word-level predictors li, fi, si, hi, and the subject-level predictor cj denoting the test score of a specific psychometric test (e.g., wordreading fluency) obtained for subject j, and again a by-subject intercept β0j , i.e.,
Mb1 : yij ∼ β0 + β0j + β1 li + β2 fi+ β3 si + β4 hi + β5 cj
(5)
To assess whether allowing surprisal or entropy to be modulated by specific cognitive profiles— operationalized in terms of the individual psychometric measures—improves the prediction of reading time, we define target models Mts1 and Mth1 that include an additional interaction term between
either surprisal or entropy and a given psychometric score cj (e.g., word-reading fluency score) obtained for subject j, xq1ij ∈ {si · cj , hi · cj}:
Mt1 : yij ∼ β0 + β0j + β1 li + β2 fi+ β3 si + β4 hi + β5 cj + β6 x q1 ij
(6)
A positive ∆LL between the target and the baseline model indicates that including the participant’s score of a given psychometric test improves the prediction on the held-out test data. We run paired permutation tests using the R library broman to establish whether a given ∆LL is significantly different from 0 at α = .05.
Results. Figure 2 shows the ∆LL across all psychometric tests and models. Overall, we see that the interaction terms between surprisal/entropy and most psychometric scores lead to significant increases in PP, except for Stroop, non-verbal RIAS and spatial short-term memory. Notably, PP is not significant (or extremely small) for these three scores across all models. Additionally, there are notable differences among different cognitive domains with respect to predictive power: modulating surprisal with scores targeting reading fluency or working-memory span yields the highest predictive power, followed by verbal intelligence scores. Scores targeting cognitive control show the lowest PP. The pattern observed in Figure 1 showing that increasing model size is associated with decreasing
predictive power is visible here as well, but only for surprisal, not for entropy. Overall, interactions with surprisal extracted from the GPT-2 family have the highest PP. Conversely, interactions with GPT-2 based entropy have the lowest PP.

heading: 4.3 Assessing the magnitude of the interaction term coefficients (H2)
text: To determine how specifically the surprisal and entropy effects are modulated by the psychometric scores, we run the target models Mts1 and Mth1 on the entire dataset and examine the effect sizes (coefficients) of the interaction term between the scores and the surprisal and entropy estimates, β6. The coefficient of the interaction term indicates to what degree the fixed effect surprisal or entropy term is adjusted, relative to a subject’s individual psychometric score, or, in other words, if individuals with a given cognitive profile are more sensitive to predictability effects. For instance, a positive coefficient for predictor cj · sj indicates that subjects with a higher score exhibit a stronger effect of surprisal, i.e. are more sensitive to a word’s predictability, while a negative coefficient implies that subjects with a lower score exhibit a higher surprisal effect.
Results. We present the effect sizes of the interaction between scores and predictability measures in Table 1. First, we notice that for a given psy-
chometric test, all models consistently modulate surprisal and entropy effects in the same direction. For most psychometric tests, higher scores result in a reduction of surprisal and entropy effects, indicated by the negative interaction term coefficients. In these cases, individuals with higher scores show less sensitivity to a word’s predictability (measured in terms of surprisal or entropy). This holds true across all tests, the only exception being the Simon test, providing a measure of non-verbal inhibitory cognitive control. Here, high-performing individuals exhibit larger surprisal effects. Positive coefficients are also found for the Stroop task and the non-verbal part of the RIAS (intelligence), although they are extremely small.

heading: 4.4 Assessing the difference in predictive power between cognitive profiles (H3)
text: Finally, we investigate whether there are differences in the predictive power of LM surprisal and entropy for reading times obtained from individuals with different cognitive profiles. In other words, we ask the question what type of psycholinguistic subject a given language model emulates. To do so, we split the reading time data into subsets of high-performing (↑) and low-performing indi-
viduals (↓) at the median of each score. Then, for each group, we compute the ∆LL between the baseline model Mb3 only including word length and lexical frequency as predictors and the target model Mt3 with an additional predictor of interest xq3i ∈ {si, hi}, i.e., either surprisal or entropy. The individual ∆LL↓ and ∆LL↑ indicate the predictive power of surprisal and entropy for each group separately. In order to answer which group exhibited a higher relative gain in PP, we assess the difference in predictive power ∆PP def= ∆LL↑ − ∆LL↓. If a given LM is calibrated towards the cognitive profile of the high-performing individuals, we expect a positive ∆PP; and the ∆PP is negative if the LM is calibrated towards the low-performing cognitive profile.
Results. Figure 3 presents the differences (mean and 95% CI) in predictive power (∆PP) of surprisal or entropy between two groups that performed above or below the median, respectively, in a given psychometric test. ∆PP > 0 indicates higher PP for the high-performing group, ∆PP < 0 indicates higher PP for the low-performing group.
First, looking at the results for entropy, we note that across all models, entropy predicts the RTs
of individuals among the high-performing groups in the memory-updating and operation-span tests significantly better. Regarding reading fluency, readers with high word-reading scores were significantly better predicted from entropy estimated via GPT-2 large, Llama 2 13B and Mixtral.
For surprisal, we find that across all models, RT predictions are significantly better for the the low-performing group in the operation span test as well as the vocabulary size test MWT. Moreover, surprisal extracted from GPT-2 large and Llama 2 7B leads to significant gains in PP for the low-performing group in the RIAS test, which like MWT assesses verbal intelligence. Finally, surprisal estimated from Llama 2 7B and Mixtral showed significantly higher PP for the group of individuals with lower scores in the verbal and nonverbal cognitive control tests (Stroop, Simon).

heading: 5 Discussion
text: Most studies on the predictive power of surprisal and entropy on reading times have been conducted on the group-level. Although individual-level effects may have been taken into account in terms of random slopes, these effects have not been associated with different cognitive profiles. Our findings suggest that (1) incorporating information on individuals’ cognitive capacities and allowing them to modulate the magnitude of surprisal and entropy effects can increase the predictive power of these predictability measures, (2) individuals exhibit surprisal and entropy effects relative to certain cognitive capacities, and (3) some language models exhibit higher predictive power of reading times for groups of individuals associated with a certain cognitive profile.

heading: 5.1 Implications for the cognitive mechanisms of language processing
text: Fluent readers exhibit lower surprisal effects.7 Our results show that the predictive power of the interaction terms between surprisal and reading fluency and to some extent entropy and reading fluency are particularly high compared to the interaction terms including other psychometric tests, as depicted in Figure 2. Including the interaction term between reading fluency scores and surprisal improves the predictions on reading times for all language models. The negative coefficient (Table 1)
7As many results for entropy are not significant or less clear, the focus of the discussion will lie on surprisal.
can be interpreted from two perspectives. From the participants’ perspective, it underlines that individuals with high reading fluency exhibit lower surprisal effects. These results might indicate that less fluent readers rely more on predictive processing, hence their reading is easily interrupted by less predictable continuations, leading to longer reading times. Experienced readers, on the other hand, might be more trained to integrate unexpected material effortlessly. From the models’ perspective, on the other hand, it means that LMs overestimate the surprisal effect exhibited by highly fluent readers. Similar arguments can be made for the verbal intelligence test (RIAS-verbal), which is correlated with reading fluency (cf. Figure 4).
More accurate predictive processing for individuals with high working memory span. The results regarding the interaction terms between surprisal and working memory test scores are more difficult to contextualize. The span tests in particular (operation span, sentence span) lead to substantial increases in PP. Moreover, when assessing the magnitude of their interaction terms, we find that individuals with higher scores in both tests show lower surprisal effects as shown in Figure 2. At first glance, this intuitively makes sense, as high working memory can be associated with the capability to hold competing continuations in memory, including less likely ones that, in the high-surprisal situation, turn out to be the actual continuation. However, conversely, O’Rourke (2013) found in an ERP study that individuals with high operation span show stronger P600 effects, an event-related potential that is typically associated with syntactic repair or reanalysis. This would suggest that individuals with higher operation span exhibit a stronger garden path effect. However, garden paths are an extreme case of very high surprisal where different processing mechanisms might be at work such as re-analysis processes. Future experiments in minimal-pair settings will be needed to examine the connection between working memory capacity and surprisal effects more closely.
Attentiveness and inhibitory cognitive control may impact predictive processing differently. Next, we discuss the interaction term between surprisal and measures from tests targeting cognitive control. The directions of the interaction coefficients indicate that individuals with higher FAIRscores (attention and concentration) exhibit weaker
surprisal effects. Although this finding would be in line with the fact that general-purpose cognitive control mechanisms are required in the revision after linguistic misanalyses (Fedorenko, 2014), Vuong and Martin (2014) has shown that the time taken to revise a garden path is correlated only with verbal Stroop reaction time effects, but not with reaction time effects from its non-verbal counterpart (Simon). Since the FAIR scores showed fairly strong correlations with working-memory scores but not the other cognitive control tests (cf. Fig. 4), it is possible that the results obtained for FAIR might be more related to working-memory principles. Secondly, the results also showed that individuals performing well in the Simon test (inhibitory non-verbal cognitive control) exhibit stronger surprisal effects. As mentioned before, Vuong and Martin (2014) showed that the Simon task is likely not associated with mechanisms related to linguistic repair. The weaker surprisal effects for lowperforming individuals in the Simon task is more likely associated with the tendency of participants with lower control to skip revising misinterpretations entirely, i.e., to rely on good-enough processing (Ferreira et al., 2002).

heading: 5.2 Cognitive profiles of language models
text: Finally, regarding the group analyses, the results presented in Figure 3 revealed that surprisal estimates across all tested models predicted RTs better for the group of individuals with low verbal intelligence scores, measured with two largely complementary tests: one that assesses word knowledge (MWT-B), and one that assesses verbal logical thinking via question answering and sentence completion (RIAS-verbal). At first glance, this result is surprising since a language model has been exposed to billions of tokens, and therefore, one might expect that it emulates a psycholinguistic subject with high verbal intelligence. However, a language model’s predictions are always relative, i.e., even if it has seen infrequent words, it will still have a preference in terms of likelihood for the more regular, frequent continuation. Individuals with high verbal intelligence do not struggle with such contexts since they are very familiar even with uncommon terminology.
Additionally, we found that the PP of entropy is significantly higher for individuals with high working memory capacities, measured via memory updating and sentence span. This result suggests
that (un)certainty measures about upcoming material exhibited by LMs are more in line with the way high-working memory individuals process language, potentially driven by taking into account longer contexts, or keeping track of relevant long dependencies.
Even though most results from all three experiments are consistent within and across different LM families, there are exceptions. For instance, entropy estimated from GPT-2 large showed the strongest increase in PP for the high reading-fluency word reading group (Figure 3). For the high readingfluency pseudo-word reading group, it even represents the only measure with a significant increase in PP. Taking into account that GPT-2 base and large showed a similar baseline PP (Figure 1), suggesting that entropy extracted from GPT-2 large is a better proxy of processing effort for readers with lower verbal intelligence than entropy estimated with GPT-2 base. This illustrates that the choice of LM to estimate predictability measures is crucial for downstream analyses in psycholinguistic studies or NLP applications, especially when working with specific target groups. In such settings, it might be worthwhile considering a model that is less biased, or, in other words, whose predictability measures are well-aligned with the target group at hand as it will most likely lead to more accurate results.
While this study aimed at uncovering modelinternal biases, it might be worthwhile to, in turn, extend the investigation on whether text produced by a given LM is biased towards being processed more easily by individuals with specific cognitive characteristics. This is particularly important for tasks such as text summarization or simplification that might need to be tailored to specific groups.

heading: 6 Conclusion
text: To date, most investigations on predictability effects have been conducted on the group-level, disregarding individual differences, assuming that the predictive power of next-word predictability metrics such as surprisal or entropy on human reading times is uniform across cognitive profiles. In this work, we have shown that indeed, LMs do exhibit systematic biases towards readers of certain cognitive profiles. This illustrates the usefulness of incorporating individual-level information within the study of LM interpretability and language modelling in general.
Limitations
Splitting the subjects at the median of their scores obtained in the respective cognitive tests is a straightforward way to split them into high- and low-performing groups and avoids the problem of class-imbalance. However, this kind of split might not group participants whose scores are distributed more narrowly. For future work, it might be sensible to utilize more sophisticated clustering approaches to obtain more cognitively homogeneous groups.
Moreover, recent work has shown that test-retest reliability of individual surprisal effects are low, i.e., a surprisal effect for the same individual might vary on different days (Haller et al., 2023), depending on numerous factors such as wakefulness, motivation, but also random fluctuations. If this is true, we have to assume the predictive power with one and the same LM for a given subject, representing a cognitive profile, might be different depending on the subject’s condition on that particular day. However, using the Indico data, this factor is controlled to some degree since it combines data from temporally separate sessions. That way, even if the surprisal effect does depend on external factors, merging the data from several sessions ensures more robust estimates of each subject’s true surprisal effect. While it might still be a limitation, it is less so than for other conventional datasets. Finally, although InDiCo represents a fairly diverse sample in terms of age and gender, many participants have an academic background (see also Reich et al. (2024) on the necessity of including diverse populations in analyses of reading data).
Ethics Statement
Working with human data requires careful ethical considerations. The Individual Differences Corpus (InDiCo; Haller et al., 2023) utilized for this study follows the Helsinki Declaration (World Medical Association, 2013).

heading: Acknowledgements
text: This work was partially funded by the Swiss National Science Foundation under grant 100015L_212276/1 (MeRID). We thank David Reich, Nora Hollenstein and Omer Shubi for valuable discussions regarding this work.

heading: A Details on predictors
text: A.1 Language Models We deployed the following German LMs from the Huggingface library (Wolf et al., 2019):
• GPT-2 base: https://huggingface.co/benjamin/gerpt2 • GPT-2 large: https://huggingface.co/benjamin/gerpt2-large • Llama 2 7B: https://huggingface.co/LeoLM/leo-hessianai-7b • Llama 2 13B: https://huggingface.co/LeoLM/leo-hessianai-13b • Mixtral: https://huggingface.co/mistralai/Mixtral-8x7B-v0.1
A.2 Pooling of surprisal and contextual entropy to word level We compute word-level surprisal by summing up the surprisal values of the individual sub-word tokens. Given k sub-word tokens un, un+1, . . . , un+k belonging to the same word token, the word token’s surprisal is computed as
s(un, un+1, . . . , un+k) = − log p(un, un+1, . . . , un+k | u<n) = − log [p(un | u<n)p(un+1 | u<n+1) . . . p(un+k | u<n+k)] = − log p(un | un) +− log p(un+1 | u<n+1) + · · ·+− log p(un+k | u<n+k),
which shows that summing up sub-word token surprisal values is equivalent to computing the surprisal of the joint distribution of the sub-word tokens.
As regards entropy, we use the sum of the sub-word token-level contextual entropies as proxy for the joint entropy of the sub-word tokens’ distribution. Given k Σ̄-valued random variables Un, Un+1, . . . , Un+k belonging to the same word token, their joint entropy is defined as:
H(Un, Un+1, . . . , Un+k) def = −
∑
un∈Σ̄
∑
un+1∈Σ̄ · · ·
∑
un+k∈Σ̄ P (un, un+1, . . . , un+1) log2 [P (un, un+1, . . . , un+1)] .
However, depending on the tokenizer, the cardinality of Σ̄ could be over 50,000, which makes the computation of the joint entropy computationally unfeasible. Instead, we use the sum of the individual entropies as proxy. This is only a proxy, since
H(Un, Un+1, . . . , Un+k) ≤ H(Un) + H(Un+1) + · · ·+H(Un+k).
This inequality is an equality iff Un, Un+1, . . . , Un+k are statistically independent. Since this is not the case here, the sum of the sub-word token-level entropies is used as an upper bound.
B Individual Differences Corpus (InDiCo)
We provide abbreviations and a brief summary of all psychometric tests in Table 2. More details can be found in Haller et al. (2023). A correlation matrix between all tests can be found in Figure 4. We can see strong correlations between many tests, in particular for the ones of the same psychological construct.
Te st
/M ea
su re
C on
st ru
ct D
es cr
ip tio
n
Cognitivecontrol
St ro
op :r
ea ct
io n
tim e
ef fe
ct V
er ba
li nh
ib ito ry co gn iti ve co nt ro
l Pa
rt ic
ip an
ts ha
d to
re ac
t( ch
oo se
be tw
ee n
co ng
ru en
ta nd
in co
ng ru
en t)
fo rc
ol or
w or
ds w
ho se
fo nt
co lo
re ith
er m
at ch ed th e co nt en t( co ng ru en t) or no t( in co ng ru en t) .R ea ct io n tim e an d ac cu ra cy w er e m ea su re d.
Si m
on :r
ea ct
io n
tim e
ef fe
ct N
on -v
er ba l in hi bi to ry
co gn
iti ve
co nt
ro l
N on
-v er
ba le
qu iv
al en
tt o
th e
St ro
op Ta
sk w
he re
pa rt
ic ip
an ts
ha d
to re
ac t(
ch oo
se be
tw ee
n co
ng ru
en ta
nd in
co ng
ru en t) to ar ro w s po in tin g to th e ri gh to rl ef t, sh ow n ei th er on th e le ft or ri gh ts id e of th e sc re en .
FA IR
:K sc
or e
(t ot
al sc
or e)
N on
-v er
ba l
co gn
iti ve
co nt
ro l/a
tte nt
io n
Pa rti
ci pa
nt s
ha d
to fin
d an
d m
ar k
ta rg
et sy
m bo
ls (e
.g .,
di ce
w ith
2 ey
es am
on g
m an
y ot
he rd
ic e)
on a
pa ge
w ith
in a
tim e
lim it.
M ea
su re
s of
at te
nt io
na lp
er fo
rm an
ce ,a
tte nt
io n
qu al
ity ,a
nd at
te nt
io n
co nt
in ui
ty w
er e
de riv
ed .
Workingmemory
Se nt
en ce
sp an
V er
ba lw
or ki ng m em or y ca pa ci
ty Pa
rt ic
ip an
ts ha
d to
ju dg
e th
e m
ea ni
ng fu
ln es
s of
se nt
en ce
s an
d re
m em
be rl
et te
rs pr
es en
te d
af te
re ac
h se
nt en
ce fo
rl at er re ca ll. In th e en d, th ey ha d to re pe at al lt he le tte rs .
O pe
ra tio
n sp
an N
on -v
er ba l w or ki ng m
em or y ca pa ci ty
Pa rt
ic ip
an ts
w er
e pr
es en
te d
w ith
co ns
on an
ts se
qu en
tia lly
.A ft
er ea
ch co
ns on
an t,
th ey
ha d
to pe
rf or
m m
at he
m at
ic al
op er
at io
ns be
fo re
th e
ne xt
co ns
on an
ta pp
ea re
d. In
th e
en d,
th ey
ha d
to re
pe at
al lc
on so
na nt
s.
M em
or y
up da
tin g
N on
-v er
ba l
w or
ki ng
m em
or y
ca pa
ci ty
Pa rt
ic ip
an ts
ha d
to re
m em
be ra
n in
iti al
se to
fd ig
its ,e
ac h
pr es
en te
d in
a se
pa ra
te fr
am e
on th
e sc
re en
,a nd
th en
up da te th es e di gi ts in pa ra lle lt hr ou gh ar ith m et ic op er at io ns .
Sp at
ia ls
ho rt
-t er
m m
em or
y N
on -V
er ba l W or ki ng M
em or y C ap ac ity
Pa rt
ic ip
an ts
ha d
to m
em or
iz e
th e
sp at
ia ll
oc at
io ns
of do
ts in
a gr
id du
ri ng
a le
ar ni
ng ph
as e,
an d
th en
lo ca
te th
em on an em pt y gr id .
Intelligence
M W
T: Pe
rc en
til e
ra nk
V er
ba li
nt el
lig en
ce /
w or
d kn
ow le
dg e
Pa rt
ic ip
an ts
w er
e pr
es en
te d
lis ts
of w
or ds
,a nd
fo re
ac h
lis t,
th ey
ha d
to de
ci de
w hi
ch of
th e
pr es
en te
d w
or ds
w er
e re al w or ds
. R IA S: ve rb al pe rc en til e ra nk V er ba lI nt el lig en ce T hi s
te st
as se
ss ed
ve rb
al re
as on
in g,
an d
ve rb
al lo
gi ca
lt hi
nk in
g vi
a qu
es tio
nan
sw er
in g
an d
se nt
en ce
co m
pl et
io n.
R IA
S: no
nve
rb al
pe rc
en til
e ra
nk N
on -V
er ba l In te lli ge nc
e T
hi s
te st
as se
ss ed
no n-
ve rb
al re
as on
in g
an d
pr ob
le m
-s ol
vi ng
te st
s w
he re
pa rt
ic ip
an ts
w er
e pr
es en
te d
w ith
se ts of im ag es an d th ey ha d to de ci de w hi ch im ag e w as no tp ar to ft he se t. In th e ot he rt es t, th ey ha d to id en tif y m is si ng el em en ts in pi ct ur es
. R IA S: to ta lp er ce nt ile ra nk In te lli ge nc e To ta li nt el lig en ce sc
or e
ba se
d on
ve rb
al an
d no
nve
rb al
pa rt
.
Reading
SL R
T: W
or d
re ad
in g
pe rc
en til
e ra
nk R
ea di
ng flu
en cy
Pa rt
ic ip
an ts
re ad
ou tl
ou d
as m
an y
w or
ds w
ith in
on e
m in
ut e
as po
ss ib
le .
SL RT
:P se
ud ow
or d
re ad
in g
pe rc
en til
e ra
nk R
ea di
ng flu
en cy
Pa rt
ic ip
an ts
re ad
ou ta
s m
an y
ps eu
do -w
or ds
w ith
in on
e m
in ut
e as
po ss
ib le
.
Ta bl
e 2:
Ps yc
ho m
et ri
c te
st s
co nd
uc te
d w
ith al
lp ar
tic ip
an ts
.

heading: Future Work
text: For future work, it might be sensible to utilize more sophisticated clustering approaches to obtain more cognitively homogeneous groups. Moreover, recent work has shown that test-retest reliability of individual surprisal effects are low, i. e. , a surprisal effect for the same individual might vary on different days (Haller et al. , 2023), depending on numerous factors such as wakefulness, motivation, but also random fluctuations. If this is true, we have to assume the predictive power with one and the same LM for a given subject, representing a cognitive profile, might be different depending on the subject’s condition on that particular day. However, using the Indico data, this factor is controlled to some degree since it combines data from temporally separate sessions. That way, even if the surprisal effect does depend on external factors, merging the data from several sessions ensures more robust estimates of each subject’s true surprisal effect. While it might still be a limitation, it is less so than for other conventional datasets. Finally, although InDiCo represents a fairly diverse sample in terms of age and gender, many participants have an academic background (see also Reich et al. (2024) on the necessity of including diverse populations in analyses of reading data)