Revenge of the Fallen? Recurrent Models Match Transformers at Predicting Human Language Comprehension Metrics
Authors: James Michaelov
Catherine Arnett
Ben Bergen
Venue: COLM (2024)
Keywords: human language comprehension, human language processing, ERP, reading time, Mamba, transformer, psycholinguistics
TL;DR: When controlled for model scale, contemporary recurrent models are now also able to match—and in some cases, exceed—transformer performance at modeling human language comprehension.

Abstract: Transformers have generally supplanted recurrent neural networks as the dominant architecture for both natural language processing tasks and for modelling the effect of predictability on online human language comprehension. However, two recently developed recurrent model architectures, RWKV and Mamba, appear to perform natural language tasks comparably to or better than transformers of equivalent scale. In this paper, we show that contemporary recurrent models are now also able to match—and in some cases, exceed—performance of comparably sized transformers at modeling online human language comprehension. This suggests that transformer language models are not uniquely suited to this task, and opens up new directions for debates about the extent to which architectural features of language models make them better or worse models of human language comprehension.

Published as a conference paper at COLM 2024
Revenge of the Fallen? Recurrent Models Match Transformers at Predicting Human Language Comprehension Metrics
James A. Michaelov Catherine Arnett Benjamin K. Bergen"
"Department of Cognitive Science, "Department of Linguistics, University of California San Diego {jimichae, ccarnett, bkbergent Seenneucdddeee
Abstract
Transformers have generally supplanted recurrent neural networks as the dominant architecture for both natural language processing tasks and for modelling the effect of predictability on online human language compre- hension. However, two recently developed recurrent model architectures, RWKV and Mamba, appear to perform natural language tasks comparably to or better than transformers of equivalent scale. In this paper, we show that contemporary recurrent models are now also able to match-and in some cases, exceed-peer perfoorperr of comparably sized transformers at modeling online human language comprehension. This suggests that trans- former language models are not uniquely suited to this task, and opens up new directions for debates about the extent to which architectural features of language models make them better or worse models of human language comprehension.
1 Introduction
The origins of recurrent neural networks lie in attempts to model human cognition, and specifically the human language system (Jordan 1986 Elman 1990). Following improve- ments such as long short-term memory (LSTM; Hochreiter & Schmidhuber 1997: Gers et al. 2000), recurrent neural networks were for a while the dominant architecture not only for modeling human language comprehension ral language systems in general (see, e.g. Goldberg! (016). In recent years, they have in turn been superseded by transformer language models, which empirically tend to show better performance at both a range of natural language tasks (see, e-g., Radford al. 2019 Dai et al., 2019) and at predicting metrics of human language comprehension (e.g. Wilcox al. 2020 Merkx & Frank 2021 Michaelov et al. 2022). Nonetheless, the question of how recurrent and transtormer language models compare as cognitive models of the human lan- guage system is still an open one. One the one hand, recurrent neural networks inherently model the process of maintaining specific informational state and integrating this with new information as it occurs incrementally. This principle is widely believed to underlie language comprehension and other real-time processing (Merkx & Frank 2021 Michaelov et 2021). On the other hand, transformers have been argued to better model cue-based retrieval accounts of language comprehension (Ryu & Lewis 2021 Merkx & Frank 2021), and their direct access to previous words may allow them to better model human-like lexical priming effects (Michaelov
Frank et al., 2015), but for natu-
al. 2021). In addition, transformers' superior performance at predicting metrics of human language comprehension in itself serves as evidence that, at the very least, the statistical patterns learned by transformer language models capture something also learned by humans.
As they have increased scale (number of parameters, number of training tokens, or both), transformers have been found to improve at natural language tasks (Brown et al. 2020; Kaplan et al.) 2020 Rae et al.) 2022 Hoffmann et al., 2022 Chowdhery et al. 2022 Touvron et al. 2023), well at predicting both behavioral (Wilcox et al. 2020 Merkx & Frank, 2021) and neural (Merkx & Frank 2021 Michaelov et 2022 metrics of online language comprehension. But in recent years, two wrinkles have emerged. The first is

---

Published as a conference paper at COLM 2024
evidence that larger models and those trained on more data may actually predict some behavioral metrics of language comprehension (such as reading time) worse than smaller models do (Kuribayashi et al. 2021 Oh et al. 2022; Oh & Schuler 2023a b Oh et al. 2024 Shain et al. 2024). Second, two recently developed recurrent language model architectures appear to pertorm natural language tasks at least as well transformers of equivalent size and training: RWKV (Peng et al. 2023) and Mamba (Gu & Dao 2023). Transformers are therefore no longer the definitively best-performing language model architecture, and it is no longer the case that we should expect further advances in transformers to necessarily lead to improved fit to metrics of human language comprehension. Thus the time is ripe to revisit the question of which language model architecture best predicts human language comprehension.
To this end, we compare the performance of the Pythia (Biderman et al. 2023), RWKV (Pegg et al. 2023), and Mamba (Gu & Dao 2023) suites of autoregressive language models
on 12 human language comprehension datasets (Federmeier et al. 2007 Hubbard et al.
2019 Michaelov et al. Szewczyk & Federmeier] 2022) 2022: Szewczyk et al. 2022 Wlotko & Federmeier 2012: Boyce & Levy, Brothere
Brothers & Kuperberg, 22021 Futrell et al. 2021
Kennedy et al. 2003 Luke & Christiason 1018//mmie & Levy| 2013) covering 5 different metrics. Since all models were trained on the same dataset and have comparable numbers of parameters, we are able to measure the effect of architecture on the extent to which a language model's predictions correlate with metrics of human language comprehension.
2 Modeling prediction in human language comprehension
Over the years, wide range of language models have been used to model data from
experiments on human language comprehension, including n-gram models (e.g., McDonald
& Shillcock] 2003 Boston et al. 2008 Demberg & Keller 2008 Mitchell et al. 2010 Smith &
Levy : Brothers & Kuperberg 2021), recurrent neural networks (RNNs; e.g., Frank
& Bod [00T]/Fossum & Levyl 2012) Monsalve et al. 2012 Frank et al.) 2015 Goodkind &
Bicknell (011 Aurnhammer & | and most recently, transformers (e.g., Wilcox
et al.] 2020 Hao et al. 2020 Merkx & Frank 2021 Kuribayashi et al., 2021; Szewezyk &
Federmeier | et al. 2022 2024 Boyce& & Levy | 2 et al., al. Oh
et 2022 2024 O024/Oh& & Schuler (2023a by Shain et al. [024). Each approach can be evaluated
model (in the vein of Marr 1982) or as a cognitive model. Language models can serve as computational-level models since they calculate the probability of word in given context; and thus, their predictions can be compared with analogous measures of prediction in humans. Humans may also be able to predict the probability of words based on the statistics of language, given evidence that they are sensitive to statistical properties of language such as word frequency (Van Petten & Kutas 1990 Van Petten 1993; Dambacher et al. 2006 Rugg 1990; Fischer- -Baumer--aum-&a et al. 2014 [0144|Shain 2024). This opens a door for thinking of language models as plausible cognitive models, something further supported by recent work arguing that language models display linguistic competence (Piantadosi) 2023 Mahowald et al 2024).
in terms of how well it performs either as a com computational-leve
Existing studies vary in where they fall along the compuuttionall-oocognniit uum. At the computational end of the scale, a range of studies (Smith & Levy| (2013] Brothers
model contin-
& =Kuperberg] (021:Meisse et al. 2021 Wilcox et al., 2023b Hoover al. 2023) /Shain al.
MichalMc & Bergen 2022 2024 have tocused on understanding the mathematical
relationship between language model probability and processing difficulty, which does not necessarily require considering the specific model features. On the other end of the scale, several researchers have argued that contemporary transformers have a strong structural resemblance to the human language system (Schrimpf et al., 2021 Hosseini et al. 2024).
Most studies fall somewhere between the two extremes. This is particularly evidenced in work on the N400, a neural signal that is often considered to index the extent to which word has been predicted based on its preceding context (DeLong et al. 2005 Van Petten & Luka 2012 DeLong et al., 2014 Kuperberg et al. 2020). Frank & Willems (0017) , for example, explicitly choose to use a model with a modified n-gram architecture to investigate the role that pure word-level surface-level statistics may have on language comprehension. More
2

---

Published as a conference paper at COLM 2024
recently, Michaelov et al. (2024) used GPT-3 to investigate the extent to which prediction based on language statistics alone could account for several known N400 effects, including the fact that more semantically plausible words are processed more easily. We can also use this approach to test the viability of specific hypotheses about the language comprehension system by comparing language models that instantiate different theories of language com- prehension. Frank et al. (2015), for example, investigate how well a traditional recurrent neural network predicts N400 amplitude compared to a model implementing probabilistic phrase-structure grammar. They find that the former out-performs the latter, which they argue suggests that prediction during human language comprehension may rely more on statistical properties of language than on explicit hierarchical grammatical structure.
In the present study, we focus on the difference between recurrent language models and transformers. Many accounts theorize that human language comprehension involves construction of lossy and in some cases incorrect representations of the utterance and its meaning, due to cognitive resource limitations (Ferreira 2003: Christiansen & Chater 2016 Futrell et al. 2020), and it has been argued that recurrent models are a good computational models of this (Merkx & Frank 2021 Michaelov et al., 2021). This is perhaps most clear in the case of the now-or-never bottleneck account of language comprehension, under which our limited working memory means that 'the brain must compress and recode linguistic input as rapidly as possible' (Christiansen & Chater 2016). In this case, the analogy with recurrent models clear-a core component of such models is that they can take inputs of any length, but are limited in that any context must be compressed into a representation with fixed size, which is updated with each new word.
This is not the case for transformers, however. While they have fixed (though ever-increasing, see, e.g. S.g.,Llama Team 2024) context windows, they have perfect access to all the represen- tations of words within these context windows. Thus, their representations of word's context are not secrementally.comprreeee versions of the input like recurrent models; but rather, their representations of the context expand with each new word- word-within their context window, they have 'unlimited working memory' (Merkx & Frank 2021), which puts them at odds with accounts such as the now-or-never bottleneck theory of language comprehension. On the other hand, such seemingly lossless working memory may be more human-like than it may first appear. As Michaelov et al. (2021) note, humans do maintain specific past words in working memory, and indeed, there is evidence that reading a given word can lead to that word being easier to process for up to 45 minutes in some specific contexts (Besson et al., 1992] for discussion see Rommers & Federmeier 2018). Furthermore, transtormers have been argued to provide good computational-Ier llvel model of cue-based retrieval accounts of language comprehension (Ryu & Lewis, 2021 Merkx & Frank 2021). Specifically, under cue-based retrieval accounts (Mckiree et al. 2003 Van Dyke & Lewis, 2003 for review see Lewis et al. 2006 Parker et 2017), words are retrieved during language comprehension based on the features of previous words in the context which are used as cues; and analo- gously, it has been argued, the features of the representations of the words that transformers attend to when they predict the next word can be considered to function as cues to which word should be predicted (Ryu & Lewis 2021 Merkx & Frank 2021).
Beyond a priori cognitive plausibility, empirical studies with N400 data have almost univer- sally shown that transformers out-perform recurrent neural networks, and larger transform- ers trained on more data (and with lower perplexities) generally perform best at predicting N400 amplitude (Merkx & Frank 2021 Michaelov et al. 2022 Michaelov & Bergen 2022).
Language models have also been used to model reading time, which has also been hypothe- sized to reflect prediction in language comprehension. However, in this area, the results have been less straightforward. Smaller models display the same pattern seen with the N400, where larger language models trained on more data and with lower perplexities perform better (Goodkind & Bicknell 2018 Merkx & Frank 2021: Wilcox et al.) 2020 2023a Hao et al.) 2020). But past certain number of parameters or training tokens, their performance appears to deteriorate (Kuribayashi et al. 2021 Oh et al. 2022 2024 Oh & Schuler 2023a b Shain et al.) 2024). On the question whether recurrent neural networks or transtormers
best predict reading time, the results have been mixed (Wilcox et al. 2020: Eisape et al.)
2020 Kuribayashi et al. 2021), and have been found to difter depending on which metric of
reading time investigated (Merkx & Frank 2021
3

---

Published as a conference paper at COLM 2024
RWKV-4
Pythia
Mamba
Name
Parameters
Name
Parameters
Name
Parameters
169M
169,342,464
160M
162,322,944
130M
129,135,360
430M
430,397,444 440
410M
405,334,016
370M
371,516,416
1B
1,011,781,632
790M
793,204,224
1.5B
1,515,106,304
1.4B
1,414,647,808
1.4B
1,372,178,432
3B
2,984,627,200
2.8B
2,775,208,960
2.8B
2,768,345,600
Table 1: All the models used in our analysis, displaying the model's named size and the size as calculated using PyTorch. Models of comparable size are displayed next to each other. Further details for each model provided in the cited papers and their linked repositories.
The advent of new recurrent architectures that are increasingly feasible to train at large scale and that can perform as well as or better than transformers--nmmell RWKV and Mamba- is thus important in two ways. First, it allows observed in transformers that larger and better models predict N400 amplitude better but past certain point predict reading time worse also holds for other architectures with comparable natural language processing performance. Second, and perhaps more crucially, it allows us to again evaluate whether, when matched on scale or performance, recurrent or transformer architectures are better models of online human language comprehension.
to test whether the patterns previously
3 Method
3.1
Language Model Architectures
The aim of this study is to investigate how well metrics of online human language com- prehension can be predicted using three types of language model: the Pythia suite of autoregressive transformers (Biderman et al, 2023); and the recurrent RWKV (Peng et al., 2023) and Mamba models (Gu & Dao 2023). All models are trained on the Pile, 300B token
English-language dataset (Gao et al. 2020). For each architecture, selected models of comparable size (i.e., weight class) as shown in Table We discuss each architecture below.
Pythia Pythia [Biderman et al. 2023) is a set of autoregressive transformer models trained to be comparable across difterent model sizes, ranging from 70M to 12B parameters. The architecture and hyperparameters are based on GPT-3 (Brown et al. 2020), with the addition
of some changes based recent advancements (Dao et al. 2022 Su et al. 2024
Wang &
Komatsuzaki 2021 Belrose et al. 2023).
RWKV RWKV is a language model architecture described by its creators as Reinvent(ion of the] RNN for the Transformer Era' (Peng al. 2023). RWKV models combine the parallelizable training of transformers with unlimited context lengths, as well as several additional features that make them RNN-like. First, their time-mixing which can mathematically formulated in similar way to the recurrent states of an RNN (Peng et al. allows the representations of past states to be combined with those of new words. In addition, RWKV models explicitly have decay parameter such that tokens earlier in the context will be weighted less than later tokens during inference, thereby explicitly introducing something analogous to working memory limitations (Merkx & Frank 2021
Mamba Mamba is another recent recurrent model architecture (Gu & Dao 2023). One of the key goals of the Mamba architecture is to allow models to optimally compress their contexts, and especially very long contexts, into a state of fixed size such that they are still able to predict effectively. Like RWKV, Mamba computational complexity scales linearly with sequence length while avoiding the quadratic complexity of transformers (Gu & Dao, 2023). This is achieved by using a novel 'selective scan' mechanism that filters the input to select the most important information. Thus, Mamba models intuitively function like the
4

---

Published as a conference paper at COLM 2024
Dataset
Metric
Stimuli
Trials
Federmeier et al. (2007)
N400
564
32
7,856
Hubbard et al. 2019)
N400
192
32
5,705
Vichaelov et al. (2024)
N400
500
50
5,526
Szewczyk redermeier
2022
N400
600
26
4,822
Szewczyk et al. (2022)
N400
672
32
4,939
Wlotko & Federmeter
(012
N400
300
16
4,440
Boyce & Levy (2023)
Maze Response Time
9,304
63
56,447
Brothers Kuperberg
(2021)
SPR Three-Word RT
648
216
46,092
Futrell et al. (202T)
SPR Response Time
9,303
181
1,566,641
Kennedy et al. (2003
Go-Past Duration
38,186
10
195,507
Luke & Christanson
2018
Go-Past Duration
2,399
84
105,570
Smuth & Levy (2013)
SPR Response Time
6,297
35
119,120
Table 2: A description of each of the datasets, including the metric, the number of stimuli, the number of experimental participants (N), and the number of trials. See $3.2 for a brief explanation of each metric, and Appendix A for further details of each dataset.
more recent recurrent neural network variants- caruials- they include a latent state that is updated with each new input (like recurrent layers), and their selective scan method filters input (much like gating mechanisms in gated recurrent units or long short-term memory).
3.2 Datasets
In this study, we use language models of each of the three architectures discussed in $3.1
model 5 metrics of human language processing from 12 datasets, the details of which are
given in Table 2 These datasets comprise 6 N400 datasets (Federmeier et al.) [0077] Hubbard
eet al. 2019 Michaelov al., 2024 Szewczyk & Federmeier] 2022) Szewczyk et al.) 2022
Wlotko & Federmeier 2012) and 6 reading time datasets. The latter comprise four types of
reading time metric: the interval between when word first fixated by a reader and when they first move onto the next word, as calculated using eye-tracking (Go-Past Duration or Kenneen et al. 2003 Luke & Christianson, 2018), the time taken click to move onto the next word in a selt seli-paced reading task (Self- P'ced Reading Response Time SPR RT;
Futrell et al. 2021 Smith & Levy 2013), the total Response Time for word and the two following words (to account for spillover effects) in self-paced reading task (Self-Paced Reading Three-Word Response Time or 3W-RT; Brothers BrothrBrothher & Kuperberg, 2021), and the time taken to respond to each word the Maze task (Maze Response Time Maze RT; Boyce
& &Levy 2023). Further details of each metric and dataset are provided in (Appendix
3.3 Evaluation Procedure
We used the language models discussed in $31 to calculate the surprisal of all critical words in all datasets given their context. For the N400 and the Brothers & Kuperberg (2021) datasets, this context was made up of the preceding words in the same sentence. In the remaining datasets (Luke & Christianson, 2018 Boyce & Levy 2023), we included the whole preceding passage, comprising multiple sentences. For crifical words made up of multiple tokens, surprisal was calculated as the sum of all the sequential tokens comprising them.
We ran regression analyses for each dataset using linear mixed-effects regression models, predicting each human language comprehension metric using the surprisal calculated using each language model, as well as baseline covariates and random effects structures as described in Appendix A For each regression, we calculate the Akaike Information Criterion (AIC; Akaike 1973), measure of how well a regression fits the data, with lower AIC indicating a better hi. All language models were run in Python (Van Rossum & Drake 2009) using the transformers (Wolf et al.) 2020 library with pytorch (Paszke et al. 2019), pandas (McKinney] 2010), and humpy (Harms et al. 2020); and analyses were carried out in
(R Core Team 2023) using Rstudio (Posit team 2023) with the tidyverse (Wickham et al.
5

---

Published as a conference paper at COLM 2024
2019) 1me4 (Bates al. 2015), and scales (Wickham et al. 2023) packages. Code and data are available at / http://ggttuuu..o.iiiiiiiiiccca
com/ htep://cwwwccccccee//eeeereeeeee httpchaaaaeuoeeeeraaoaaaooaaaotn
4 Results
4.1
N400 Datasets
Scale impacts model performance a range of tasks, so we consider the differences between models while accounting for scale. Following previous work (e.g., Oh & Schuler (003a),, we consider differences between models when accounting for model size and for model perplexity. While the two are generally correlated, perplexity can help explain the effect of model size. Better models might align better with metrics human language compre- hension given own powerful predictive capabilities (Monsalve et al. 2012 Goodkind & Bicknell 2018 Michaelov et al. 2022), but by the same token, language models may learn to predict words loo well to model human language comprehension (Oh al. 2024).
We first consider the results arranged by model size (Figure TA). Overall, we find that in most cases, Mamba and RWKV performance better than that of Pythia, and Mamba is also better than RWKV. On the ledeermeerr et al. (2007) data, Mamba outperforms Pythia at all model sizes. On the Michaelov al. (2024), Szewczyk & F'edemmierr (2022), and Wlotko & Federmeier (2012) datasets, Mamba is better at all but one scale. Lastly, on the SheeSzewzzyk et al. (2022) and Hubbard et al. (2019) datasets, Mamba is better for all but two model sizes and roughly equal at an additional one for the latter dataset). On the Federmeier et al. 2007), Hubbard et al. (2019), and Szewezyk & Federmeier (2022) datasets, RWKV outpertorms Pythia at all but one size. For the other studies, RWKV outperforms Pythia at all but 2 sizes. For all studies, the best model fit across all model sizes is recurrent model; either Mamba RWKV.
One additional pattern relates to scaling. In contrast to recent work on reading time (e.g. Oh & Schuler 2023b) but in line with previous work the N400 (Merkx & Frank, 2021
(A) Model performance on N400 data by model size
(2007)
et
18490
135530
to N400 dala (AIC)
13670
Model
135510
Pyhha
Szewcayk & eedermeter (2022)
Szewceyk et (2022)
Wlokee & edeemmier 12012)
Mamba
9030
12050
12045
10320
10315
RWKV
9625
12040
10305
10300
9020
12035
10a
1045
30a
10
10a
1015
10f
10
10a
1045
30f
105s
Number of M odel Parameters
(B) Model performance N400 data by perplexity
Federmeler at (2007)
Hubbard etal. (2019)
Michaelov etal. (2024)
18490
FiLto N4C0 (NIC)
18470
13670
135530
18450-
135520
10430
13660
135530
Model
Pynis
Szewczyk Federmeter (2022)
5 Szowczyk at. (2022)
Wiodeo & Fedemaier 12012)
Mamba
12050
10320
RWIKV
12045
10315
10310
9825
10305
9820
12035
10300
25
20
15
30
15
25
70
15
Perplexity
Figure 1: Language model performance at predicting N400 amplitude.

---

Published as a conference paper at COLM 2024
Michaelov et al., 2022; Michaelov & Bergen 2022) we see that 4 of the 6 datasets (Federmeier et 2007 Hubbard et al. 2019 Szewczyk & Federmeier 2022 Wlotko & Federmeter 2012
show positive scaling eftects larger models tend to fit the data better.
In order to test how robust these patterns are, we run ordinary least-squares linear models for each dataset, predicting the AIC of the linear mixed-effects regressions based on language model scale and model architecture (Pythia, Mamba, or RWKV). After correction for multiple comparisons (Benjamini & Yekutieli 2001), we see that model scale is significant predictor of AIC, with surprisals calculated from larger models fitting the N400 data from 4 of the 6 datasets (Federmeier et al. 2007 Hubbard et al. 2019 Szewczyk & Federmeier 2022 Wlotko & Federmeier] 2012) significanlly better than smaller models. Given the low power of our analysis (only observations per dataset), it is also worth noting that before correction for multiple comparisons, Mamba models produce surprisals that fit the N400 data significantly better than Pythia models on the the/ Fedeemmeeii et al. (2007) and Wlotko & Federmeier (2012) datasets. While these latter results are suggestive rather than conclusive, they are consistent with the patterns observed in Figure 1A. The full results of our statistical analyses are provided in Table 3
Next, we consider the results arranged by model perplexity (Figure IB). Within each archi- tecture, there is no difference in pattern depending on whether order language models by size or perplexity. However, we do see a difference across architectures. In the four datasets that show positive scaling as a function of model size (larger models predict N400 amplitude better), when arranged by perplexity, Mamba models appear to perform worse relative to the other model architectures than they do when arranged by model size, while RWKV models appear to perform better. Conversely, on the dataset where two recurrent models show negative scaling (Szewczyk et al., 2022), we see the opposite pattern-Mamba Mamba appears to perform better, and RWKV appears to perform worse.
When we run ordinary least-squares linear models predicting AIC based on model perplexity and architecture, we see a similar effect to that seen for size. After correction for multiple comparisons, better language models (i.e., those with a lower perplexity) produce surprisals that better fit the N400 data on half of the datasets (Federmeier et all 2007 Hubbard et al. 2019 Wlotko & Federmeier 2012). The Szewczyk & Federmeier (2022) dataset also shows this pattern betore correction. Full results of our statistical analyses are provided in Table 5
4.2 Reading Time Datasets
For the behavioral reading data, we again first look at the data arranged by model size
Figuure 2A). The clearest effects are seen on the eye-tracking datasets (Kennedy al. 2003 Luke & Christianson 2018), where Pythia outperforms (or in one case, performs equally as well the better of) Mamba and RWKV at all sizes. We also see that Pythia tends to perform best overall on two of the other datasets (Boyce & Levy 2023 Futrell et al. 2021), with either Mamba or RWKV performing better at one or two sizes. The clearest exception to this pattern is the Brothers & Kuperberg (0221) dataset, where Mamba and RWKV outperform Pythia at all but one size. We also see a different scaling cattern-unlice
the the 5 datasets (Boyce & Levy 2023 Futrell et al., 2018: Kennedy et al., 2003 Luke & Christianson 2018 Smith & Levy] 2013) that generally show a negative scaling pattern(larger models perform produce less well-fitting surprisals), the Brothers & Kuperberg (2021) shows the positive scaling effect seen with the N400 data. The & Levy (2013) results are less clear, both in terms of differences between models and terms of overall scaling patterns.
An ordinary least-squares linear model predicting AIC based on number of parameters and architecture also shows this difference. Even after correction for multiple comparisons, model size has a significant effect on 4 of the 6 datasets (Boyce & Levy. (2023] Futrell et al.)
2021 Kennedy et al.
2003 Luke & Christianson, 2018 Smith & Levy 2013 in addition
to the Smith & Levy (2013) dataset betore correction for multiple comparisons), with the
surprisal calculated from larger models showing a worse fit to the data. Intriguingly,
in line with the aforementioned observations based on Figure 2A,, before correction, the Brothers & Kuperberg (2021) dataset shows the opposite effect- the same positive scaling
we see on some of the N400 datasets. Returning to the differences between architectures,
7

---

Published as a conference paper at COLM 2024
[A] Model performance on reading time data by model size
Bayce axy
Brothers 3w 3W-RT
F.troll (2021);
Fit Reading Time (AIC)
134000
1849000 -
133800
133000
85900
85910
85890
1849400
Model
85850-
Pyhis
Kennedy etal. GPD
2832300
Luce Christansan (2018:: GPD
Smir and Lovy 12013): SPR
Mamba
524600
2812200
283110
RWKV
224400
281300
283100
281050
524200
201020
200950
203000
10f
1044
10'
1043
10
10
10'
10
10f
1045
10'
1044
Number Parameters
(B) Model performance reading time data by perplexity
Soyce & Levy (2023): Maze RT
Brothers Kuperberg (2021): 3W -TT
Futroll etallietall (2021): SPR RT
Fi Reading Titre (NC)
134000
133800
85920
85910
85930
1849600
1849400
133400
85890
1849200
Model
Pyhts
Kennedy etal. GPO
2812300
Lute Christanson (2018):
283120
Smith and Levy 12013): SPR SPR RT
Mamba
624600
283110
RWKV
524400
281300
203100
281050
203090
281000
280950
283080
30
15
20 Perplexity
30
15
Figure 2: Language model performance at predicting 4 reading time metrics (see $3.2)
after correction, surprisals calculated using Mamba fit the Luke Christianson (2018) and Kennedy et al. (2003) data significantly worse than Pythia, with this also being true of RWKV on both datasets before correction. Further details are provided TTble 4
For the perplexity-ordered data (Figure 2B), the same pattern emerges as for the N400 data-for the dataset where positive scaling is found (Brothers & Kuperberg 2021 , Mamba models appear to perform relatively worse relative to Pythia than they do when the models are ordered by size and RWKV models perform relatively better, and for datasets where negative scaling is found, Mamba models appear to perform relatively better and RWKV models relatively worse. Again, while we see N400-like positive scaling on the Brothers & KKueerbrrg (021) dataset-with lower- wwwrr-peppeextyy models showing a better fit to the Sata-we see the opposite pattern with the remaining 5.
Further confirmation of the different scaling patterns comes from ordinary least-squares linear models predicting AIC based on perplexity and architecture. After correction for multiple comparisons, models with a lower perplexity produce surprisals that are signifi- cantly better at predicting the (Brothers & Kuperberg (2021) data, but significantly worse at predicting reading time in 4 datasets (Boyce & |222] Futrell et al., 22021 Kennedy 2023
et al. 2003 Luke & Christianson 2018 and again, the Smith & Levy 2013 dataset before correction). In this analysis, surprisal values calculated from the Mamba and RWKV models also show significantly worse fit to the Kennedy et al. (2003) data, with those from the RWKV models also showing this on the Luke & Christianson (018)) dataset before correction for multiple comparisons. The details of all statistical analyses are provided in Table 6
Discussion
To the best of our knowledge, the present study is the first to compare the extent to which transformers and contemporary recurrent language models can model online human lan- guage comprehension. Previous work has overwhelmingly found that transformers better predict the N400 than recurrent neural networks (Merkx & Frank 2021 Michaelov et al.
8

---

Published as a conference paper at COLM 2024
2021 2022). We show, by contrast, that on 6 datasets, when comparing models of the same size and trained on the same data, contemporary recurrent language model architectures generally out-perform transformers, with surprsal values calculated using Mamba models tending to provide the best fit to the N400 data. When accounting for model perplexity, the comparison across architectures is less clear-cut; however, the contemporary recurrent architectures at least match transformer performance.
The results are more mixed for reading time metrics. On the Kennedy et al. (2003) dataset, for example, the Pythia models predict go-past duration best at any scale or perplexity; while on the other hand, the recurrent models predict self-paced reading time on the Brotthrr & Kuperberg 2021) dataset best except at the 1.4-1.5B scale. Such mixed results for behavioral data is perhaps unsurprising given the conflicting results in previous work (Goodkind &
Bicknell 2018 Merkx & Frank 2021: Hao et al. 2020 Wilcox et al. 2020 2023a Kuribayashi
al. 2021
Oh et al. 2022] 2024; Oh & Schuler 2023a Shain et al. 2024).
We also report several interesting scaling results. First, on the whole, scaling patterns are consistent across architectures. For datasets where larger, lower perplexity models tend to predict the human metric better (Federmeier et 20., Hubbardd et al., 2019; Szewczyk & Federmeier 2022 Wlotko & Federmeier 2012:Brrothers Brothers & Kuperberg, 2021), this tends to be true for all model architectures. The same true for datasets where smaller models and those with a higher perplexity tend to predict the human metric better (Boyce & Levy 2023 Futrell et al. 2021 Kennedy et al., 2003 Luke & Christianson 2018). The (Szewzyyk et al. (0022) and (mith & Levy (2013) datasets offer possible exceptions, where different models appear to show different scaling effects. However, without more models of each architecture, it is impossible to be certain.
Another surprising result is that contrary to the recent work that finds the same negative scaling pattern across all reading time datasets (including both self-paced reading and eye-tracking metrics; Oh & Schuler, 2023b) Oh et al.) 2024], here one dataset (Brothers & Kuperberg, 2021) actually showed positive scaling. One possible explanation for this is that it not trg-transformed like the other reading time metrics. Another is that it includes the reading times of the following two words. However, the fact that it is slightly better predicted by taking the surprisal of the first word rather than that of the combined three words Shain et al. 2024 SI 1) suggests that the metric itself may not differ in this way as much as may be expected. A more likely explanation for the difference is that unlike the other behavioral studies which involved the reading of naturalistic stimuli, the stimuli in the Brothers & Kuperberg (202T) were carefully constructed to have different degrees of predictability. All the N400 studies use such stimuli, and this may therefore explain why the Brothers & Kuperberg (222)) results more closely resemble the positively-scaling N400 results. In any case, the finding highlights the point made by Brotther & Kuperberg (2021) that the task and stimuli used in such studies should not be overlooked when making wider claims about the relationship between probability and processing difficulty. It further suggests that the recent and ostensibly robust findings of negative scaling with behavioral data (Oh et al. 2022 2024 Oh & Schuler 2023a b) may be limited to a specific type of reading time study, and that further analyses should be carried out.
Finally, we note the finding that when comparing architectures by model perplexity rather than model size, there was consistent pattern in terms of which model best predicted the data. Specifically, compared to when ordered by model size, when the dataset showed positive scaling, the performance of Mamba appeared worse relative to other architectures, and the performance of RWKV appeared better; and when the dataset showed negative scaling, the reverse was true Given that at each size, Mamba has lower perplexity than Pythia and RWKV has a higher perplexity (Gu & Dao 2023 Appendix B), this suggests that a language model's ability to predict the next word a sequence does impact the extent to which it can model online human language comprehension above and beyond model size and architecture. Specifically, this result suggests that there are scaling effects across model architectures related to model quality (i.e., performance at word prediction). Even when controlling for number of parameters and training data, on a dataset that exhibits positive scaling, models that are better at next-word prediction are better at the human metric; and the converse is true for datasets that exhibit negative scaling.
9

---

Published as a conference paper at COLM 2024
5.1 Theoretical implications
Ultimately, the results highlight number of complicating facts. First, there is no single universal pattern accounting for the relationship between language model probability and all metrics of online human language comprehension. Second, general language modeling performance has an effect on the extent to which language models can predict such metrics. And third, there are idiosyncratic differences between datasets, metrics, and model architectures.
Nonetheless, the present study opens up new lines of research. Crucially, in contrast to previous work, the results show that transformers are not uniquely well-suited to modeling the N400. They also align with previous research showing the same for some measures of reading time (Eisape et al. 2020; Kuribayashi et al. 2021; Merkx & Frank 2021: Oh al., 2022). Indeed, in our results, the differences in modeling pertormance between models of different architectures at a given scale or perplexity tend be dwarfed by the differences within architectures across these dimensions.
In the present study, the performance of transformers and recurrent models is comparable, and thus our results are not able to evaluate whether there are specific architectural fea- tures of transformers or the recurrent models that make them better able to model human language comprehension. As discussed in reeuurrettt models provide a better model of the role of the working memory bottleneck (Merkx & Frank 2021: Michaelov et al. 2021), while transformers better simulate cue-based retrieval models of comprehension (Ryu & we
Lewis 2021 Merkx & Frank 2021). It not necessarily straightforward to disentangle the two. For example, |Ru & Lewis s argument that transformers are good models of cue-based retrieval is based on the linding that they display interference effects on agreement (also known as agreement attraction effects) and show patterns in attention that align with the theory. But recurrent neural networks have been observed to display such effects (Arehalli & Linzen 2020), which could be plausibly explained by lossy compression of the context. Nonetheless, identifying cases where the behavior of recurrent models differs from that of transformers qualitatively- possible exception of the Szewczyk et al. 2022 data)-and comparing these to the human data is likely to be valuable across the the more purely computational end, it is important to know which type of model to use to best capture the possible effects of statistics on language comprehension. Further towards the direction of cognitive modeling, such experiments can help to evaluate which theories provide more viable explanations of human language comprehension, for example, by comparing the predictions of the now-or-never bottleneck and cue-based retrieval accounts.
than quantitatively, as in the present study (with the
model continuum. At
The new generation of recurrent models is in its infancy. As these models continue to be developed, optimized, and scaled up, the question of whether they or transformers provide better models of human language comprehension (or at least, show a stronger degree of correlation to specific metrics of online human language comprehension) is likely to become clearer. In the meantime, the results presented here suggest that recurrent models not only match, but in some cases exceed, the performance of contemporary transformers at modeling human language comprehension, and may provide valuable way to test hypotheses about the neurocognitive mechanisms underlying it.
Conclusions
We compare how well transformers and two contemporary recurrent language model architecture--RWee and Mamba-can predict 5 different metrics of online human lan- guage comprehension. We find that overall, the recurrent models tend to match the perfor- mance of transformers at predicting both neural and behavioral human metrics, and that when specifically comparing across architectures by number model paramaters, recurrent models in fact appear to be best at predicting N400 amplitude.
10

---

Published as a conference paper at COLM 2024
Acknowledgments
We would like to thank the San Diego Social Sciences Computing Facility Team for the use of the Social Sciences Research and Development Environment (SSRDE) cluster. Models were evaluated using hardware provided by the NVIDIA Corporation as part of an NVIDIA Academic Hardware Grant.
References
Hirotogu Akaike. Information Theory and an Extension of the Maximum Likelihood Principle. B. N. Petrov and Csáki (eds.), Second International Symposium on Information Theory, Springer Series in Statistics, pp. 267-281, Budapest, Hungary, 1973. Akadémiai Kiadó. doi: 10.1007 19977998---61221999-11
Suhas Arehalli and Tal Linzen. Neural Language Models Capture Some, But Not All Agree- ment Attraction Effects. In Stephanie Denison, Michael Mack, Yang Xu, and Blair C. Armstrong (eds.), Proceedings of the 42th Annual Meeting of the Cognitivee Science Soci- efy. cognttivesceencssoiiett...gg 2020.
Christoph Aurnhammer and Stefan L. Frank. Evaluating information-theorret mea- sures word prediction in naturalistic sentence reading. Neuropeyychologia, 134:107198,
2019. ISSN 0028-3932. doi: 10. 200006//jmnunoopyyyooolli22202.0
URL http:
Douglas Bates, Martin Mächler, Ben Bolker, and Steve Walker. Fitting linear mixed-effects models using 1me4. Journal of Statistical Software, 67(1)11-48, 2015. doi: 10.18637 /jss.v0677. i01.
Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. Eliciting Latent Predictions from Transformers with the Tuned Lens, 2023. /ttp///tp//rrxi orgeooggg//aooo abs/2303, 08112
Yoav Benjamini and Yosef Hochberg. Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. Journal of the Royal Statistical Society. Series B (Methodological), 57(1))289-300) 1995. ISSN 0035-9246. URL httph:::///w// jst.:www..................... org/ stable/2346101
Yoav Benjamini and Daniel Yekutieli. The Control of the False Discovery Rate in Multiple Testing under Dependency. The Annals of Statistics, 1944) 2001. ISSN 0090-5364.
URL https://ww// jstor http:::s//ww...ooooororr/aaaaaaa
Mireille Besson, Marta Kutas, and Cyma Van Petten. An Event-Related Potential (ERP) Analysis of Semantic Congruity and Repetition Effects in Sentences. Journal of Cognitive Neurosciennce 4(2):132-14,, 1992. ISSN 0898-929X. doi: 10.1162/jocn. 111162//ocn.1992.222333 URL
1.9..4.2.11. 132
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, Usvsn Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar Van Der Wal. Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling. In Proceedings of the 40th International Conference on Machine Learning, pp. 2397-2430, PMLR, 2023. URL https://proceedingss m1r www.ppree//veeeiiieennaaa22a html
Marisa Ferrara Boston, John Hale, Reinhold Kliegl, Umesh Patil, and Shravan Vasishth. Parsing costs as predictors of reading difficulty: An evaluation using the Potsdam Sentence Corpus. Journal of Eye Movement Research, 2(1), 2008 ISSN 1995-8692. doi: 10.16910/jemr..11111 URLhhttps://bog unibe. caaeee.hh/indeee http:ppp/////aarriel////////////
Veronica Boyce and Roger Levy. A-maze of Natural Stories: Comprehension and surprisal in the Maze task. Glossa Psycholinguusticc,, 2(1), 2023. ISSN 2767-0279. doi: 10.5070/
URL |ttppp://eeccoaasssspppoooo//o/p
11

---

1ttp:///ooo..or//11115555 Published as a conference paper at COLM 2024
Trevor Brothers and Gina R. Kuperberg. Word predictability effects are linear, not logarith- mic: Implications for probabilistic models models of sentence comprehension. Journal of Memory and Language, (ttpnnge,1166 116:104174, 2021. ISSN 0749-596X. doi: 00.10016mmmmm2220001111.1..
URL
http::/www.sccience. http://twwwc...acceactcteaaa/aaa
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Vonr Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz: Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Πlya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL httpp:://pppp:rrrr nips. cttpssccccpp//22//// hah/4457coaofoo44444111166 wateac142f444444 42114422f4aa-saa wwwteeefaaaatttttttttaaaaaaaa
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Hen- ryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanu- malayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstemn Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. PaLM: Scaling Language Modeling with Pathways, 2022. URL httt:: http:///aaxivv.o//bbb/ss//// wwt.//gg///////eeeoeo.ooooe 2204. 02311
Morten H. Christiansen and Nick Chater. The Now-or-Never bottleneck: A fundamental constraint on language. Behavioral and Brain Sciences, 39, 2016. ISSN 0140-525X, 1469-1825. doi: 10:0017/5010055555000111 URL
https::/wwwe
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhut- dinov. Transformer-XI: Attentive Language Models beyond Fixed-Length Context. In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pp. 2978-2988, Florence, Italy, 2019. Association for Computational Linguistics. doi: 10.18653/v1 11t1//111P19118/
Michael Dambacher, Reinhold Kliegl, Markus Hofmann, and Arthur M. Jacobs. Frequency and predictability effects on event-related potentials during reading. Brain Research,
1084(1):89-103, 2006. ISSN 0006-8993, doi: 200.00066/braainees22000660222 02.001. URL https
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, Advances in Neural Information
neur cctps.ccppppprrrrrp fftprp/wwr2z.aaniaa22/o//ooooooo
Processing Systems, 35:11344-166331 2022. URL http://///../tttt/////////ennenn
html
Katherine A. DeLong, Thomas P. Urbach, and Marta Kutas. Probabilistic word pre-activation during language comprehension inferred from electrical brain activity. Nature Neuroscience,
1(8):1117-1121, 2005. ISSN 1546-1726. doi: 10.1038/ 0..1038/nn104,, nn1504 URL https:: //wws. nature com/articles/nm
Katherine A. DeLong, Melissa Troyer, and Marta Kutas. Pre-Processing in Sentence Com- prehension: Sensitivity to Likely Upcoming Meaning and Structure. Language and Lin-
12

---

Published as a conference paper at COLM 2024
guistics Compass, 8(12)66116645, 2014. ISSN 1749-818X. doi: 10.1111/ /21111////33. /tt..........33/ URL
https://omlineeibrary httpp/awwwaaaaaacccooooooooaaa//
12893
Vera Demberg and Frank Keller. Data from eye-tracking corpora as evidence for theories
of syntactic processing complexity. Cognition, 109(2)193-210,, 2008. ISSN 0010-0277. doi:
10.1016 www.ogonnnniiooon.00077 pi1/S00166277880017411
Tiwalayo Eisape, Noga Zaslavsky, and Roger Levy. Cloze Distillation: Improving Neu- ral Language Models with Human Next-Word Prediction. In Proceedings of the 24th Conference on Computational Natural Langwage Learning, pp. 609-619, Online, 2020. As- sociation for Computational Linguistics. doi: 10.18653/v1/2022ccoml111
URL
conll-1.49
Jeffrey L. Elman. Finding Structure in Time. Cognitite Science, 14())179-2111, 1990. ISSN 1551-6709. doi: 10. 10..220///0l 105507700001 140__1. URL http:::s//onniiinnirrrrrrr wiley. com/ ho//6bs/10.1200//155166666880
Kara D. Federmeier, Edward W. Wlotko, Esmeralda De Ochoa-Dewald, and Marta Kutas. Multiple effects of sentential constraint word processing. Brain Research, 1146:75 84, 2007. ISSN 00068993, doi: 10.1016/j.brainres. 2t0p://brraareee22000..0..... 2006.06.00. URL https://linkinghub
elsevier.com//// allevveecccmmeerrrreeee//////669
Fernanda Ferreira. The misinterppretation of noncanonical sentences. Cognitive Psychology, 47(2):164-208 2003. ISSN 0010 htt.://555...2.00000000000000000 URL https
Simon Fischer-Baum, Danielle S. Dickson, and Kara D. Federmeier. Frequency and regularity effects reading are task dependent: Evidence from ERPs. Language, Cognition and Neuroscience, 29(10):1342-1335 2014. ISSN 2327-3798. doi: 10.1080
htt/hhttss:///oi.orrr org/10. 1080/23273798 2014. 4.77767
Victoria Fossum and Roger Levy. Sequential vs. hierarchical syntactic models of human incremental sentence processing. In Proceedings of the 3rd Workshop on Cognitive Modeling and Computational Linguistics (CMCL 2012), pp. 61-69, 2012.
W. Nelson Francis and Henry Kučera. A Standard Corpus of Present-Day Edited Ameri- can English, for use with Digital Computers, 1964. URL http::/korpus......///// uib. no/icame/ manuals/ Homuals/BRRMiee BROWN/INDEX.. HTMM
Stefan Frank and Rens Bod. Insensitivity of the Human Sentence-Processsing System to Hierarchical Structure. Psychological Science, 22 2011. ISSN 0956-7976. doi: 1171 179157797771199899
10.1177/095679761140000
URL
Stefan L. Frank and Roel M. Willems. Word predictability and semantic similarity show
distinct patterns of brain activity during language comprehension. Language, Cognition and 2017. ISSN 2327-3798. doi: 10.1000//222337922277.222
(tto:sccinnce,3299)1199922 URL httt:p:s::/oii.orr/////// /ttp://////0.0000////////3//9990 2017. 1323109 2017.1323109
Stefan L. Frank, Leun J. Otten, Giulia Galli, and Gabriella Vigliocco. The ERP response to the amount of information conveyed by words in sentences. Brain and Language, 140:1-11, 2015. ISSN 0093-934X. doi: 10.1016//jbmmdd.2244..00.0
URL hhttp://www.scienceee.
Richard Futrell, Edward Gibson, Harry J.
Tily, Idan Blank, Anastasia Vishnevetsky, Steven Piantadosi, and Evelina Fedorenko. The Natural Stories Corpus. In Nicoletta Calzolari, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Koiti Hasida, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, Stelios Piperidis, and Takenobu Tokunaga (eds.), Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, 2018. European Language Resources Association (ELRA). URL https: http://aaaaanooooyycoo org/
L18-1012
13

---

Published as a conference paper at COLM 2024
Richard Futrell, Edward Gibson, and Roger P. Levy. Lossy-Context Surprisal: An Information-Theorrtte Model of Memory Effects in Sentence Processing. Cognitive
Science, 1284), 2020. ISSN 1551-6709. doi: 10.1111/cogs.12111
URL https
//onlinelibrary. wwt.:wwwe...c...... wwtm///o/abb////////ooo/oo wttp/////..///111////./..///
Richard Futrell, Edward Gibson, Harry J. Tily, Idan Blank, Anastasia Vishnevetsky, Steven T. Piantadosi, and Evelina Fedorenko. The Natural Stories corpus: reading-time corpus of English texts containing rare syntactic constructions. Language Resources and Evaluation, 55(1) 77, 2021. ISSN 1574-0218. doi: 10.1007/110599-000--99---- URL httpps//doi org/10.
09500777779
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: An 800GB Dataset of Diverse Text for Language Modeling. 2020. URL
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Éric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation. Zenodo, 2021. URL https //doi.org/10.522 0ttpg/0o.521//2eod..5533333
Felix A. Gers, Jürgen Schmidhuber, and Fred Cummins. Learning to Forget: Continual Prediction with LSTM. Neural Computation, 12(10))2451-2471, 2000. ISSN 0899-7667 doi: 10.1162/08997660000000
Yoav Goldberg. A Primer on Neural Network Models for Natural Language Processing.
Journal of Artificial Intelligence Research, 57:34554400 2016. ISSN 1076-9777. 10.1613/ jair.4922. BtLhhttp::/// hwwp:://wwwwww/
Adam Goodkind and Klinton Bicknell. Predictive power of word surprisal for reading times a linear function of language model quality. In Proceedings of the 8th Workshop on Cognitive Modeling and Computational Linguisfics (CMCL 2018), pp. 10-18, Salt Lake City, Utah, 2018. Association for Computational Linguistics. doi: 10.18653 /v1553v1//// 11111100022 URL
Albert Gu and Tri Dao. Mamba: Limear-Time Sequence Modeling with Selective State Spaces, 2023.
Yiding Hao, Simon Mendelsohn, Rachel Sterneck, Randi Martinez, and Robert Frank. Probabilistic Predictions of People Perusing: Evaluating Metrics of Language Model Performance for Psycholinguistic Modeling. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pp. 75-86, Online, 2020. Association for Computa- tional Linguistics. doi: 10.18653/vi /v15 /0035v11/22000/ ht//2200ccmcl1..10 URL /tLphttts:///alanthooo. org/ 2020. cmc1-1.10
Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Vir- tanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre Gérard-Marchane Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke and Travis E. Oliphant. Array programming with NumPy. Na- ture, 585(7725))357-366) 2020. ISSN 1476-4687. doi: (0.1038/s41586-000 http:://ww.natuue/ com wttp///ee..oomattcleeeeeeee/e/e/ 000.cceee///11156-002----66----1 220-2649-2
URL
586
Sepp Hochreiter and Jürgen Schmidhuber: Long Short-Term Memory, Neural Computation, 9(8):13351770, 1997. ISSN 0899-7667.
doi: http:///eeeco1977...11.77 URL https
1tt..oorg/11..166/eeee.. 1997.9999... 1735
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de las Casas, Lisa Anne Hendricks, Johannes Welbi, Aidan Clark, Tom Hennigan, Eric Noland, Katherine Millican, George den Driessche, Bogdan Damoc,
14

---

Published as a conference paper at COLM 2024
Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack William Rae, and Laurent Sifre. Training Compute-Optima Large Language Models. In Advances in Neural Information Processing Systems, 2022. URL httpss//ppeerrvieee..eefoorrrrr iBBCRULOAPR
Jacob Louis Hoover, Morgan Sonderegger, Steven T. Piantadosi, and Timothy O'Donelll. The Plausibility of Sampling as an Algorithmic Theory of Sentence Processing, Open Mind, 7:350-391, 2023. ISSN 2470-2986. doi: 10.1162/opmi a 00086. URL https://doi. 1162/opmi a 00086
Eghbal A. Hosseini, Martin Schrimpf, Yian Zhang, Samuel Bowman, Noga Zaslavsky, and Evelina Fedorenko. Artificial Neural Network Language Models Predict Human Brain Responses to Language Even After a Developmentally Realistic Amount of Training. Neurobiology of Language, pp. 1-21, 2024. ISSN 2641-4368. doi: 10.1162 nt.12//nola.00133..... URL https://doi. org/ 10. 10. 1162/nola.000133
Ryan J. Hubbard, Joost Rommers, Cassandra L. Jacobs, and Kara D. Federmeier. Down- stream Behavioral and Electrophysiological Consequences of Word Prediction on Recog- nition Memory. Frontiers in Human Neuroscience, 13, 2019. ISSN 1662-5161. URL 2019.0022
Michael I. Jordan. Serial Order: A Parallel Distributed Processing Approach. Technical Report 8604, Institute for Cogmitive Science, University of California, San Diego, La Jolla, California, USA, 1986.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling Laws for Neural Language Models, 2020. URL http:///aaaxi...ograaaa/////////
08361
Alan Kennedy and Joël Pynte. Parafoveal-on-foveal effects in normal reading. Vision Research, 45(2):153-168 2005. ISSN 0042-6989. doi: 200.111. .037. URL
Alan Kennedy, Robin Hill, and Joël Pynte. The Dundee Corpus. In The 12th European Conference on Eye Movements, Dundee, UK, 2003.
Alan Kennedy, Joël Pynte, Wayne S. Murray, and Shirley-Anne Paul. Frequency and predictability effects in the Dundee Corpus: An eye movement analysis. Quarterly Journal of Experimental Psychology, 66(3))601--16)) 2013. İSSN 1747-0218. doi: 1.1000/17470018.. 2012.676054. URL httptht/tsti.orreeeeee 108001777000 2012.67625. 676054
Gina R. Kuperberg, Trevor Brothers, and Edward W. Wlotko. Tale of Two Positiv- ities and the N400: Distinct Neural Signatures Are Evoked by Confirmed and Vi- olated Predictions at Different Levels of Representation. Journal of Cognitive Neuro- science, 32(1):1223 2020. ISSN 0898-929X, 1530-8898. doi: 10.1162 /jonnn 01465. URL 1162/jocn a 01465
Tatsuki Kuribayashi, Yohei Oseki, Takumi Ito, Ryo Yoshida, Masayuki Asahara, and Kentaro Inui. Lower Perplexity is Not Always Human-Like. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joinf Confer- ence on Natural Language Processing (Volume 1: Long Papers), pp. 5203-5217, Online, 2021. Association for Computational Linguistics. doi: 10.18653 / /tt1///..///.//./oogogooo acl-llng URL
acl long..og.. 405
Marta Kutas and Steven A. Hillyard. Reading senseless sentences: Brain potentials reflect semantic incongruity. Science, 207(442))2000 1980. ISSN 0036-8075, 1095-9203. doi: 10. 1126//cciecce7350655
Marta Kutas and Steven A. Bra and semantic association. Nature, Nature, 307547221337/5547711111163,, 1984. ISSN 0028-0836, 1476-4687. doi:
reading word expectancy Brain potential during
15

---

Published as a conference paper at COLM 2024
Richard L. Lewis, Shravan Vasishth, and Julie A. Van Dyke. Computational principles of working memory in sentence comprehension. Trends in Cognitive Sciences, 10(10):
447-454, 2006. ISSN 1364-6613, 1879--3377111193 1879-307X. doi:
URL https
Llama Team. The Llama 3 Herd of Models. 2024. URL http:::/aai.metaaacmm////
iooliccaiios/thhe-e oftp:/hee-----m--------e--------
Steven G. Luke and Kiel Christianson. The Provo Corpus: A large eye-tracking corpus with predictability norms. Behavior Research Methods, 50(2): 2018. ISSN 1554-3528. doi: 10.3758/ 0111888/11328----0008888
Kyle Mahowald, Anna A. Ivanova, Idan A. Blank, Nancy Kanwisher, Joshua B. Tenen- baum, and Evelina Fedorenko. Dissociating language and thought in large language models. Trends in Cognitive Sciences, 0(0), 2024. ISSN 1364-6613, 1879-307X. doi: 21.1166/ abstract/S1364- 660))64446-66134300007777--
URL https://wwww cell. wwtp////eends/oogpiiieeeeeiereee
David Marr. Vision: A Computational Investigation into the Human Representation and Processing of Visual Information. W.H. Freeman, San Francisco, 1982. ISBN 978-0-7167-1284-8
Scott A. McDonald and Richard C. Shillcock. Eye Movements Reveal the On-Line Computation of Lexical Probabilities During Reading. Psychological Science, 14(6):648- doi:
652, 2003. ISSN 0956-7976.
psc...sss..8..... URL https
//doi.org/1e. 1046/j (tt5:/j..956777 8956-7976 2003, pwtpppccc..1.11 14801.11400.x
Brian McElree, Stephani Foraker, and Lisbeth Dyer. Memory structures that subserve sen- tence comprehension. Journal of Memory and Language, 48(1))67-91 2003. ISSN 0749-596X. doi: 10.1016//00749-99660000001111111 URL /ttir////..///tttt//////////ssaa http:///www.cceeeeccieeeet.cce/s
Brricle//01//0779955X220555
Wes McKinney. Data Structures for Statistical Computing in Python. In Stéfan van der Walt and Jarrod Millman (eds.), Proceedings of the 9th Python in Science Conference, pp. 56 61, Austin, Texas, 2010. doi: 10. 1025500//Maioro-92229222222
URL https://conference
scipy.
cclpy..rgg/poceeddngsss http://www.reeeeeeee///////////m
Clara Meister, Tiago Pimentel, Patrick Haller, Lena Jäger, Ryan Cotterell, and Roger Levy. Revisiting the Uniform Information Density Hypothesis. In Proceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Processing, pp. 963-9993)91 180,980, Online and Punta Cana, Dominican Republic, 2021. Association for Computational Linguistics. doi: 10. Btt5:///122221.mmp--ma
URL URL/ntps::/acllanholeee org/
org/2021....22.ooo 2021. .emnl--main 74
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer Sentinel Mixture Models. In Infernational Conference on Learning Representations, 2017. URL https
Danny Merkx and Stefan L. Frank. Human Sentence Processing: Recurrence or Attention?
In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pp- 12-22, Online, 2021. Association for Computational Linguistics. doi: 10.18653/v1/ /t2.l//122222m1m11111
URL / .rgg 2ttp//2221.ccc112222.2
James A. Michaelov and Benjamin K. Bergen. The more human-like the language model, the more surprisal is the best predictor of N400 amplitude. In NeurIPS 2022 Workshop on - Principles in Cognitive Systems, 2022. URL htttttss///oeeeeeieeeee net/ forum?)dd=uGgYVb66N22
James A. Michaelov and Benjamin K. Bergen. On the Mathematical Relationship Between Contextual Probability and N400 Amplitude. Open Mind, 8:859 897, 2024. ISSN 2470-2986.
doi: 10.1162/opmi 000150. URL http:ttts://o.orrg/////
1162/opmi.a.00 0000:p0pmaaa00110000000000
James A. Michaelov, Megan D. Bardolph, Seana Coulson, and Benjamin K. Bergen. Different kinds of cognitive plausibility: Why are transformers better than RNNs at predicting N400 amplitude? In Proceedings of the 43rd Annual Meeting of the Cognitive Science Society, pp. 300-306, University of Vienna, Vienna, Austria (Hybrid), 2021.
16

---

Published as a conference paper at COLM 2024
James A. Michaelov, Seana Coulson, and Benjamin K. Bergen. So Cloze yet Far: N400 Amplitude is Better Predicted by Distributional Information than Human Predictability Judgements. IEEE Transactions on Cognitive and Deoelopmental Systems, 2022. ISSN 2379- 8939. doi: 001109/TTCSS2222311122
James A. Michaelov, Megan D. Bardolph, Cyma K. Van Petten, Benjamin K. Bergen, and Seana Coulson. Strong Prediction: Language Model Surprisal Explains Multiple N400 Effects. Neurobiology of Langwage, 5(1):107-135, 2024. ISSN 2641-4368. doi: 10.1162/ nol a 00105.
/ttps://doi.or////// 1162/nol a 00105
Jeff Mitchell, Mirella Lapata, Vera Demberg, and Frank Keller. Syntactic and semantic factors in processing difficulty: An integrated measure. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pp. 196-206, 2010.
Irene Fernandez Monsalve, Stefan L. Frank, and Gabriella Vigliocco. Lexical surprisal as general predictor of reading time. In Proceedings of the 13th Conference of the Euro- pean Chapter of the Association for Computational Linguistics, pp. 398-408. Association for Computational Linguistics, 2012.
Byung-Doh Oh and William Schuler. Transformer-Based Language Model Surprisal Predicts Human Reading Times Best with About Two Billion Training Tokens In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Com- pulational Linguistics: EMNLP 2023, pp. 1915-1921, Singapore, 2023a. Association for Computational Linguistics. doi: 10.18653/ /v1 /t1///22...nmiiii finddings-emmlp.12s URL https!
//aclanthology wwwpooloooooogg//////o www.o///.....nnmmee findings-nnnnnnnpp.. 128
Byung-Doh Oh and William Schuler. Why Does Surprisal From Larger Transformer-Basee Language Models Provide a Poorer Fit to Human Reading Times? Transactions of the Association for Computational Linguistics, Linguittii,iiiLiiiiiiiiiiiii 11:336-350, 2023b. ISSN 2307-387X. doi: 10.1162/
ta04.8 htllfttpp///bll.og/////11///////
Byung-Doh Oh, Christian Clark, and William Schuler. Comparison of Structural Parsers and Neural Language Models as Surprisal Estimators. Fronttiers in Artificial Intelligence, 5, 2022. ISSN 82122. doi: 10.3389 2022.77769. httph:tpp::/www Frontiersin
org/articles/1e.c (aatriccee/1e.338/ffaa.....22... 2022. 777963
Byung-Doh Oh, Shisen Yue, and William Schuler. Frequency Explains the Inverse Corre- lation of Large Language Models' Size, Training Data Amount, and Surprisal's Fit to Reading Times. In Yvette Graham and Matthew Purver (eds.), Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2644-2663, St. Julian's, Malta, 2024. Association for Computational org/ 2024. eacl eacl-long. 162
Linguistics.
Dan Parker, Michael Shvartsman, and Julie Van Dyke. The cue-based retrieval theory of sentence comprehension: New findings and new challenges. In Linda Escobar, Vicenç Tor- rens, and Teresa Parodi (eds.), Language Processing and Disorders, pp. 121-144. Cambridge Scholars Publishing, Newcastle, 2017.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, An- dreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Py'oorch: An Im- perative Style, High-Perfoormncce Deep Learning Library. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL http:/ttts///ooeeddiiiiii/ neurips. hou.gp/.ccaaer22/////////d2222e0
Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Leon Derczynski, Xingjian Du, Matteo Grella, Kranthi Gv, Xuzheng He, Haowen Hou, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bart,lomic Koptyra, Hayden Lau, Jiaju Lin, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito, Guangyu Song, Xiangru Tang, Johan Wind, Stanis) law Wožniak, Zhenyuan Zhang, Qinghua Zhou, Jian Zhu, and Rui-Jie Zhu. RWKV: Reinventing RNNs for the
17

---

Published as a conference paper at COLM 2024
Transformer Era. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 14048-14077, Singapore,, 2023. Association for Computational Linguistics. doi: 10.18653/ (116533/1/// 222.....niingg-emm//./. URL
wewddfinnninnnnnn..n 936
Steven Piantadosi. Modern mndgee refute Chomsky's approach to language, 2023.
URL httpps://inggguunnnn///nnguuuuuu
Posit team. RStudio: Integrated Development Environment for R. Boston, MA, 2023. URL
httpp://www.posi..ccm
R Core Team. R: A Language and Encironment for Statistical Computing. Vienna, Austria, 2023.
BtRhhtts:://wet //ww:. RR-projecttoor. org/
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language Models are Unsupervised Multitask Learners. pp. 24, 2019.
Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptist Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d' d'Autmme, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, lason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. Scaling Language Models: Methods, Analysis & Insights from Training Gopher, 2022. /tRl:tttp://arxx..rrg/////
11446
Joost Rommers and Kara D. Federmeier. Predictability's aftermath: Downstream conse- quences of word predictability revealed by repetition effects. Cortex, 101:16-30, 2018. ISSN 0010-9452, doi:
com/
sciencce/rrtiile/pii//// 0tt0//00000992227442220
Michael D. Rugg. Event-related brain potentials dissociate repetition effects of high-and low-frequency words. Memory & Cognitiio,, 18(4):367-379, 1990. ISSN 1532-5946. doi:
(0.3758/BF0319911 URL https:://doo..org//...oooo http://///...00000//000111111110
Soo Hyun Ryu and Richard Lewis. Accounting for Agreement Phenomena in Sentence Com- prehension with Transformer Language Models: Effects of Similarity-based Interference on Surprisal and Attention. In Proceedings of the Workshop on Cognitive Modeling and Compu- fational Linguistics, Pp. 61-71, Online, 2021. Association for Computational Linguistics. doi: 10.18653/
URL https:tt//// http:::/www..ccllc// awwweb.bwbb........... org/anntologyy/2221111111 1t1.88211..11111111111111111 1.mcl-1.66
Martin Schrimpf, Idan Asher Blank, Greta Tuckute, Carina Kauf,, Eghbal A. Hosseini, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. The neural architecture of language: Integrative modeling converges predictive processing. Proceedings of the National Academny of Sciences, 1 118/45)e21006646118 2021. doi: http:///aaa..1054445. 18.
URL https://www/ pnas,
http://wwppaaa..orr////////////e wwtp:o/g////oooooo 1003///a0...3///as.. 2105646118
Cory Shain. Word Frequency and Predictability Dissociate in Naturalistic Reading, Open Mind, 8:177-201, 2024. ISSN 2470-2986. doi: 10.1162 00119. URL. https://doi
org/10: 1162/opmi 11t11110mmpmmaa00111 _aa0m_a___011____0
Cory Shain, Clara Meister, Tiago Pimentel, Ryan Cotterell, and Roger Levy. Large-scale evidence for logarithmic effects of word predictability reading time. Proceedings of the
18

---

Wilson L. Taylor. "Cloze" readability scores as indices of individual differences in com- prehension and aptitude. Journal of Applied Psychology, 41(1):19-26, 1957. ISSN 1939- (ttielecctooniii..22191000000000 Published as a conference paper at COLM 2024
National Academy of Sciences, 121(110)e230777611111 2024. doi: 10.1073 http://///maaa2..07720/2/
www..lnass.. wwt.a/oor////////eeeeeeee 1073/paas 2307876121 250
Nathaniel J. Smith and Roger Levy. The effect of word predictability on reading time is
logarithmic. Cogmition, 128(3):302-311 2013. ISSN 0010-0277, doi: 10.0/
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer:
Enhanced transformer with Rotary Position Embedding. Neurocomputingg5666:22222 2024. ISSN 0925-2312. doi: 10.1016/jjneucomm.. 22201000.....22....22...... 127063. URL /RLLittss//www.sciieeeddde
Jakub M. Szewczyk and Kara D. Federmeier. Context-based facilitation of semantic access follows both logarithmic and linear functions of stimulus probability. Journal of Memory and Language, 123:104311, 2022. ISSN 0749-596X. doi: 10.1016//jmm..2221.1..
URL
https://ww..scienceet httpa///wws..c.ceeee///e/a/aa//a
Jakub M. Szewczyk, Emily N. Mech, and Kara D. Federmeier. The power of "good": Can adjectives rapidly decrease well increase the availability of the upcoming noun? Journal of Experimental Psychology: Learning, Memory, and Cognition, 48:856-875, 2022, ISSN 1939-1285 doi: 10.1037 10.1037/xlm0000 1000000091
Wilson L. Taylor. "Cloze Procedure": New Tool for Measuring Readability. Journalism Quarterly, (044):41--433 1953. ISSN 0022-5533. doi: 10.11771177699053003300
URL
http://journals sagepub. www/ cttcc///00..117/7779999000000
doi: 10.1037 10.1037/h0040599. /h0040591.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and Efficient Foundation Language Models, 2023. URLL/http//arxiw.orr//// 230/ 13971
Julie A Van Dyke and Richard I. Lewis. Distinguishing effects of structure and decay on attachment and repair: A cue-based parsing account of recovery from misanalyzed ambiguities. Journal of Memory and Language, 49(3):285-316, 2003, ISSN 0749-596X. doi: 10. 00.1016/8574999966000000000
URL https://www.ccieeeedrrr...c.o
ar articleepp1//5074955666888
Cyma Van Petten. A comparison of lexical and sentence-level context effects in event-related potentials. Language and Cognitive Processes, 8(4):485-531, 1993. ISSN 0169-0965. doi: 0011///9119999904777666 /ttpa//////////////////////////0
Cyma Van Petten and Marta Kutas. Interactions between sentence context and word frequency in event-related brainpotentials. Memory & Cogmiition, 18(4):380-333, 1990. ISSN
1532-5946. doi: 10.3758 (0.3758/BB0331990 BF03197127. /0Jjitttpp:///olloor//////11/113
Cyma Van Petten and Barbara J. Luka. Prediction during language comprrensi::.......nn.ccoooo Benefits, costs, and ERP components. International Journal of Paychhoplussolooyyyy8 83()) 2012 ISSN 0167-8760. doi: 10. wttp:/pcccccc...1099900 cctp:/ccienee//rtttlle/////////1
2012.
URL http::/ww..ccinnettee
Guido Van Rossum and Fred L. Drake. Python 3 Reference Manual. CreateSpace, Scotts CA, 2009. ISBN 14414-1229-7.
Scotts Valley,
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 billion parameter autoregressive language model, 2021. URL httpp::/giithul http://www.....ioooooo///eeeeaae
19

---

Published as a conference paper at COLM 2024
Hadley Wickham, Mara Averick, Jennifer Bryan, Winston Chang, Lucy D'Agostino Mc- Gowan, Romain François, Garrett Grolemund, Alex Hayes, Lionel Henry, Jim Hester, Max Kuhn, Thomas Lin Pedersen, Evan Miller, Stephan Milton Bache, Kirill Müller, Jeroen Ooms, David Robinson, Dana Paige Seidel, Vitalie Spinu, Kohske Takahashi, Davis Vaughan, Claus Wilke, Kara Woo, and Hiroaki Yutani. Welcome to the tidyverse. Journal of Open Source Software, 444)):1686, 2019. doi: 10.21105/joss: /o...////.s....666..
Hadley Wickham, Thomas Lin Pedersen, and Dana Seidel. Scales: Scale Functions for Visualization, 2023. URL http://CCAAN.--Pjjjjc..
Ethan Wilcox, Clara Meister, Ryan Cotterell, and Tiago Pimentel. Language Model Qual- ity Correlates with Psychometric Predictive Power in Multiple Languages. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Em- pirical Methods in Natural Language Processing, pp. 7503-7511, Singapore, 2023a. Asso- ciation for Computational Linguistics. doi: 10.18653 /0/2223.emllp https://aaclannhooooo.or//////// ennep-mmnnnpmmnn 466
URL
Ethan G Wilcox, Jon Gauthier, Jennifer Hu, Peng Qian, and Roger Levy. On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behavior. In Proceedings of the 42nd Annual Meeting of the Cognitive Science Society (CogSci 2020), pp- 2020.
Ethan G. Wilcox, Tiago Pimentel, Clara Meister, Ryan Cotterell, and Roger P. Levy. Testing the Predictions of Surprisal Theory in 11 Languages. Transactions of the Association for Computational Linguistics, 11:1451-1470, 2023b. ISSN 2307-387X. doi: 10.1162/ 11/////acl.. a 00612.
URL /ttLpht:s///o..oooo org/ 1162/tacc 1162/tacl_a_066111
Edward W. Wlotko and Kara D. Federmeier. Finding the right word: Hemispheric asymmetries in the use of sentence context information. Neuropsychologia, 45(13): 3001-3014, 2007. ISSN 00283932. doi: 1010f6//neuroopyccololiaaa w0w...000000000 0.0007005001... URL
elsevier ottpo//err.eerrrrrrr/////////222
Edward W. Wlotko and Kara D. Federmeier. So that's what you meant! Event-related poten- tials reveal multiple aspects of context use during construction of message-level meaning. Neurolmage, (2()):356616)))3 -366,,,,,66666 2012. ISSN 1053-8119. doi: htwp//////e///eeeeraaaae00000010
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-Arr Natural Language Processing.
Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38-45, Online, 2020. Association for Computational Linguistics. doi: 10.18653 /000 /ttp///////2200em//deeeooos
URL
URLhttpss: //ww:///wwwwwww http://wwweb..rg/annoo.oooo wwt.:/ogggg..oggggooooooooo 2020. www.0eddenosss..
20

---

Published as a conference paper at COLM 2024
A Data and Analysis Details
A.1 N400 Amplitude
The N400 is negative-going component of the event-related brain potential that occurs roughly 300-500ms after the presentation of a stimulus, peaking at around 400ms (Kutas & Hillyard 1980). A well-replicated finding is that the amplitude of the N400 response to a word is sensitive to the contextual probability of a word, either operationalized as cloze probability (Kutas & Hillyard, 1984) the proportion of people to fill in gap in sentence with a given word (laylor 1953 1957 1957)-or or when using the predictions of language models (Frank et al. 2015). Specifically, the amplitude of the N400 response elicited by word is large by detault, and decreases by the extent to which it is predictable based on the preceding context.
In this study, we compare how well the Pythia, RWKV, and Mamba models predict N400 amplitude based on the results of 6 experiments (Federmeier et al., 2007 Hubbard al., Wlotko & Federmeier 2012). The details of these datasets and how they were analyzed are outlined below.
Michaelov et al. 2024 Szewczyk & Federmeier 2022; Szewczyk et al. |
Federmeier et al. (0077) measured N400s to low- and high-cloze words in low- and high-constraint contexts. We use the data from this study as preprocessed by Szewczyk & Federmeier (0022). In this dataset, N400 amplitude is operationalized at four centro-parietal electrodes (MiCe, MiPa, LMCe, RMCe) over the 300-500ms time window. N400 amplitudes are also not
the mean voltage
instead, the mean amplitude in the -1000mms time window is intended to be included as a covariate in analysis. This dataset contains 7856 trials from 32 participants reading 564 stimuli.
To calculate model fit to the N400 data, we followed as closely possible the approach used by Szewezyk Federmeier 2022). , which involved predicting N400 amplitude using a linear mixed-effects regression with surprisal, baseline amplitude, rog-transformed frequency, the position of the word in the sentence, orthographic neighborhood distance, and concreteness as fixed effects. We also used the same random effects structure, removing variables until structure that would not lead to singular fits for any regression was reached, which included random slopes of baseline for each subject and experimental item, random slopes of word position for each subject, and random intercepts of subject and item. We then compared the fit of regressions using surprisal calculated from each language model.
With the exception of the Michaelov et al. 20244) dataset, our remaining N400 datasets are the others provided online by |Szewezyk & Federmeier (2022), and thus are preprocessed and analyzed in the same way.
Wlotko & Federmeier 2012) used stimuli from Federmeier et al. (2007 well as (Wlotko
& Federmeier (2007), ) which were selected to cover a wide range of probabilities. This dataset was made up of 4, 4,400 trials (300 stimuli; 16 experimental participants).
Hubbard al. 2019 used 192 stimuli from Federmeier et al. 2007). The dataset comprises of 5,705 trials (32 participants).
Szewczyk al. 2022 also based their stimuli on those in Federmeier et al. (2007), with adjectives added betore critical words, making them more or less predictable. The dataset is comprised of 4,939 trials (672 stimuli; 32 participants).
Szewczyk & Federmeier 2022) also release an additional dataset with data from
previouslyumpuubisseee study using stimuli based on Federmeier et al. (2007) and including data from 4,822 trials (600 stimuli; 26 experimental participants). We reter to this as the
Szewezyk & Federmeier (2022) dataset.
21

---

Published as a conference paper at COLM 2024
Michaelov et al. (2024) The stimuli of Michaelo et al. (2024) dataset differ from the other datasets in their design. Rather than having two versions of each sentence- most likely continuation and an unlikely one sentence has four possible endings: the highest-cloze continuation, a low-cloze but plausible continuation that is semantically equally low-cloze but unrelated continuation,
related to this highest-cloze continuation, and an implausible continuation. The two low-cloze completions were matched for cloze probability and plausibility. There were 125 sentence frames, for total of 500 sentences. There were fifty participants, and data from a total of 5,526 trials after cleaning.
The N400 was operationalized as the mean voltage in the 300-500ms time-window at each of the C3, Cz, C4, CP3, CPz, CP4, P3, Pz, and P4 centro-parietal electrodes. Unlike the data released by Szyeezykk & Federmeier (2022), the voltage each electrode was treated as a separate data point and N400 amplitudes were baselined using the mean amplitude in the 100ms period before stimulus presentation. Thus this dataset comprises of 49,734 data points. We analyzed these data in the same way as in Michaelov et al. (2024), fitting regression that predicted N400 amplitude using Surprisal, log-transformed word frequency, orthographic neighborhood distance as main effects, and included random intercepts of experimental subject, sentence context, critical word, and electrode.
A.2 Self-Paced Reading Response Time
Self-Paced Reading is an experimental paradigm in which participants read text one word at a time, pressing button or key to proceed to the next word. The reading time of word is the time taken between button presses (i.e., between pressing the button to proceed to that word and pressing the button to proceed to the next word). Self-Paced Reading Response Time is generally considered to reflect processing difficulty, with longer reading times indexing more difficult word.
Futrell et al. (2021) The Natural Stories Corpus (Futrell et al. 2021 is made up of self- paced reading times from 181 experimental participants reading 10 texts, each comprising roughly 1000 words. These texts were constructed by taking publicly available texts and editing them to contain rare and hard-to-process syntactic constructions. Following Ohl et al.] (2024), we excluded reading times for all words that appeared at the beginning or end of a sentence, all reading times shorter than 100ms or longer than 3000ms, and all data from participants who answered three or fewer comprehension questions correctly. The regression used to predict log-transee log-transformed reading time also followed that described by Ohl lt al. (2024). In addition to language model surprisal surprisal, we included word length, log-transtormed word frequency, and the word's position in the sentence as predictors, as well as random slopes of each of these predictors for each subject, and random intercepts of subject and sentence.
Smith & Levy 2013) This dataset is comprised of self-paced reading times from 35 experi- mental participants reading 292-902 word passages from the Brown Corpus of American English (Francis & Kučera 1964), for total of 2860-4999 words per subject. We used the same exclusion criteria as with the Natural Stories data (with the exception of the compre- hension score, which was not available), and the linear mixed-effects regressions each had the same structure as those used in analyzing the Natural Stories data.
A.3 Self-Paced Reading Three-Word Response Time
A well-known phenomenon in self-paced reading is that of spillover effects, where the extent to which a word is difficult to process also impacts the reading time of one or more following words. One way to account for this to use the reading time of given word and the following two words as measure of the processing difficulty associated with the word.
Brothers & Kuperberg 22021) In this self-paced reading study, there were 216 sentence sets, each in a low-, medium- or high-cloze condition, for total of 648 stimulus sentences. Participants were excluded by Brothers & Kuperberg (2021) if they had an average com- prehension check score of less than 75%. After exclusions, data from 216 of the total 240
22

---

Published as a conference paper at COLM 2024
participants were included in the analysis with total of 46,092 data points. Data were cleaned and preprocessed by BBothers & Kuperberg (2021). We fit regressions following the method in the original study, predicting reading time (un-transformed) using linear mixed-effects model with a main effect of language model surprisal, random intercepts for each subject and item, and random slopes of surprisal for each subject and item.
A.4 Maze Task
Like self-paced reading, in the Maze task, participants read a text one word at a time. However, in the Maze task, participants see pairs of words and can only proceed to the next word in the text by choosing the correct next word on the screen. If the participant chooses the incorrect word, they receive feedback and are prompted to choose again. The time it takes for participants to choose a word is recorded as the reaction time. We look at the reaction times from a previous study by Boyce & Levy (2023). Dataset and analysis details are provided below.
Boyce & Levy 2023) In this study, participants completed Maze task using the stimuli from the Natural Stories corpus (Futrell et al. 2021), which comprises 10 texts based on publicly available texts, each approximately 1000 words long. In total, Natural Stories contains 10,245 words. [ooyce& & Levy (0233) recruited 100 participants, but participants
were excluded if they did not selt-report as native speakers of English. Following Boyce& & Levy (2023) and Shain et al. (024), we exclude data for all words with a reading time of less than 100ms or greater than 5000ms, incorrect words, words that were at the start or end of a sentence, and all data from participants that correctly answered fewer than 80% of comprehension questions correctly. We construct línear mixed effects regressions predicting log-tra log-transformed reaction time with surprisal, word length, log-transformer word frequency, and the word's position in the sentence. We also included random slopes of surprisal, word length, and word position for each subject, as well as random intercept of sentence. Our analysis is based on the preprocessed version of this dataset provided by /Shain al. (2024).
A.5 Go-Past Duration
Go-past duration is an eye-tracking- based metric of reading time. In eye-tracking stud- ies, participants generally read text naturalistically. Unlike in the other experimental paradigms, participants can see the whole text at one time and are able to look at previously read words. The location of each participant's gaze is recorded using an eye tracker, which also records how long participants' gaze is fixated on given location. There are many different possible eye-tracking metrics for a given word that can be calculated (see, e-g., Shain et al. 2024), but following recent work analyzing how well different language models predict eye-tracking data (Oh & Schuler [203b| al Oh et al. 2024), we look at log- log-transformed go-past duration, which is defined as the amount of time from when the word was first fix- ated to when the participant first looked to the right of that word (in left-to-right languages like English; see Luke & Christianson 2018 Shain et al. 2024 We use data from the Provo corpus (Luke & Christianson 2018). The details of this dataset and how was analyzed are provided below.
Luke & Christianson 2018 The Provo Corpus (Luke & Christianson 2018) is dataset consisting of eye-tracking data for 84 participants reading 55 passages (news articles, popular science magazines, and fiction). Passages averaged 50 words long. In total, the texts comprised 2,689 words. Participants' go-past durations were recorded while they read each text. As with the N400, we use linear mixed-effects regressions to calculate the fit of the surprisals calculated by each model to the data. Following recent work (Oh et al., 2024). we exclude from our analysis all words that were not fixated, that followed saccades of longer than 4 words, and that were at the start or end of sentences or files. Also following Oh et al. (2024), we constructed a regression to predict trg-fraanformme go-past duration based on surprisal as well as the following covariates: saccade length (in words), word length (in characters), word position in the sentence, log-transformed word frequency, and whether the previous word was fixated. We also included random slopes of all predictors for each
23

---

Published as a conference paper at COLM 2024
subject, well random intercepts for each subject and sentence. Our analysis is based the preprocessed version of this dataset provided by Shain et al. (2024) which was combined with the full set of stimuli provided by Luke & Christanson (0018).
Kennedy et al. (0003) The Dundee Corpus (Kennedy et al. 2003) is a dataset consisting
of eye-tracking data from 10 participants reading 20 text files, each roughly 2,800 words
in length, for total of 56,212 words (Kennedy & Pynte 2005 Kennedy et al. 2013 for additional details, see). Our analysis approach was the same as for the Provo Corpus (Luke
& Christianson 2018): exclusion criteria for data were the same (except that, following Oh jet al. 2024] we also exclude words at the start and end of lines and of the screen, which not annotated in the Provo Corpus), was the structure of each linear mixed-effects regression. Our analysis is based on the preprocessed version of this dataset provided by Shain et al. (2024).
24

---

Published as a conference paper at COLM 2024
B Comparison of model scale and perplexity
In the original Mamba paper, Gu & Dao 2023 report the performance of Mamba models on a number of benchmarks against comparable language models with different architectures. In the 1.4-1.5B and 2.8-3B parameter range, Gu & Dao (2023) find that Mamba has lower perplexity than Pythia on the Pile (Gao al. 2020) validation set, and that RWKV has higher perplexity. This result suggests that at these scales, the Mamba models are best able to learn the statistics of language, followed by the Pythia transformers, which are in turn followed by the RWKV models.
We further replicate this finding for the other model sizes in Figure 3 finding that with the exception of the smallest models (130-170M parameters) where Pythia and RWKV have the same perplexity, at every model size, Mamba has the lowest perplexity, followed by Pythia, followed by RWKV.
Model
Perplexity
Pythia
20
Mamba
RWKV
10
10a5
10a
10
Number of Model Parameters
Figure 3: Comparison of the WikiText perplexity of each model of each architecture. Word- level perplexity of the WikiText-2 test set (Merity et al. 2017) was calculated using the Language Model Evaluation Harness (Gao et al. (002))
25

---

Published as a conference paper at COLM 2024
C
Statistical Analyses
C.1 Fit to N400 data by model size
Dataset
Predictor
Estimate
SE
(10)
P
p (uncor.)
Federmeier et al. 2007
Intercept 0.3139
0.1737
1.8068
1.0000
0.1009
Mamba
-0.5605
0.2459
-2.2797
0.6657
0.0458
RWKV
-0.3979
0.2605
-1.5276
1.0000
0.1576
Scale
-0.9164
0.1078
-8.4972
0.0003
<0.0001
Hubbard et al. 2019
Intercept
0.3005
0.2758
1.0897
1.0000
0.3014
Mamba
-0.7025
0.3904
-1.7997
1.0000
0.1021
RWKV
-0.1738
0.4136
-0.4202
1.0000
0.6832
Scale
-0.7949
0.1712
4.6428
0.0267
0.0009
Michaelov et al. 2024
Intercept
-0.0591
0.4811
-0.1229
1.0000
0.9046
Mamba
-0.1731
0.6809
-0.2543
1.0000
0.8044
RWKV
0.4234
0.7214
0.5869
1.0000
0.5703
Scale
-0,2270
0.2987
-0.7599
1.0000
0.4649
Szewezyk Federmeier 2022)
Intercept
0.4910
0.2899
1.6940
1.0000
0.1211
Mamba
-0.2099
0.4103
-20008
0.9786
0.0733
RWKV
-0.6925
0.4347
-1.5932
1.0000
0.1422
Scale
-0.7424
0.1800
4.1257
0.0484
0.0021
Szewczyk et al.
2022
Intercept
0.3879
0.4560
0.8506
1.0000
0.4149
Mamba
-0.8438
0.6455
-1.3073
1.0000
0.2204
RWKV
-0.3029
0.6838
-0.4429
1.0000
0.6673
Scale
0,2281
0.2831
0.8056
1.0000
0.4392
Wlotko & Federmeier (0022
Intercept 0.3373
0.2122
1.5894
1.0000
0.1431
Mamba
-0.7280
0.3004
-2.4237
0.5711
0.0358
RWKV -0.2705
0.3182
-0.8501
1.0000
0.4152
Scale
-0.8666
0.1317
-6.5779
0.0026
<0.0001
Table Results of statistical analyses on the N400 datasets based on model size (opera- tionalized as number of parameters). Because all variables were z-scored before analysis, the estimate does not directly reflect the difference but helpful as an indication of effect direction-a negative estimate indicates lower AIC, and thus, better fit to the data. The estimate for predictors Mamba and RWKV reflects their effect relative the Pythia models. Scale is operationalized as the logarithm of the number of parameters. We bold predictors that are significant after correction for multiple comparisons (Benjamini & Hochberg 1995). Given the low power of our study (see 44 also italicize variables that are significant before multiple comparisons.
26

---

Published as a conference paper at COLM 2024
C.2 Fit to reading time data by model size
Dataset
Predictor
Estimate
SE
t (10)
P
p (uncor.)
Boyce & Levy (0033)
Intercept
-0.2065
0.1726
-1.1965
1.0000
0.2591
Mamba
0.2558
0.2443
1.0471
1.0000
0.3197
RWKV
0.4031
0.2588
1.5573
1.0000
0.1505
Scale
0.9279
0.1072
8.6588
0.0003
<0.0001
Brothers & Kuperberg 2021
Intercept
0.3774
0.3298
1.1445
1.0000
0.2791
Mamba
-0.7448
0.4668
-1.5956
1.0000
0.1417
RWKV
-0.3900
0.4945
-0.7887
1.0000
0.4486
Scale
-0.7049
0.2047
-3.4225
0.1298
0.0063
Futrell al. (2021
Intercept
-0.2114
0.1511
-1.3997
1.0000
0.1918
Mamba
0.4669
0.2138
2.1841
0.7605
0.0539
RWKV
0.1563
0.2265
0.6902
1.0000
0.5058
Scale
0.9428
0.0938
10.0537
0.0001
<0.0001
Kennedy et al. 2003
Intercept
-0.4226
0.1032
4.0972
0.0484
0.0022
Mamba
0.7710
0.1460
5.2809
0.0110
0.0004
RWKV
0.5155
0.1547
3.3326
0.1441
0.0076
Scale
0,9320
0.0640
14.5533
0.0000
<0.0001
Luke Christianson 2018
Intercept
-0.4129
0.1328
-3.1089
0.1999
0.0111
Mamba
0.7922
0.1880
4.2141
0.0442
0.0018
RWKV
0.4550
0.1992
2.2845
0.6657
0.0454
Scale
0.9165
0,0825
11.1145
<0.0001
<0.0001
Smith & Levy (0013)
Intercept
-0.3274
0.3627
-0.9025
1.0000
0.3880
Mamba
0.3384
0.5134
0.6591
1.0000
0.5247
RWKV
0.7229
0.5439
1.3290
1.0000
0.2134
Scale
0.6377
0.2252
2.8318
0.2931
0.0178
Table 4: Results of statistical analyses on the reading time datasets based on model size (operationalized number of parameters). Because all variables analysis, the estimate does not directly reflect the difference but is helpful as an indication of effect direction-a negative estimate indicates lower ATC, and thus, better fit to the data. The estimate for predictors Mamba and RWKV reflects their effect relative to the Pythia models. Scale is operationalized bold predictors that are significant after correction for multiple comparisons (Benjamini & Hochberg 1995). Given the low power of our study (see $4), we also italicize variables that are significant before multiple comparisons.
z-scored before
the logarithm of the number of parameters. We
27

---

Published as a conference paper at COLM 2024
C.3 Fit to N400 data by model perplexity
Dataset
Predictor
Estimate
(10)
P
p (uncor.)
Federmeier al. (007
Intercept
0.2441
0.1526
1.5991
1.0000
0.1409
Mamba
-0.1333
0.2184
-0.6103
1.0000
0.5553
RWKV
-0.6877
0.2309
-2.9784
0.2359
0.0138
Perplexity
-0.9651
0.0983
-9.8209
0.0001
<0.0001
Hubbard et al. (0199)
Intercept
0.2389
0.2386
1.0013
1.0000
0.3403
Mamba
-0.3204
0.3413
-0.9386
1.0000
0.3701
RWKV
-0.4356
0.3609
-1.2070
1.0000
0.2552
Perplexity
-087710
0.1536
-5.6713
0.0068
0.0002
Michaelov al. 2024
Intercept
-0.0739
0.4882
-0.1513
1.0000
0.8828
Mamba
-0.9944
0.6985
-0.1336
1.0000
0.8963
RWKV
0.3752
0.7386
0.5080
1.0000
0.6225
Perplexity
-0.1624
0.3143
-0.5167
1.0000
0.6166
Szewczyk & Federmeier 2022
Intercept
0.4354
0.2995
1.4538
1.0000
0.1766
Mamba
-0.4841
0.4285
-1.1298
1.0000
0.2849
RWKV
-0.9188
0.4530
-2.0281
0.9611
0.0700
Perplexity
-0.7544
0.1928
-3,1227
0.0623
0.0029
Szewczyk al. 2022)
Intercept
0.4018
0.4657
0.8629
1.0000
0.4084
Mamba
-0.9151
0.6663
-1.3735
1.0000
0.1996
RWKV
+0,6225
0.7045
-0.3726
1.0000
0.7172
Perplexity
0.1371
0.2998
0.4574
1.0000
0.6572
Wlotko & Federmeier (0012)
Intercept
0.2723
0.2288
1.1904
1.0000
0.2614
Mamba
-0.3345
0.3273
-1.0221
1.0000
0.3308
RWKV
-0.5350
0.3461
-1.5459
1.0000
0.1532
Perplexity -0.8816
0.1473
-5.9859
0.0051
0.0001
Table 5: Results statistical analyses based on model perplexity. Because variables were z-scored before analysis, the estimate does not directly reflect the difference but is helpful as an indication of effect direction a negative estimate indicates lower AIC, and thus, better fit to the data. The estimate for predictors Mamba and RWKV reflects their effect relative to the Pythia models. Perplexity is operationalized as negative log-perplexity in order to preserve the relationship of the other variables (where negative indicates a better fit). We bold predictors that are sigmificant after correction for multiple comparisons (Benjamini Hochberg 1995). Given the low power of our study (see $4), we also italicize variables that are signiticant before multiple comparisons.
28

---

Published as a conference paper at COLM 2024
C.4 Fit to reading time data by model perplexity
Dataset
Predictor
Estimate
SE
t (10)
p (uncor.)
Boyce & Levy 2023
Intercept
-0.1385
0.2406
-0.5754
1.0000
0.5777
Mamba
-0.1504
0.3443
-0.4369
1.0000
0.6715
RWKV
0.6726
0.3640
1.8479
1.0000
0.0944
Perplexity
0.8996
0.1549
5,8072
0.0060
0.0002
Brothers & Kuperberg 2021
Intercept
0.3219
0.2891
1.1134
1.0000
0.2916
Mamba
-0.3969
0.4136
-0.9596
1.0000
0.3599
RWKV
-0.6304
0.4373
-1.4416
1.0000
0.1800
Perplexity
-0.7990
0.1861
-429932
0.0410
0.0016
Futrell et al. 2021
Intercept
-0.1406
0.1704
+0,8250
1.0000
0.4286
Mamba
0.0371
0.2438
0.1521
1.0000
0.8821
RWKV
0.4457
0.2578
1.7290
1.0000
0.1145
Perplexity
0.9643
0.1097
8.7906
0.0003
0.0001
Kennedy et al. 2003)
Intercept
003516
0.0518
-6.7941
0.0021
<0.0001
Mamba
0.3359
0.0740
4.5363
0.0297
0.0011
RWKV
0.8108
0.0783
10.3565
0.0001
<0.0001
Perplexity
0,9833
0.0333
29.5132
<0.0001
<0,0001
Luke & Christianson 2018
Infercept
-0.3438
0.143
-2.4040
0.5722
0.0371
Mamba
0.3721
0.2046
1,8182
1.0000
0.0991
RWKV
0,7383
0.2164
3.4126
0.1310
0.0066
Perplexity
0.9441
0.0921
10.2535
0.0001
<0.0001
Smith & Levy (2013)
Intercept
-0.2781
0.3479
-0,7994
1.0000
0.4427
Mamba
0.0336
0.4978
0.0676
1.0000
0.9475
RWKV
0.9313
0.5263
1.7696
1.0000
0.1072
Perplexity 0.6934
0.2240
3.0960
0.1999
0.0113
Table 6: Results statistical analyses based on model perplexity. Because variables were z-scored before analysis, the estimate does not directly reflect the difference but is helpful as an indication of effect direction a negative estimate indicates lower AIC, and thus, better fit to the data. The estimate for predictors Mamba and RWKV reflects their effect relative to the Pythia models. Perplexity is operationalized as negative log-perplexity in order to preserve the relationship of the other variables (where negative indicates a better fit). We bold predictors that are sigmificant after correction for multiple comparisons (Benjamini Hochberg 1995). Given the low power of our study (see $4), we also italicize variables that are signiticant before multiple comparisons.
29


Reviews:
[{"review_id": "p0xk0oTQgp", "reviewer": "Reviewer_sgiH", "summary": "This is a well written paper with a clear structure. The paper is well motivated with clear arguments. The paper also presents a good amount of empirical evidence to back the conclusions.\n\nThe paper is quite original as it shows evidence that RNNs can outperform transformers in human language compression. However, in terms of technicality, the paper doesn't proposal noticeable novelty.", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback. We address the main concerns below:\n\n\nCognitive Plausibility and Context Windows:\n\nWe thank the reviewer for noting this, and agree that limited context lengths are only an issue in practice (and one that is likely to be reduced/overcome as context windows continue to increase). We also agree that human memory does have limitations—we currently discuss what kinds of limitations exist and what this means for the recurrent model vs. transformer comparison in paragraph 2 of page 3. We will expand upon both points in the revised paper.\n\n\nModel Selection:\n\nWe explain model selection in the final sentence of Section 1 and the first paragraph of Section 3: all models are trained on the same dataset (The Pile), and there are 4 models of each architecture that have roughly the same number of parameters, which we show in Table 1. This allows us to compare models of different architectures while keeping the dataset constant and accounting for differences in number of model parameters, which is the main aim of the paper. We thank the reviewer for identifying this as something that could be made clearer, and will emphasize this further in the revised manuscript.\n\nCurrently there are no other comparable publicly available models (trained on exactly the same dataset and of the same sizes) that are publicly available apart from the GPT-Neo models. The Pythia models are a more recent and more-optimized version of the GPT-Neo models, and so we chose them to give transformers the best chance in the comparison across architectures.\n\n\nArchitecture vs. Optimization:\n\nOur aim in this study was to compare the performance of models across architectures, and (apart from GPT-Neo) there are no other comparable models of any of the three architectures. For this reason, we focused in Section 3 on the main differences between the architectures; however, if the reviewer believes that there are specific additional details that would be important to include in the paper, we would appreciate any clarification. We will mention in the revised paper that the full details of all models and their training are provided in the original papers and the linked Github repositories. \n\nFinally, we are not aware of any simple tricks that one can use to improve a model’s performance at modeling human language comprehension, and would be grateful for any more information that the reviewer could provide on this.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I enjoyed reading the paper. The paper presents well the background and the literature.", "reasons_to_reject": "I'm skeptical about some points in the paper. Firstly, the authors argue that one reason why transformers are less cognitively plausible is their finite context window. I believe that the finite context window is only a limitation in practice because in theory, one can choose the window size as large as possible. Besides, isn't that human memory has some certain limitation? \n\nMy second concern is the models chosen in the experiments. The authors specifically chose Mamba and RWKV because they have been shown to be very fast and perform on par with transformers. However, there are several variations of transformers yet the authors didn't give a good argument why Pythia was picked. \n\nAnother related concern is about how the chosen models were trained. The paper seems lack that details. Besides, the authors draw conclusions without disentangling model architecture and model optimization. In fact, with reasonable optimization (e.g. some simple tricks), one can improve a model performance.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A5YAJt24uR", "reviewer": "Reviewer_WYrb", "summary": "The paper evaluates a recent crop of recurrent language models on predicting the N400 response and reading times, finding that the modern recurrent models perform on par with Transformers on this task, updating previous old findings that Transformers outperform LSTMs.", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback. We address the main concerns below:\n\nTheoretical Implications:\n\nWe agree that work is needed on exactly which features of transformers and contemporary recurrent models affect performance at predicting metrics of human language processing. As the reviewer says, this work would require pre-training new (sometimes ablated) versions of Transformer and recurrent models. We view the primary value of the current study as setting the stage for this line of work. For the first time, it compares how well parameter- and training data-matched transformers, Mamba, and RWKV models can be used to predict human human metrics.\n\nWe also agree that it would be valuable to expand our discussion of the theoretical stakes and implications of the study. To this end, we thank the reviewer for highlighting the cue-based retrieval account of transformers in Ryu & Lewis (2021). We also plan to expand our discussion of previous empirical work, noting that while Transformers have often been assumed to perform better than RNNs (including LSTMs) at modeling human language comprehension, this has not always been the case (Eisape et al., 2020; Merkx & Frank, 2021; and Kuribayashi et al., 2021). We also plan to expand our discussion (see response to reviewer RQJH) and suggest some theoretically-motivated avenues for future research, e.g. running analyses similar to those in Arehalli and Linzen (2020) or Ryu and Lewis (2021).\n\nMeasuring performance:\n\nFor each dataset, all regressions have the same structure, matching previous work as closely as possible. Given this (and the differences between the metrics), we do not believe that using the regressions with the same structure across all datasets is appropriate. However, we welcome the reviewer’s suggestions about whether calculating (R)MSE or delta-log-likelihood on a test set would nonetheless be informative.\n\nPrevious work:\n\nWe thank the reviewer for highlighting these studies, which we will add.\n\nDataset Selection:\n\nOur dataset selection goal was broad coverage of experimental paradigms and metrics—in addition to the N400 studies, we included one eye-tracking, one self-paced reading, and one Maze study. In the revised manuscript, we will follow the reviewer’s suggestion to expand our analyses, adding the Dundee, the Natural Stories SPR, and Smith & Levy (2013) datasets. Having now run the analyses, these datasets show the same general pattern as the Boyce & Levy (2023) and Provo Corpus datasets.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper makes an important contribution to an ongoing research program about the match between neural network architectures and human language processing, based on the surprisal linking mechanism.\n- The experiments are straightforward and their main result (modern recurrent architectures can match Transformers or exceed them) is clear.\n- The analyses involving model size and perplexity are well-motivated and the results are interesting.", "reasons_to_reject": "I ultimately felt dissatisfied by this paper because I wanted more, both in terms of results/experiments and in terms of cognitive upshots. The current paper feels like an incremental progress update, although the beginning of something better. I think that is probably enough for publication in CoLM, but I have concerns below:\n\nWhen comparing neural architectures on their ability to predict N400/RT, the question is always what we are learning about human language processing from the comparison. I don't feel like the current paper sheds much light on this yet. What's on my mind is: What is it exactly about Transformers that made them do better than LSTMs, and what has changed about the new recurrent models that make them match Transformers? Is it the better gating mechanism in RWKV? Or is it the restriction to linear dynamics as in Mamba? I think to get real cognitive insights, more models would have to be compared, including some custom models (like, various ablated versions of Mamba). In particular, Mamba could be compared to its predecessors like S4. It would also be necessary to give LSTMs a fair shot. The LLM field basically stopped training LSTMs: how would they perform if trained on the same data and with the same large number of parameters as the modern Transformers? A comparison with n-gram models trained on large datasets would also be informative. Basically, I feel like deriving real cognitive insights would require a much more thorough comparison than what is here.\n\nEven given the current results, the discussion about cognitive upshots is sparse. What are we to make of the relationship between scaling direction and RWKV vs. Mamba performance? If recurrent models really are just as good as Transformers, what are we to make of the arguments that Transformers act like cue-based retrieval models---was it just a spurious connection? There are rich cognitive implications that aren't explored.\n\nOther issues\n\n- Measuring performance by AIC is not as good as measuring performance by delta-log-likelihood on heldout data. AIC is just log likelihood with some complexity penalties. These complexity penalties would ideally be the same across different datasets, reflecting use of the same model; and if different models were used, the complexity penalties are only crude measures of the actual complexities of these models. I think pure delta-log-likelihood based on maximally similar regression models and heldout data would be the best evaluation.\n\n- Coverage of the literature is incomplete. The most important omission in the literature review is is Ryu & Lewis (2021) who argue that Transformers directly parallel the cue-based retrieval models that are common in psycholinguistics, in contravention of the general wisdom that recurrent models are more cognitively realistic. Two other omissions are Wilcox et al. (2023, TACL) and Xu et al. (2023, EMNLP) which look at the surprisal-reading time relationship cross linguistically.", "questions_to_authors": "- What is the motivation for the datasets used? Why not use Natural Stories SPR in addition to Natural Stories Maze, or any other eyetracking corpus like Dundee?", "extra_scores": {}}, {"review_id": "GZYkiu0dqI", "reviewer": "Reviewer_RQJH", "summary": "An empirical comparison of how different LM architectures (Transformers vs. two latest-generation recurrent models, RWKV and Mamba) predict various metrics of language processing difficulty by human subjects.\n\nThe paper starts from the somewhat suprising observation that traditional RNNs have been superseded by a less cognitively plausible architecture (Transformers) not only in a range for NLP tasks but also as predictors of behavioral metrics of language comprehension by humans.\nIt asks whether latest-generation RNNs would, in turn, fair better than Transformers in this context.\nThe comparison is performed on a suite of 9 language processing datasets using models of different architectures and sizes, all trained on the same corpus.\n\nResults show a complex picture whereby different architectures are best across different datasets and model sizes. \nAlso, an earlier finding is confirmed by which processing difficulty tends to be better predicted by models that have better language modeling performance (i.e. perplexity) in general.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback. We address the main concerns below:\n\nInconsistent results:\n\nWe agree that the results are not consistent across all datasets.\n\n\nLimitations in Discussion and Conclusion:\n\nWe will substantially expand our discussion section in our revision, in which we will argue that:\n\n(1) Overall performance at predicting metrics of human language processing is similar across architectures.\n\n(2) There is a general tendency of inverse scaling between model size (or perplexity) and performance at predicting reading time, but positive scaling between model size (or perplexity) and N400 amplitude.\n\n(3) These scaling patterns are the same within architectures.\n\n(4) Some of the differences between architectures on specific datasets may be partly accounted for by general model performance—Mamba models have the lowest perplexity (i.e. are best at the pretraining task of autoregressive language modeling), and tend to perform relatively better on positive scaling datasets and relatively worse on inverse scaling datasets when considering how model parameter count impacts performance (see Figs 1A and 2A). Crucially, this pattern disappears when we focus on how model perplexity impacts performance (see Figs 1B and 2B).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- timely and thorough evaluation of post-Transformer recurrent architectures on the task of predicting human processing difficulty\n- no less than 9 datasets and four different metrics of processing difficulty are included in the study\n- models are compared across different sizes to give a more complete picture", "reasons_to_reject": "- The results themselves are rather inconsistent and the overall conclusions a bit vague\n- No attempt is made at explaining why some architectures may be outperforming others in different conditions", "questions_to_authors": "Clarity improvements:\n- AIC -> spell out acronym\n- For clarity, I'd give an acronym/ID to each dataset in Table 2 and use those acronyms/IDs for the discussion in Section 4. All those citations make it hard to follow the text\n- Sect. 4.1 could end with some cleared take-aways. Overall, what have you learned from this experiment?\n\nTypos:\n- identifies filters the input\n- initialized in based\n- In thus study\n- again see -> we again see (sect 4.2)", "extra_scores": {}}, {"review_id": "HwM3BoYZQw", "reviewer": "Reviewer_LzzU", "summary": "This paper compares performances of the state of the art recurrent\nneural networks RWKX and Mamba, with that of a transformer Pythia on 9\nfrom a human oriented perspective, through 9 datasets by use of 4\nmetrics of human language processing.\n\nThe method is by computing the surprisal.  The authors empirically\nshow that Pythia does not necessarily outperform RWKX and Mamba,\nrather the contrary. The authors argue that RNNs perform better than\nTransformer, from a human-oriented perspective.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback. We address the main concerns below:\n\nFairness in model scales:\n\nWe agree with the reviewer that using number of parameters to measure scale may favor some model architectures over others. This is why we also include perplexity to measure scale (Figs. 1B & 2B). Perplexity gives a measure of how good each model is overall, which allows for a comparison across architectures that is not impacted by how effective models are at using parameters.\n\n\nFocus on human cognition only:\n\nWe are not sure that we fully understand the reviewer’s concern about not including performance on other types of tasks given the topic of the paper (comparing the performance of models of each architecture at predicting metrics of human language comprehension) and the submission track (“Human mind, brain, philosophy, laws and LMs”), and would appreciate additional clarification the reviewer could give. In the revised paper, we will make note that in the original Mamba paper, Gu & Dao (2023; Table 3) include a comparison of the performance of Mamba models at various tasks in comparison to other models with similar numbers of parameters (including RWKV).\n\n\nQ: “Did you compare the three systems on the same data when using the maximum number of parameters available for each system? If results do you think you would get?”\n\nWe used the largest Mamba models available. For the other architectures, we picked all available sizes of comparable number of parameters to the Mamba models available, which were also trained on the Pile (see response to Reviewer sgiH above for additional details). As for the largest models of the other two architectures, we would expect even better performance on the datasets that show positive scaling (for example, Michaelov et al., 2022, cited in paper, find that GPT-3 predicts N400 amplitude better than smaller transformers), and worse performance on the datasets on which we see inverse scaling.\n\n\nQ: “Did you compare the results for other data that are not oriented to human cognition? If you do that, what do you think you would get?”\n\nIn the submitted version of the paper, we calculate model perplexity on WikiText-103, and find that, at each size (number of parameters), Mamba performs best (has the lowest perplexity). Except at the smallest model size when they perform the same, Pythia performs better than RWKV at this task. As noted above, comparisons on a range of tasks can be found in Gu & Dao (2023; Table 3).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The problem setting is interesting that is to compare the capacity of RNN\nagainst Transformer how they perform differently for human cognitive tasks.", "reasons_to_reject": "There are two large concerns of this paper.\n\n1. \n\nAccording to Table 1, the authors compare three models under\ncomparable settings. The number of parameters for every experiment is\ncomparable by scale.  Although I understand this is one \"fair\" way of\nsetting, I wonder whether Transformer's capability is fully exploited\nthis way.\n\nTransformer performs better because it is able to accomodate larger\nnumber of parameters than RNN, that is, O(n^2) vs. O(n), respectively,\nfor n being the length of the sequence.  Transformer architecture\nincludes that of RNN.\n\nThe present result shows that if Tranformer is trained on a system\nwith equivalent number of parameters of RNN, then Transformer (could)\nperform worse than RNN. This is trivial to me, because Transformer\nuses parameters to extract relation among tokens that cannot be\ncaptured by RNN. Therefore for Transformer to perform fully, it must\nuse much more parameters than RNN.\n\n2. \n\nThe authors focus only on human cognition.\n\nIf comparing under the same setting of Table 1, the data does not have\nto be be restricted to human cognition. I guess even for a general\nevaluation dataset, the same comparison of Transformer vs. RNN can be\nmade and presumably the same conclusion could be obtained.\n\nIn other words, since the authors do not show the evaluation results\nfor other evaluation dataset, the value of this work is not well\nhighlighted.", "questions_to_authors": "-Did you compare the three systems on the same data when using the\nmaximum number of parameters available for each system?  If results do\nyou think you would get?\n\n-Did you compare the results for other data that are not oriented to\nhuman cognition?  If you do that, what do you think you would get?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper evaluates two recent recurrent models, in comparison with a transformer, on predicting measures of human language comprehension. Results suggest that -- for the specific models (with the same training data domain and comparable parameter counts), recurrent architectures show similar performance to transformers, sometimes exceeding them.

Despite discussion, this remained a controversial paper, with ratings ranging from 4 to 8. Having a strong background in this field, I believe that the weaknesses raised by sgiH (who provided the only rejection score) are satisfactorily addressed by the authors, even though sgiH did not raise their score. 

Taken all reviews and my reading of the paper into consideration, I believe this to be a timely and worthwhile contribution to the sub-field evaluating language models as predictors of human language comprehension measurement.

[At least one review was discounted during the decision process due to quality]
Decision: Accept