Revenge of the Fallen? Recurrent Models Match Transformers at Predicting Human Language Comprehension Metrics

Authors: james a michaelov
catherine  arnett
benjamin k bergen
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-23 05:14:27

Abstract: Transformers have generally supplanted recurrent neural networks as the dominant architecture for both natural language processing tasks and for modelling the effect of predictability on online human language comprehension. However, two recently developed recurrent model architectures, RWKV and Mamba, appear to perform natural language tasks comparably to or better than transformers of equivalent scale. In this paper, we show that contemporary recurrent models are now also able to match—and in some cases, exceed—performance of comparably sized transformers at modeling online human language comprehension. This suggests that transformer language models are not uniquely suited to this task, and opens up new directions for debates about the extent to which architectural features of language models make them better or worse models of human language comprehension.

Main paper: 
 introduction: 
 The origins of recurrent neural networks lie in attempts to model human cognition, and specifically the human language system (Jordan, 1986; Elman, 1990) . Following improvements such as long short-term memory (LSTM; Hochreiter & Schmidhuber, 1997; Gers et al., 2000) , recurrent neural networks were for a while the dominant architecture not only for modeling human language comprehension (e.g. Frank et al., 2015) , but for natural language systems in general (see, e.g. Goldberg, 2016) . In recent years, they have in turn been superseded by transformer language models, which empirically tend to show better performance at both a range of natural language tasks (see, e.g., Radford et al., 2019; Dai et al., 2019) and at predicting metrics of human language comprehension (e.g. Wilcox et al., 2020; Merkx & Frank, 2021; Michaelov et al., 2022) . Nonetheless, the question of how recurrent and transformer language models compare as cognitive models of the human language system is still an open one. One the one hand, recurrent neural networks inherently model the process of maintaining a specific informational state and integrating this with new information as it occurs incrementally. This principle is widely believed to underlie language comprehension and other real-time processing (Merkx & Frank, 2021; Michaelov et al., 2021) . On the other hand, transformers have been argued to better model cue-based retrieval accounts of language comprehension (Ryu & Lewis, 2021; Merkx & Frank, 2021) , and their direct access to previous words may allow them to better model human-like lexical priming effects (Michaelov et al., 2021) . In addition, transformers' superior performance at predicting metrics of human language comprehension in itself serves as evidence that, at the very least, the statistical patterns learned by transformer language models capture something also learned by humans.As they have increased in scale (number of parameters, number of training tokens, or both), transformers have been found to improve at natural language tasks (Brown et al., 2020; Kaplan et al., 2020; Rae et al., 2022; Hoffmann et al., 2022; Chowdhery et al., 2022; Touvron et al., 2023) , as well as at predicting both behavioral (Wilcox et al., 2020; Merkx & Frank, 2021) and neural (Merkx & Frank, 2021; Michaelov et al., 2022) metrics of online language comprehension. But in recent years, two wrinkles have emerged. The first is evidence that larger models and those trained on more data may actually predict some behavioral metrics of language comprehension (such as reading time) worse than smaller models do (Kuribayashi et al., 2021; Oh et al., 2022; Oh & Schuler, 2023a; b; Oh et al., 2024; Shain et al., 2024) . Second, two recently developed recurrent language model architectures appear to perform natural language tasks at least as well as transformers of equivalent size and training: RWKV (Peng et al., 2023) and Mamba (Gu & Dao, 2023) . Transformers are therefore no longer the definitively best-performing language model architecture, and it is no longer the case that we should expect further advances in transformers to necessarily lead to improved fit to metrics of human language comprehension. Thus the time is ripe to revisit the question of which language model architecture best predicts human language comprehension.To this end, we compare the performance of the Pythia (Biderman et al., 2023) , RWKV (Peng et al., 2023) , and Mamba (Gu & Dao, 2023) suites of autoregressive language models on 12 human language comprehension datasets (Federmeier et al., 2007; Hubbard et al., 2019; Michaelov et al., 2024; Szewczyk & Federmeier, 2022; Szewczyk et al., 2022; Wlotko & Federmeier, 2012; Boyce & Levy, 2023; Brothers & Kuperberg, 2021; Futrell et al., 2021; Kennedy et al., 2003; Luke & Christianson, 2018; Smith & Levy, 2013) covering 5 different metrics. Since all models were trained on the same dataset and have comparable numbers of parameters, we are able to measure the effect of architecture on the extent to which a language model's predictions correlate with metrics of human language comprehension. 
modeling prediction in human language comprehension: 
 Over the years, a wide range of language models have been used to model data from experiments on human language comprehension, including n-gram models (e.g., McDonald & Shillcock, 2003; Boston et al., 2008; Demberg & Keller, 2008; Mitchell et al., 2010; Smith & Levy, 2013; Brothers & Kuperberg, 2021) , recurrent neural networks (RNNs; e.g., Frank & Bod, 2011; Fossum & Levy, 2012; Monsalve et al., 2012; Frank et al., 2015; Goodkind & Bicknell, 2018; Aurnhammer & Frank, 2019) , and most recently, transformers (e.g., Wilcox et al., 2020; Hao et al., 2020; Merkx & Frank, 2021; Kuribayashi et al., 2021; Szewczyk & Federmeier, 2022; Michaelov et al., 2022; 2024; Boyce & Levy, 2023; Wilcox et al., 2023a; b; Oh et al., 2022; 2024; Oh & Schuler, 2023a; b; Shain et al., 2024) . Each approach can be evaluated in terms of how well it performs either as a computational-level model (in the vein of Marr, 1982) or as a cognitive model. Language models can serve as computational-level models since they calculate the probability of a word in a given context; and thus, their predictions can be compared with analogous measures of prediction in humans. Humans may also be able to predict the probability of words based on the statistics of language, given evidence that they are sensitive to statistical properties of language such as word frequency (Van Petten & Kutas, 1990; Van Petten, 1993; Dambacher et al., 2006; Rugg, 1990; Fischer-Baum et al., 2014; Shain, 2024) . This opens a door for thinking of language models as plausible cognitive models, something further supported by recent work arguing that language models display linguistic competence (Piantadosi, 2023; Mahowald et al., 2024) .Existing studies vary in where they fall along the computational-to-cognitive model continuum. At the computational end of the scale, a range of studies (Smith & Levy, 2013; Brothers & Kuperberg, 2021; Meister et al., 2021; Wilcox et al., 2023b; Hoover et al., 2023; Shain et al., 2024; Michaelov & Bergen, 2022; 2024) have focused on understanding the mathematical relationship between language model probability and processing difficulty, which does not necessarily require considering the specific model features. On the other end of the scale, several researchers have argued that contemporary transformers have a strong structural resemblance to the human language system (Schrimpf et al., 2021; Hosseini et al., 2024) .Most studies fall somewhere between the two extremes. This is particularly evidenced in work on the N400, a neural signal that is often considered to index the extent to which a word has been predicted based on its preceding context (DeLong et al., 2005; Van Petten & Luka, 2012; DeLong et al., 2014; Kuperberg et al., 2020) . Frank & Willems (2017) , for example, explicitly choose to use a model with a modified n-gram architecture to investigate the role that pure word-level surface-level statistics may have on language comprehension. More recently, Michaelov et al. (2024) used GPT-3 to investigate the extent to which prediction based on language statistics alone could account for several known N400 effects, including the fact that more semantically plausible words are processed more easily. We can also use this approach to test the viability of specific hypotheses about the language comprehension system by comparing language models that instantiate different theories of language comprehension. Frank et al. (2015) , for example, investigate how well a traditional recurrent neural network predicts N400 amplitude compared to a model implementing a probabilistic phrase-structure grammar. They find that the former out-performs the latter, which they argue suggests that prediction during human language comprehension may rely more on statistical properties of language than on explicit hierarchical grammatical structure.In the present study, we focus on the difference between recurrent language models and transformers. Many accounts theorize that human language comprehension involves construction of lossy and in some cases incorrect representations of the utterance and its meaning, due to cognitive resource limitations (Ferreira, 2003; Christiansen & Chater, 2016; Futrell et al., 2020) , and it has been argued that recurrent models are a good computational models of this (Merkx & Frank, 2021; Michaelov et al., 2021) . This is perhaps most clear in the case of the now-or-never bottleneck account of language comprehension, under which our limited working memory means that 'the brain must compress and recode linguistic input as rapidly as possible' (Christiansen & Chater, 2016) . In this case, the analogy with recurrent models is clear-a core component of such models is that they can take inputs of any length, but are limited in that any context must be compressed into a representation with a fixed size, which is updated with each new word. This is not the case for transformers, however. While they have fixed (though ever-increasing, see, e.g., Llama Team, 2024) context windows, they have perfect access to all the representations of words within these context windows. Thus, their representations of a word's context are not incrementally-compressed versions of the input like recurrent models; but rather, their representations of the context expand with each new word-within their context window, they have 'unlimited working memory' (Merkx & Frank, 2021) , which puts them at odds with accounts such as the now-or-never bottleneck theory of language comprehension. On the other hand, such seemingly lossless working memory may be more human-like than it may first appear. As Michaelov et al. (2021) note, humans do maintain specific past words in working memory, and indeed, there is evidence that reading a given word can lead to that word being easier to process for up to 45 minutes in some specific contexts (Besson et al., 1992;  for discussion see Rommers & Federmeier, 2018) . Furthermore, transformers have been argued to provide a good computational-level model of cue-based retrieval accounts of language comprehension (Ryu & Lewis, 2021; Merkx & Frank, 2021) . Specifically, under cue-based retrieval accounts (McElree et al., 2003; Van Dyke & Lewis, 2003;  for review see Lewis et al., 2006; Parker et al., 2017) , words are retrieved during language comprehension based on the features of previous words in the context which are used as cues; and analogously, it has been argued, the features of the representations of the words that transformers attend to when they predict the next word can be considered to function as cues to which word should be predicted (Ryu & Lewis, 2021; Merkx & Frank, 2021) .Beyond a priori cognitive plausibility, empirical studies with N400 data have almost universally shown that transformers out-perform recurrent neural networks, and larger transformers trained on more data (and with lower perplexities) generally perform best at predicting N400 amplitude (Merkx & Frank, 2021; Michaelov et al., 2022; Michaelov & Bergen, 2022) .Language models have also been used to model reading time, which has also been hypothesized to reflect prediction in language comprehension. However, in this area, the results have been less straightforward. Smaller models display the same pattern seen with the N400, where larger language models trained on more data and with lower perplexities perform better (Goodkind & Bicknell, 2018; Merkx & Frank, 2021; Wilcox et al., 2020; 2023a; Hao et al., 2020) . But past a certain number of parameters or training tokens, their performance appears to deteriorate (Kuribayashi et al., 2021; Oh et al., 2022; 2024; Oh & Schuler, 2023a; b; Shain et al., 2024) . On the question of whether recurrent neural networks or transformers best predict reading time, the results have been mixed (Wilcox et al., 2020; Eisape et al., 2020; Kuribayashi et al., 2021) , and have been found to differ depending on which metric of reading time is investigated (Merkx & Frank, 2021) .Name Parameters Name Parameters Name Parameters 169M 169,342,464 160M 162,322,944 130M 129,135,360 430M 430,397,440 410M 405,334,016 370M 371,516 Further details for each model are provided in the cited papers and their linked repositories.The advent of new recurrent architectures that are increasingly feasible to train at a large scale and that can perform as well as or better than transformers-namely, RWKV and Mambais thus important in two ways. First, it allows us to test whether the patterns previously observed in transformers-that larger and better models predict N400 amplitude better but past a certain point predict reading time worse-also holds for other architectures with comparable natural language processing performance. Second, and perhaps more crucially, it allows us to again evaluate whether, when matched on scale or performance, recurrent or transformer architectures are better models of online human language comprehension. 
language model architectures: 
 The aim of this study is to investigate how well metrics of online human language comprehension can be predicted using three types of language model: the Pythia suite of autoregressive transformers (Biderman et al., 2023) ; and the recurrent RWKV (Peng et al., 2023) and Mamba models (Gu & Dao, 2023) . All models are trained on the Pile, a 300B token English-language dataset (Gao et al., 2020) . For each architecture, we selected models of comparable size (i.e., weight class) as shown in Table 1 . We discuss each architecture below.Pythia Pythia (Biderman et al., 2023) is a set of autoregressive transformer models trained to be comparable across different model sizes, ranging from 70M to 12B parameters. The architecture and hyperparameters are based on GPT-3 (Brown et al., 2020) , with the addition of some changes based on recent advancements (Dao et al., 2022; Su et al., 2024; Wang & Komatsuzaki, 2021; Belrose et al., 2023) .RWKV RWKV is a language model architecture described by its creators as a 'Reinvent[ion of the] RNN for the Transformer Era' (Peng et al., 2023) . RWKV models combine the parallelizable training of transformers with unlimited context lengths, as well as several additional features that make them RNN-like. First, their time-mixing block-which can mathematically formulated in a similar way to the recurrent states of an RNN (Peng et al., 2023) -allows the representations of past states to be combined with those of new words.In addition, RWKV models explicitly have a decay parameter such that tokens earlier in the context will be weighted less than later tokens during inference, thereby explicitly introducing something analogous to working memory limitations (Merkx & Frank, 2021) .Mamba Mamba is another recent recurrent model architecture (Gu & Dao, 2023) . One of the key goals of the Mamba architecture is to allow models to optimally compress their contexts, and especially very long contexts, into a state of fixed size such that they are still able to predict effectively. Like RWKV, Mamba computational complexity scales linearly with sequence length while avoiding the quadratic complexity of transformers (Gu & Dao, 2023) . This is achieved by using a novel 'selective scan' mechanism that filters the input to select the most important information. Table 2 : A description of each of the datasets, including the metric, the number of stimuli, the number of experimental participants (N), and the number of trials. See §3.2 for a brief explanation of each metric, and Appendix A for further details of each dataset.more recent recurrent neural network variants-crucially, they include a latent state that is updated with each new input (like recurrent layers), and their selective scan method filters input (much like gating mechanisms in gated recurrent units or long short-term memory).In this study, we use language models of each of the three architectures discussed in §3.1 to model 5 metrics of human language processing from 12 datasets, the details of which are given in Table 2 . These datasets comprise 6 N400 datasets (Federmeier et al., 2007; Hubbard et al., 2019; Michaelov et al., 2024; Szewczyk & Federmeier, 2022; Szewczyk et al., 2022; Wlotko & Federmeier, 2012) and 6 reading time datasets. The latter comprise four types of reading time metric: the interval between when a word is first fixated by a reader and when they first move onto the next word, as calculated using eye-tracking (Go-Past Duration or GPD; Kennedy et al., 2003; Luke & Christianson, 2018) , the time taken to click to move onto the next word in a self-paced reading task (Self-Paced Reading Response Time or SPR RT; Futrell et al., 2021; Smith & Levy, 2013) , the total Response Time for a word and the two following words (to account for spillover effects) in a self-paced reading task (Self-Paced Reading Three-Word Response Time or 3W-RT; Brothers & Kuperberg, 2021) , and the time taken to respond to each word on the Maze task (Maze Response Time or Maze RT; Boyce & Levy, 2023) . Further details of each metric and dataset are provided in Appendix A.We used the language models discussed in §3.1 to calculate the surprisal of all critical words in all datasets given their context. For the N400 and the Brothers & Kuperberg (2021) datasets, this context was made up of the preceding words in the same sentence. In the remaining datasets (Luke & Christianson, 2018; Boyce & Levy, 2023) , we included the whole preceding passage, comprising multiple sentences. For critical words made up of multiple tokens, surprisal was calculated as the sum of all the sequential tokens comprising them.We ran regression analyses for each dataset using linear mixed-effects regression models, predicting each human language comprehension metric using the surprisal calculated using each language model, as well as baseline covariates and random effects structures as described in Appendix A. For each regression, we calculate the Akaike Information Criterion (AIC; Akaike, 1973) , a measure of how well a regression fits the data, with a lower AIC indicating a better fit. All language models were run in Python (Van Rossum & Drake, 2009) using the transformers (Wolf et al., 2020)  
n400 datasets: 
 Scale impacts model performance at a range of tasks, so we consider the differences between models while accounting for scale. Following previous work (e.g., Oh & Schuler, 2023a) , we consider differences between models when accounting for model size and for model perplexity. While the two are generally correlated, perplexity can help explain the effect of model size. Better models might align better with metrics of human language comprehension given our own powerful predictive capabilities (Monsalve et al., 2012; Goodkind & Bicknell, 2018; Michaelov et al., 2022) , but by the same token, language models may learn to predict words too well to model human language comprehension (Oh et al., 2024) .We first consider the results arranged by model size (Figure 1A ). Overall, we find that in most cases, Mamba and RWKV performance is better than that of Pythia, and Mamba is also better than RWKV. On the Federmeier et al. (2007) Michaelov et al., 2022; Michaelov & Bergen, 2022) , we see that 4 of the 6 datasets (Federmeier et al., 2007; Hubbard et al., 2019; Szewczyk & Federmeier, 2022; Wlotko & Federmeier, 2012) show positive scaling effects-larger models tend to fit the data better.In order to test how robust these patterns are, we run ordinary least-squares linear models for each dataset, predicting the AIC of the linear mixed-effects regressions based on language model scale and model architecture (Pythia, Mamba, or RWKV). After correction for multiple comparisons (Benjamini & Yekutieli, 2001) , we see that model scale is a significant predictor of AIC, with surprisals calculated from larger models fitting the N400 data from 4 of the 6 datasets (Federmeier et al., 2007; Hubbard et al., 2019; Szewczyk & Federmeier, 2022; Wlotko & Federmeier, 2012) significantly better than smaller models. Given the low power of our analysis (only 14 observations per dataset), it is also worth noting that before correction for multiple comparisons, Mamba models produce surprisals that fit the N400 data significantly better than Pythia models on the Federmeier et al. (2007) and Wlotko & Federmeier (2012) datasets. While these latter results are suggestive rather than conclusive, they are consistent with the patterns observed in Figure 1A . The full results of our statistical analyses are provided in Table 3 .Next, we consider the results arranged by model perplexity (Figure 1B ). Within each architecture, there is no difference in pattern depending on whether we order language models by size or perplexity. However, we do see a difference across architectures. In the four datasets that show positive scaling as a function of model size (larger models predict N400 amplitude better), when arranged by perplexity, Mamba models appear to perform worse relative to the other model architectures than they do when arranged by model size, while RWKV models appear to perform better. Conversely, on the dataset where two recurrent models show negative scaling (Szewczyk et al., 2022) , we see the opposite pattern-Mamba appears to perform better, and RWKV appears to perform worse.When we run ordinary least-squares linear models predicting AIC based on model perplexity and architecture, we see a similar effect to that seen for size. After correction for multiple comparisons, better language models (i.e., those with a lower perplexity) produce surprisals that better fit the N400 data on half of the datasets (Federmeier et al., 2007; Hubbard et al., 2019; Wlotko & Federmeier, 2012) . The Szewczyk & Federmeier (2022) dataset also shows this pattern before correction. Full results of our statistical analyses are provided in Table 5 .For the behavioral reading data, we again first look at the data arranged by model size (Figure 2A ). The clearest effects are seen on the eye-tracking datasets (Kennedy et al., 2003; Luke & Christianson, 2018) , where Pythia outperforms (or in one case, performs equally as well as the better of) Mamba and RWKV at all sizes. We also see that Pythia tends to perform best overall on two of the other datasets (Boyce & Levy, 2023; Futrell et al., 2021) , with either Mamba or RWKV performing better at one or two sizes. The clearest exception to this pattern is the Brothers & Kuperberg (2021) dataset, where Mamba and RWKV outperform Pythia at all but one size. We also see a different scaling pattern-unlike the the 5 datasets (Boyce & Levy, 2023; Futrell et al., 2018; Kennedy et al., 2003; Luke & Christianson, 2018; Smith & Levy, 2013 ) that generally show a negative scaling pattern(larger models perform produce less well-fitting surprisals), the Brothers & Kuperberg (2021) shows the positive scaling effect seen with the N400 data. The Smith & Levy (2013) results are less clear, both in terms of differences between models and in terms of overall scaling patterns.An ordinary least-squares linear model predicting AIC based on number of parameters and architecture also shows this difference. Even after correction for multiple comparisons, model size has a significant effect on 4 of the 6 datasets (Boyce & Levy, 2023; Futrell et al., 2021; Kennedy et al., 2003; Luke & Christianson, 2018; Smith & Levy, 2013 ; in addition to the Smith & Levy (2013) dataset before correction for multiple comparisons), with the surprisal calculated from larger models showing a worse fit to the data. Intriguingly, in line with the aforementioned observations based on Figure 2A , before correction, the Brothers & Kuperberg (2021) dataset shows the opposite effect-the same positive scaling we see on some of the N400 datasets. Returning to the differences between architectures, after correction, surprisals calculated using Mamba fit the Luke & Christianson (2018) and Kennedy et al. (2003) data significantly worse than Pythia, with this also being true of RWKV on both datasets before correction. Further details are provided in Table 4 .For the perplexity-ordered data (Figure 2B ), the same pattern emerges as for the N400 data-for the dataset where positive scaling is found (Brothers & Kuperberg, 2021) , Mamba models appear to perform relatively worse relative to Pythia than they do when the models are ordered by size and RWKV models perform relatively better, and for datasets where negative scaling is found, Mamba models appear to perform relatively better and RWKV models relatively worse. Again, while we see N400-like positive scaling on the Brothers & Kuperberg (2021) dataset-with lower-perplexity models showing a better fit to the data-we see the opposite pattern with the remaining 5.Further confirmation of the different scaling patterns comes from ordinary least-squares linear models predicting AIC based on perplexity and architecture. After correction for multiple comparisons, models with a lower perplexity produce surprisals that are significantly better at predicting the Brothers & Kuperberg (2021) data, but significantly worse at predicting reading time in 4 datasets (Boyce & Levy, 2023; Futrell et al., 2021; Kennedy et al., 2003; Luke & Christianson, 2018 ; and again, the Smith & Levy, 2013 dataset before correction). In this analysis, surprisal values calculated from the Mamba and RWKV models also show a significantly worse fit to the Kennedy et al. (2003) data, with those from the RWKV models also showing this on the Luke & Christianson (2018) dataset before correction for multiple comparisons. The details of all statistical analyses are provided in Table 6 . 
discussion: 
 To the best of our knowledge, the present study is the first to compare the extent to which transformers and contemporary recurrent language models can model online human language comprehension. Previous work has overwhelmingly found that transformers better predict the N400 than recurrent neural networks (Merkx & Frank, 2021; Michaelov et al., 2021; 2022) . We show, by contrast, that on 6 datasets, when comparing models of the same size and trained on the same data, contemporary recurrent language model architectures generally out-perform transformers, with surprisal values calculated using Mamba models tending to provide the best fit to the N400 data. When accounting for model perplexity, the comparison across architectures is less clear-cut; however, the contemporary recurrent architectures at least match transformer performance.The results are more mixed for reading time metrics. On the Kennedy et al. (2003) dataset, for example, the Pythia models predict go-past duration best at any scale or perplexity; while on the other hand, the recurrent models predict self-paced reading time on the Brothers & Kuperberg (2021) dataset best except at the 1.4-1.5B scale. Such mixed results for behavioral data is perhaps unsurprising given the conflicting results in previous work (Goodkind & Bicknell, 2018; Merkx & Frank, 2021; Hao et al., 2020; Wilcox et al., 2020; 2023a; Kuribayashi et al., 2021; Oh et al., 2022; 2024; Oh & Schuler, 2023a; b; Shain et al., 2024) .We also report several interesting scaling results. First, on the whole, scaling patterns are consistent across architectures. For datasets where larger, lower perplexity models tend to predict the human metric better (Federmeier et al., 2007; Hubbard et al., 2019; Szewczyk & Federmeier, 2022; Wlotko & Federmeier, 2012; Brothers & Kuperberg, 2021) , this tends to be true for all model architectures. The same is true for datasets where smaller models and those with a higher perplexity tend to predict the human metric better (Boyce & Levy, 2023; Futrell et al., 2021; Kennedy et al., 2003; Luke & Christianson, 2018) . The Szewczyk et al. ( 2022) and Smith & Levy (2013) datasets offer possible exceptions, where different models appear to show different scaling effects. However, without more models of each architecture, it is impossible to be certain.Another surprising result is that contrary to the recent work that finds the same negative scaling pattern across all reading time datasets (including both self-paced reading and eye-tracking metrics; Oh & Schuler, 2023b; Oh et al., 2024) , here one dataset (Brothers & Kuperberg, 2021) actually showed positive scaling. One possible explanation for this is that it is not log-transformed like the other reading time metrics. Another is that it includes the reading times of the following two words. However, the fact that it is slightly better predicted by taking the surprisal of the first word rather than that of the combined three words (see Shain et al., 2024, SI 1) suggests that the metric itself may not differ in this way as much as may be expected. A more likely explanation for the difference is that unlike the other behavioral studies which involved the reading of naturalistic stimuli, the stimuli in the Brothers & Kuperberg (2021) were carefully constructed to have different degrees of predictability. All the N400 studies use such stimuli, and this may therefore explain why the Brothers & Kuperberg (2021) results more closely resemble the positively-scaling N400 results. In any case, the finding highlights the point made by Brothers & Kuperberg (2021) that the task and stimuli used in such studies should not be overlooked when making wider claims about the relationship between probability and processing difficulty. It further suggests that the recent and ostensibly robust findings of negative scaling with behavioral data (Oh et al., 2022; 2024; Oh & Schuler, 2023a; b) may be limited to a specific type of reading time study, and that further analyses should be carried out.Finally, we note the finding that when comparing architectures by model perplexity rather than model size, there was a consistent pattern in terms of which model best predicted the data. Specifically, compared to when ordered by model size, when the dataset showed positive scaling, the performance of Mamba appeared worse relative to other architectures, and the performance of RWKV appeared better; and when the dataset showed negative scaling, the reverse was true. Given that at each size, Mamba has a lower perplexity than Pythia and RWKV has a higher perplexity (Gu & Dao, 2023; Appendix B) , this suggests that a language model's ability to predict the next word in a sequence does impact the extent to which it can model online human language comprehension above and beyond model size and architecture. Specifically, this result suggests that there are scaling effects across model architectures related to model quality (i.e., performance at next-word prediction). Even when controlling for number of parameters and training data, on a dataset that exhibits positive scaling, models that are better at next-word prediction are better at the human metric; and the converse is true for datasets that exhibit negative scaling.Ultimately, the results highlight a number of complicating facts. First, there is no single universal pattern accounting for the relationship between language model probability and all metrics of online human language comprehension. Second, general language modeling performance has an effect on the extent to which language models can predict such metrics. And third, there are idiosyncratic differences between datasets, metrics, and model architectures.Nonetheless, the present study opens up new lines of research. Crucially, in contrast to previous work, the results show that transformers are not uniquely well-suited to modeling the N400. They also align with previous research showing the same for some measures of reading time (Eisape et al., 2020; Kuribayashi et al., 2021; Merkx & Frank, 2021; Oh et al., 2022) . Indeed, in our results, the differences in modeling performance between models of different architectures at a given scale or perplexity tend to be dwarfed by the differences within architectures across these dimensions.In the present study, the performance of transformers and recurrent models is comparable, and thus our results are not able to evaluate whether there are specific architectural features of transformers or the recurrent models that make them better able to model human language comprehension. As discussed in §2,recurrent models provide a better model of the role of the working memory bottleneck (Merkx & Frank, 2021; Michaelov et al., 2021) , while transformers better simulate cue-based retrieval models of comprehension (Ryu & Lewis, 2021; Merkx & Frank, 2021) . It is not necessarily straightforward to disentangle the two. For example, Ryu & Lewis's argument that transformers are good models of cue-based retrieval is based on the finding that they display interference effects on agreement (also known as agreement attraction effects) and show patterns in attention that align with the theory. But recurrent neural networks have been observed to display such effects (Arehalli & Linzen, 2020) , which could be plausibly explained by lossy compression of the context. Nonetheless, identifying cases where the behavior of recurrent models differs from that of transformers qualitatively-rather than quantitatively, as in the present study (with the possible exception of the Szewczyk et al., 2022 data)-and comparing these to the human data is likely to be valuable across the computational-to-cognitive model continuum. At the more purely computational end, it is important to know which type of model to use to best capture the possible effects of statistics on language comprehension. Further towards the direction of cognitive modeling, such experiments can help to evaluate which theories provide more viable explanations of human language comprehension, for example, by comparing the predictions of the now-or-never bottleneck and cue-based retrieval accounts.The new generation of recurrent models is in its infancy. As these models continue to be developed, optimized, and scaled up, the question of whether they or transformers provide better models of human language comprehension (or at least, show a stronger degree of correlation to specific metrics of online human language comprehension) is likely to become clearer. In the meantime, the results presented here suggest that recurrent models not only match, but in some cases exceed, the performance of contemporary transformers at modeling human language comprehension, and may provide a valuable way to test hypotheses about the neurocognitive mechanisms underlying it.The N400 is a negative-going component of the event-related brain potential that occurs roughly 300-500ms after the presentation of a stimulus, peaking at around 400ms (Kutas & Hillyard, 1980) . A well-replicated finding is that the amplitude of the N400 response to a word is sensitive to the contextual probability of a word, either operationalized as cloze probability (Kutas & Hillyard, 1984) -the proportion of people to fill in a gap in a sentence with a given word (Taylor, 1953; 1957) -or when using the predictions of language models (Frank et al., 2015) . Specifically, the amplitude of the N400 response elicited by a word is large by default, and decreases by the extent to which it is predictable based on the preceding context.In this study, we compare how well the Pythia, RWKV, and Mamba models predict N400 amplitude based on the results of 6 experiments (Federmeier et al., 2007; Hubbard et al., 2019; Michaelov et al., 2024; Szewczyk & Federmeier, 2022; Szewczyk et al., 2022; Wlotko & Federmeier, 2012) . The details of these datasets and how they were analyzed are outlined below. 2007) measured N400s to low-and high-cloze words in low-and high-constraint contexts. We use the data from this study as preprocessed by Szewczyk & Federmeier (2022) . In this dataset, N400 amplitude is operationalized as the mean voltage at four centro-parietal electrodes (MiCe, MiPa, LMCe, RMCe) over the 300-500ms time window. N400 amplitudes are also not baseline-corrected; instead, the mean amplitude in the -100-0ms time window is intended to be included as a covariate in analysis. This dataset contains 7856 trials from 32 participants reading 564 stimuli.To calculate model fit to the N400 data, we followed as closely as possible the approach used by Szewczyk & Federmeier (2022) , which involved predicting N400 amplitude using a linear mixed-effects regression with surprisal, baseline amplitude, log-transformed frequency, the position of the word in the sentence, orthographic neighborhood distance, and concreteness as fixed effects. We also used the same random effects structure, removing variables until a structure that would not lead to singular fits for any regression was reached, which included random slopes of baseline for each subject and experimental item, random slopes of word position for each subject, and random intercepts of subject and item. We then compared the fit of regressions using surprisal calculated from each language model. With the exception of the Michaelov et al. (2024) dataset, our remaining N400 datasets are the others provided online by Szewczyk & Federmeier (2022) , and thus are preprocessed and analyzed in the same way.Wlotko & Federmeier (2012) used stimuli from Federmeier et al. (2007) as well as (Wlotko & Federmeier, 2007) , which were selected to cover a wide range of probabilities. This dataset was made up of 4,440 trials (300 stimuli; 16 experimental participants). Federmeier et al. (2007) . The dataset comprises of 5,705 trials (32 participants). 2022) also based their stimuli on those in Federmeier et al. (2007) , with adjectives added before critical words, making them more or less predictable. The dataset is comprised of 4,939 trials (672 stimuli; 32 participants).The stimuli of the Michaelov et al. (2024) dataset differ from the other datasets in their design. Rather than having two versions of each sentence-the most likely continuation and an unlikely one-each sentence has four possible endings: the highest-cloze continuation, a low-cloze but plausible continuation that is semantically related to this highest-cloze continuation, an equally low-cloze but unrelated continuation, and an implausible continuation. The two low-cloze completions were matched for cloze probability and plausibility. There were 125 sentence frames, for a total of 500 sentences. There were fifty participants, and data from a total of 5,526 trials after cleaning.The N400 was operationalized as the mean voltage in the 300-500ms time-window at each of the C3, Cz, C4, CP3, CPz, CP4, P3, Pz, and P4 centro-parietal electrodes. Unlike the data released by Szewczyk & Federmeier (2022) , the voltage at each electrode was treated as a separate data point and N400 amplitudes were baselined using the mean amplitude in the 100ms period before stimulus presentation. Thus this dataset comprises of 49,734 data points. We analyzed these data in the same way as in Michaelov et al. (2024) , fitting a regression that predicted N400 amplitude using Surprisal, log-transformed word frequency, orthographic neighborhood distance as main effects, and included random intercepts of experimental subject, sentence context, critical word, and electrode.Self-Paced Reading is an experimental paradigm in which participants read a text one word at a time, pressing a button or key to proceed to the next word. The reading time of a word is the time taken between button presses (i.e., between pressing the button to proceed to that word and pressing the button to proceed to the next word). Self-Paced Reading Response Time is generally considered to reflect processing difficulty, with longer reading times indexing a more difficult word.The Natural Stories Corpus (Futrell et al., 2021) is made up of selfpaced reading times from 181 experimental participants reading 10 texts, each comprising roughly 1000 words. These texts were constructed by taking publicly available texts and editing them to contain rare and hard-to-process syntactic constructions. Following Oh et al. (2024) , we excluded reading times for all words that appeared at the beginning or end of a sentence, all reading times shorter than 100ms or longer than 3000ms, and all data from participants who answered three or fewer comprehension questions correctly. The regression used to predict log-transformed reading time also followed that described by Oh et al. (2024) . In addition to language model surprisal surprisal, we included word length, log-transformed word frequency, and the word's position in the sentence as predictors, as well as random slopes of each of these predictors for each subject, and random intercepts of subject and sentence.This dataset is comprised of self-paced reading times from 35 experimental participants reading 292-902 word passages from the Brown Corpus of American English (Francis & Kučera, 1964) , for a total of 2860-4999 words per subject. We used the same exclusion criteria as with the Natural Stories data (with the exception of the comprehension score, which was not available), and the linear mixed-effects regressions each had the same structure as those used in analyzing the Natural Stories data.A well-known phenomenon in self-paced reading is that of spillover effects, where the extent to which a word is difficult to process also impacts the reading time of one or more following words. One way to account for this is to use the reading time of a given word and the following two words as a measure of the processing difficulty associated with the word.In this self-paced reading study, there were 216 sentence sets, each in a low-, medium-, or high-cloze condition, for a total of 648 stimulus sentences. Participants were excluded by Brothers & Kuperberg (2021) if they had an average comprehension check score of less than 75%. After exclusions, data from 216 of the total 240 participants were included in the analysis with a total of 46,092 data points. Data were cleaned and preprocessed by Brothers & Kuperberg (2021) . We fit regressions following the method in the original study, predicting reading time (un-transformed) using a linear mixed-effects model with a main effect of language model surprisal, random intercepts for each subject and item, and random slopes of surprisal for each subject and item.Like self-paced reading, in the Maze task, participants read a text one word at a time. However, in the Maze task, participants see pairs of words and can only proceed to the next word in the text by choosing the correct next word on the screen. If the participant chooses the incorrect word, they receive feedback and are prompted to choose again. The time it takes for participants to choose a word is recorded as the reaction time. We look at the reaction times from a previous study by Boyce & Levy (2023) . Dataset and analysis details are provided below.Boyce & Levy (2023) In this study, participants completed a Maze task using the stimuli from the Natural Stories corpus (Futrell et al., 2021) , which comprises 10 texts based on publicly available texts, each approximately 1000 words long. In total, Natural Stories contains 10,245 words. Boyce & Levy (2023) recruited 100 participants, but participants were excluded if they did not self-report as native speakers of English. Following Boyce & Levy (2023) and Shain et al. (2024) , we exclude data for all words with a reading time of less than 100ms or greater than 5000ms, incorrect words, words that were at the start or end of a sentence, and all data from participants that correctly answered fewer than 80% of comprehension questions correctly. We construct linear mixed effects regressions predicting log-transformed reaction time with surprisal, word length, log-transformed word frequency, and the word's position in the sentence. We also included random slopes of surprisal, word length, and word position for each subject, as well as a random intercept of sentence. Our analysis is based on the preprocessed version of this dataset provided by Shain et al. (2024) .Go-past duration is an eye-tracking-based metric of reading time. In eye-tracking studies, participants generally read a text naturalistically. Unlike in the other experimental paradigms, participants can see the whole text at one time and are able to look at previously read words. The location of each participant's gaze is recorded using an eye tracker, which also records how long participants' gaze is fixated on a given location. There are many different possible eye-tracking metrics for a given word that can be calculated (see, e.g., Shain et al., 2024) , but following recent work analyzing how well different language models predict eye-tracking data (Oh & Schuler, 2023b; a; Oh et al., 2024) , we look at log-transformed go-past duration, which is defined as the amount of time from when the word was first fixated to when the participant first looked to the right of that word (in left-to-right languages like English; see Luke & Christianson, 2018; Shain et al., 2024) . We use data from the Provo corpus (Luke & Christianson, 2018) . The details of this dataset and how it was analyzed are provided below.The Provo Corpus (Luke & Christianson, 2018 ) is a dataset consisting of eye-tracking data for 84 participants reading 55 passages (news articles, popular science magazines, and fiction). Passages averaged 50 words long. In total, the texts comprised 2,689 words. Participants' go-past durations were recorded while they read each text. As with the N400, we use linear mixed-effects regressions to calculate the fit of the surprisals calculated by each model to the data. Following recent work (Oh et al., 2024) , we exclude from our analysis all words that were not fixated, that followed saccades of longer than 4 words, and that were at the start or end of sentences or files. Also following Oh et al. (2024) , we constructed a regression to predict log-transformed go-past duration based on surprisal as well as the following covariates: saccade length (in words), word length (in characters), word position in the sentence, log-transformed word frequency, and whether the previous word was fixated. We also included random slopes of all predictors for eachIn the original Mamba paper, Gu & Dao (2023) report the performance of Mamba models on a number of benchmarks against comparable language models with different architectures. In the 1.4-1.5B and 2.8-3B parameter range, Gu & Dao (2023) find that Mamba has a lower perplexity than Pythia on the Pile (Gao et al., 2020) validation set, and that RWKV has a higher perplexity. This result suggests that at these scales, the Mamba models are best able to learn the statistics of language, followed by the Pythia transformers, which are in turn followed by the RWKV models.We further replicate this finding for the other model sizes in Figure 3 , finding that with the exception of the smallest models (130-170M parameters) where Pythia and RWKV have the same perplexity, at every model size, Mamba has the lowest perplexity, followed by Pythia, followed by RWKV. (Merity et al., 2017) was calculated using the Language Model Evaluation Harness (Gao et al., 2021) .ConclusionsWe compare how well transformers and two contemporary recurrent language model architectures-RWKV and Mamba-can predict 5 different metrics of online human language comprehension. We find that overall, the recurrent models tend to match the performance of transformers at predicting both neural and behavioral human metrics, and that when specifically comparing across architectures by number of model paramaters, recurrent models in fact appear to be best at predicting N400 amplitude.Szewczyk & Federmeier (2022) also release an additional dataset with data from a previously-unpublished study using stimuli based onFedermeier et al. (2007) and including data from 4,822 trials (600 stimuli; 26 experimental participants). We refer to this as the Szewczyk & Federmeier (2022) dataset. 
 
 Appendix:  


Decision: This paper evaluates two recent recurrent models, in comparison with a transformer, on predicting measures of human language comprehension. Results suggest that -- for the specific models (with the same training data domain and comparable parameter counts), recurrent architectures show similar performance to transformers, sometimes exceeding them.

Despite discussion, this remained a controversial paper, with ratings ranging from 4 to 8. Having a strong background in this field, I believe that the weaknesses raised by sgiH (who provided the only rejection score) are satisfactorily addressed by the authors, even though sgiH did not raise their score. 

Taken all reviews and my reading of the paper into consideration, I believe this to be a timely and worthwhile contribution to the sub-field evaluating language models as predictors of human language comprehension measurement.

[At least one review was discounted during the decision process due to quality]

Reviews: review: paper_summary: paper_summary: This is a well written paper with a clear structure. The paper is well motivated with clear arguments. The paper also presents a good amount of empirical evidence to back the conclusions.

The paper is quite original as it shows evidence that RNNs can outperform transformers in human language compression. However, in terms of technicality, the paper doesn't proposal noticeable novelty.
strength_weakness: strength_weakness: I'm skeptical about some points in the paper. Firstly, the authors argue that one reason why transformers are less cognitively plausible is their finite context window. I believe that the finite context window is only a limitation in practice because in theory, one can choose the window size as large as possible. Besides, isn't that human memory has some certain limitation? 

My second concern is the models chosen in the experiments. The authors specifically chose Mamba and RWKV because they have been shown to be very fast and perform on par with transformers. However, there are several variations of transformers yet the authors didn't give a good argument why Pythia was picked. 

Another related concern is about how the chosen models were trained. The paper seems lack that details. Besides, the authors draw conclusions without disentangling model architecture and model optimization. In fact, with reasonable optimization (e.g. some simple tricks), one can improve a model performance.
paper_summary: paper_summary: The paper evaluates a recent crop of recurrent language models on predicting the N400 response and reading times, finding that the modern recurrent models perform on par with Transformers on this task, updating previous old findings that Transformers outperform LSTMs.
strength_weakness: strength_weakness: I ultimately felt dissatisfied by this paper because I wanted more, both in terms of results/experiments and in terms of cognitive upshots. The current paper feels like an incremental progress update, although the beginning of something better. I think that is probably enough for publication in CoLM, but I have concerns below:

When comparing neural architectures on their ability to predict N400/RT, the question is always what we are learning about human language processing from the comparison. I don't feel like the current paper sheds much light on this yet. What's on my mind is: What is it exactly about Transformers that made them do better than LSTMs, and what has changed about the new recurrent models that make them match Transformers? Is it the better gating mechanism in RWKV? Or is it the restriction to linear dynamics as in Mamba? I think to get real cognitive insights, more models would have to be compared, including some custom models (like, various ablated versions of Mamba). In particular, Mamba could be compared to its predecessors like S4. It would also be necessary to give LSTMs a fair shot. The LLM field basically stopped training LSTMs: how would they perform if trained on the same data and with the same large number of parameters as the modern Transformers? A comparison with n-gram models trained on large datasets would also be informative. Basically, I feel like deriving real cognitive insights would require a much more thorough comparison than what is here.

Even given the current results, the discussion about cognitive upshots is sparse. What are we to make of the relationship between scaling direction and RWKV vs. Mamba performance? If recurrent models really are just as good as Transformers, what are we to make of the arguments that Transformers act like cue-based retrieval models---was it just a spurious connection? There are rich cognitive implications that aren't explored.

Other issues

- Measuring performance by AIC is not as good as measuring performance by delta-log-likelihood on heldout data. AIC is just log likelihood with some complexity penalties. These complexity penalties would ideally be the same across different datasets, reflecting use of the same model; and if different models were used, the complexity penalties are only crude measures of the actual complexities of these models. I think pure delta-log-likelihood based on maximally similar regression models and heldout data would be the best evaluation.

- Coverage of the literature is incomplete. The most important omission in the literature review is is Ryu & Lewis (2021) who argue that Transformers directly parallel the cue-based retrieval models that are common in psycholinguistics, in contravention of the general wisdom that recurrent models are more cognitively realistic. Two other omissions are Wilcox et al. (2023, TACL) and Xu et al. (2023, EMNLP) which look at the surprisal-reading time relationship cross linguistically.
questions: questions: - What is the motivation for the datasets used? Why not use Natural Stories SPR in addition to Natural Stories Maze, or any other eyetracking corpus like Dundee?
paper_summary: paper_summary: An empirical comparison of how different LM architectures (Transformers vs. two latest-generation recurrent models, RWKV and Mamba) predict various metrics of language processing difficulty by human subjects.

The paper starts from the somewhat suprising observation that traditional RNNs have been superseded by a less cognitively plausible architecture (Transformers) not only in a range for NLP tasks but also as predictors of behavioral metrics of language comprehension by humans.
It asks whether latest-generation RNNs would, in turn, fair better than Transformers in this context.
The comparison is performed on a suite of 9 language processing datasets using models of different architectures and sizes, all trained on the same corpus.

Results show a complex picture whereby different architectures are best across different datasets and model sizes. 
Also, an earlier finding is confirmed by which processing difficulty tends to be better predicted by models that have better language modeling performance (i.e. perplexity) in general.
strength_weakness: strength_weakness: - The results themselves are rather inconsistent and the overall conclusions a bit vague
- No attempt is made at explaining why some architectures may be outperforming others in different conditions
questions: questions: Clarity improvements:
- AIC -> spell out acronym
- For clarity, I'd give an acronym/ID to each dataset in Table 2 and use those acronyms/IDs for the discussion in Section 4. All those citations make it hard to follow the text
- Sect. 4.1 could end with some cleared take-aways. Overall, what have you learned from this experiment?

Typos:
- identifies filters the input
- initialized in based
- In thus study
- again see -> we again see (sect 4.2)
paper_summary: paper_summary: This paper compares performances of the state of the art recurrent
neural networks RWKX and Mamba, with that of a transformer Pythia on 9
from a human oriented perspective, through 9 datasets by use of 4
metrics of human language processing.

The method is by computing the surprisal.  The authors empirically
show that Pythia does not necessarily outperform RWKX and Mamba,
rather the contrary. The authors argue that RNNs perform better than
Transformer, from a human-oriented perspective.
strength_weakness: strength_weakness: There are two large concerns of this paper.

1. 

According to Table 1, the authors compare three models under
comparable settings. The number of parameters for every experiment is
comparable by scale.  Although I understand this is one "fair" way of
setting, I wonder whether Transformer's capability is fully exploited
this way.

Transformer performs better because it is able to accomodate larger
number of parameters than RNN, that is, O(n^2) vs. O(n), respectively,
for n being the length of the sequence.  Transformer architecture
includes that of RNN.

The present result shows that if Tranformer is trained on a system
with equivalent number of parameters of RNN, then Transformer (could)
perform worse than RNN. This is trivial to me, because Transformer
uses parameters to extract relation among tokens that cannot be
captured by RNN. Therefore for Transformer to perform fully, it must
use much more parameters than RNN.

2. 

The authors focus only on human cognition.

If comparing under the same setting of Table 1, the data does not have
to be be restricted to human cognition. I guess even for a general
evaluation dataset, the same comparison of Transformer vs. RNN can be
made and presumably the same conclusion could be obtained.

In other words, since the authors do not show the evaluation results
for other evaluation dataset, the value of this work is not well
highlighted.
questions: questions: -Did you compare the three systems on the same data when using the
maximum number of parameters available for each system?  If results do
you think you would get?

-Did you compare the results for other data that are not oriented to
human cognition?  If you do that, what do you think you would get?

Comments: title: Another reviewer jumps in
response
thanks for the clarifications
comment: We would like to thank Reviewer sgiH for their response.

On the first point (optimization/training details), because we used pre-trained models, we did not include the training details and hyperparameters described in the original papers associated with each model type (Pythia: [Biderman et al., 2023](https://proceedings.mlr.press/v202/biderman23a.html), p. 2 and p. 22; RWKV: [Peng et al., 2023](https://aclanthology.org/2023.findings-emnlp.936/), pp. 14065-14066; Mamba: [Gu and Dao, 2023](https://arxiv.org/abs/2312.00752), pp. 29-36). In addition, the Pythia GitHub repository (https://github.com/EleutherAI/pythia) and the RWKV repository (https://github.com/BlinkDL/RWKV-LM) each provide the training code, and the Mamba repository (https://github.com/state-spaces/mamba) also provides code that can be used to train the models.

On the second point (cognitive plausibility), we agree with the reviewer that there are certain features of the models’ implementations that may not necessarily be cognitively plausible. However, as Reviewer WYrb notes in their reply to the response, there is still value in (and an active research community engaged in) investigating the differences in how well language models of different architectures can model human language processing. In this case, as we note in the original submission and Reviewer WYrb mentions in their initial review, there are competing accounts of human language processing that can be considered analogous at a computational level to recurrent and transformer architectures—the now-or-never bottleneck in the case of recurrent models (see [Merkx and Frank, 2021](https://aclanthology.org/2021.cmcl-1.2/)), and cue-based retrieval in the case of transformers (see [Ryu and Lewis, 2021](https://aclanthology.org/2021.cmcl-1.6/);  [Merkx and Frank, 2021](https://aclanthology.org/2021.cmcl-1.2/)). We thank the reviewer for pointing out that this may not necessarily be clear in the submitted version; we will clarify and expand upon this point in the revised version.

On the third point (conclusions), one of the key questions for our study was how well models of different architectures perform, for the reasons noted in the previous paragraph. Work by [Merkx and Frank (2021)](https://aclanthology.org/2021.cmcl-1.2/) suggests that transformers are indeed empirically better at predicting the amplitude of the N400 at least, and [Ryu and Lewis (2021)](https://aclanthology.org/2021.cmcl-1.6/) provide empirical and theoretical support for the idea that transformers are good models of human language processing. However, our results show that if there is a difference, it is small (and in fact, in the case of the N400, recurrent models tend to be better), which suggests that the transformer architecture may not be as important as previously thought for modeling human language processing. Instead, our results suggest that the question of which type of architecture provides a better computational-level model of human language processing is far from resolved, and that, since the new generation of recurrent models is only in its infancy, this question may need to be revisited as new developments in the field arise.
Jumping into this discussion as another reviewer, responding to this point:

> But I really don't think that's relevant because the models have several technical details that aren't proved to be cognitively plausible. 

For what it's worth, as a cognitive scientist, I'd like to point out that there's already a large literature on using models to predict EEG data exactly like in this paper. It's true that some of the mechanisms in the models are cognitively implausible (like multi-head attention in Transformers), but this paper is responding to and joining existing work which has already dealt with a lot of the subtleties in linking LM surprisals to human data.
I would like to thank the authors for the response. My response is below:

First, re model selection & optimization, one specific point unclear to me is how the models were trained. I couldn't find training details in the paper, and thus I can't tell if there're tricks to optimize the models.

Second, the authors shape the contents under human cognition theme; and from cognitive plausibility point of view, the authors discuss transformers vs recurrent models. But I really don't think that's relevant because the models have several technical details that aren't proved to be cognitively plausible. How do we know that the used recurrent models outperform the transformer because their recurrent mechanism or because of their cognitively unplausible technical details?

Third, it's unclear what the authors want to conclude here. The authors show performance results, but then what could we learn from them to help us understand human cognition?
Thanks for the response, I've upped my rating.
Thank you for your rebuttal.  After reading through other reviewers' comments and all the rebuttal, I've changed my opinion and  increased my score.
Thanks for your answers. I stand by my overall positive judgement of this paper. Even in the lack of a clear winner, the paper has enough contributions with its analysis of novel models on this task.