Words, Subwords, and Morphemes: What Really Matters in the Surprisal-Reading Time Relationship?
Authors: Sathvik Nair
Philip Resnik
Venue: EMNLP 2023 Findings (2023)
Keywords: psycholinguistics
sentence processing
tokenization
TL;DR: We evaluate the surprisal of n-gram language models trained under orthographic, morphological, and BPE tokenizations against human reading time data.

Abstract: An important assumption that comes with using LLMs on psycholinguistic data has gone unverified. LLM-based predictions are based on subword tokenization, not decomposition of words into morphemes. Does that matter? We carefully test this by comparing surprisal estimates using orthographic, morphological, and BPE tokenization against reading time data. Our results replicate previous findings and provide evidence that *in the aggregate*, predictions using BPE tokenization do not suffer relative to morphological and orthographic segmentation. However, a finer-grained analysis points to potential issues with relying on BPE-based tokenization, as well as providing promising results involving morphologically-aware surprisal estimates and suggesting a new method for evaluating morphological prediction.

Words, Subwords, and Morphemes: What Really Matters in the Surprisal- Time Relationship?
Sathvik Nair and Philip Resnik
Linguistics & Institute for Advanced Computer Studies
University of Maryland (sathvik, (resnkkeeumn edu
Abstract
Michaelov al., 2023), and reading times (Demberg and Keller, 2008; Smith and Levy, 2013; Goodkind and Bicknell, 2018; Wilcox et al., 2020; Shain et al., 2022). This correlation paradigm has produced useful insights about the role of prediction in language comprehension. In addition, correlations between language model surprisal and human processing activity have been taken be an indication that language models are capturing relevant aspects of how human language works (Ryu and Lewis, 2021; Goldstein al.. 2022).
An important assumption that comes with us- ing LLMs on psycholinguistic data has gone unverified. LLLM-based predictions are based on subword tokenization, not decomposition of words into morphemes. Does that matter? We carefully test this by comparing surprisal estimates using orthographic, morphological, and BPE tokenization against reading time data. Our results replicate previous findings and pro- vide evidence that the aggregate, predictions using BPE tokenization do not suffer relative to morphological and orthographic segmenta- tion. However, a finer-graine analysis points potential issues with relying BPE-based tokenization, as well as providing promising results involving morphologically-wwerr prisal estimates and suggesting new method for evaluating morphological prediction.
Lost in the shuffle of this research progress, how- ever, is the question of what, exactly, "prediction at the word level" is supposed mean. Within cur- rent linguistic theory, the very construct of "word" receiving critical attention: albeit controversially, convincing arguments exist that the term word lacks a consistent, scientifically valid definition that holds consistently across the full range of human languages (Halle and Marantz, 1993; Haspelmath, 2017); see Krauska and Lau (2023) for an overview connecting these theoretical claims to psycholin- guistics and neurolinguistics. Even setting aside that theoretical debate, however, the move psy- cholinguistics and neurolinguistics using large lan- guage models has generated a mismatch between the units analyzed in human studies - typically or- thographic words - and the subword tokens over which LLMs operate via approaches like Word- Piece and Byte-Pair Encoding (Wu et al., 2016; Sennrich al., 2016).
1 Introduction
There is widespread consensus that human sentence processing includes word-level prediction (Ehrlich and Rayner, 1981); see Staub (2015) for a review. A growing body of research is making use of lan- guage models as computational proxies for human prediction at the word level, including traditional n-gram models (Smith and Levy, 2013), syntax- based models (Hale, 2001; Demberg and Keller, 2008), and more recent work that makes use of neural language models (Goodkind and Bicknell, 2018; Wilcox et al., 2020; Shain et al., 2022).
As an overall paradigm, research in this area generally correlates surprisal, computed using corpus-based probability estimates (-log Pr(word(context)). against measurable indices of human processing effort. These include measurement of processing activity using fMRI (Henderson al., 2016: Bhattasali and Resnik, 2021; Shain et al., 2020), MEG (Donhauser and Baillet, 2020; Brodbeck et al., 2022), EEG (Rabovsky et al., 2018; Li and Ettinger, 2023;
To take an example, a typical LLM tokenization of the word decomposition using a widely used BPE tokenization (Sennrich et al., 2016) yields subword units dec, om, posi tion. In typical
'These two kinds of results are important distinguish. the case, human beings and their cognitive systems are the object of study. The second an instance of what Simon (1969) referred to as "science of artificial", where in this case the constructed computational system itself the object study.
We use the GPT2 implementation of BPE throughout.

---

human subjects experiment, measurements such as first-pass reading time would typically involve a region from the beginning of the whole word to its end. More generally, the measurement of activity for a word ir in the human experimentation is related to language model predictions using w's subword units $182... ....... 8n. How is a correlation computed between measurements over different units?
LLMs. If statistically-driven subword units work just as well, then we have checked an important, previously unchecked box in terms of validating their use.
This work is very much in the spirit of Wilcox et al. (2020), who evaluated a "fleet" of language models across architectures, plus orthographic n- gram models, against eyetracking and self- self-paced reading data. However to our knowledge, this study is the first to consider the assumption that LLM- subwords can be used in lieu of morphological units.
The solution adopted by most researchers (Hao al., 2020; Wilcox et al., 2020; Stanojević al., 2022, and others) is to compute surprisal separately for the 8; and then approximate the model's sur- prisal for w as the sum of those individual subword surprisals. The logic behind this choice is that, if the linking hypothesis behind the work connects surprisal with cognitive effort, the effort for the entire word should be the sum of the effort on each of the parts (Smith and Levy, 2013).²
Methods
We trained three n-gram models under different word segmentation methods and evaluated them against reading time data from psycholinguistic experiments conducted in English. Our choice of evaluation corpora and metries are consistent with previous literature, such that were only examin- ing the effect of word segmentation without model architecture as a confound. Our implementation, along with instructions for accessing the associ- ated data, is available at https: h/gitthub. com/ wtwpii/n//d-pscchhohohhheeeeeeee
This, however, leads to another question: that a reasonable approximation, given that the sub- words produced by LLM tokenization bear no clear correspondence to the subword decompositions in human processing? Consider again the word de- composition. A large body of theoretical and empir- ical work would suggest that the extent subword effort takes place, would involve morphological units, in this case de, ce), and (i) tion (Gwilliams, 2020).*
2.1 Language Models
Each n-gram model was 5-gram model trained the publicly available section of the Cor- pus of Contemporary American English (COCA, Davies (2010)). The models were trained under KenLM (Heafield, 2011) using modified Kneser- Ney smoothing (Ney et al., 1994). As a control, we used a model trained on COCA which was trained to predict the next orthographic word without any subword tokenization. To test BPE-based tokeniza- tion, we used the Huggingface implementation of the GPT2 tokenizer (Wolf et al., 2020; Radford et al., 2018) for each sentence in the corpus, and trained the n-gram model on individual tokens rather than words. Most current psycholinguis- tic work uses off-the-shelf GPT2 implementations and GPT2 (and variants) have been shown to be better fits to reading time data than larger models (Shain et al., 2022; Oh and Schuler, 2023). Fi- nally, for more linguistically informed approach to word segmentation, we trained an n-gram model based on the output of morphological transducer (Wehrli al., 2022) that is far more accurate than WordPiece tokenization at word and sentence-level morpheme segmentation tasks in a variety of lan-
The question we set out to answer in this paper, then, is whether the divergence between LLM sub- word tokenization and human morphological de- composition is something to worry about in compu- tational psycholinguistics and computational neu- rolinguisties research. Operationalizing the ques- tion, would the surrosssus approach with morphologically valid units yield a better corre- spondence with human measurements than the stan- dard approach using LLM subword tokens? We would argue that the result is important regard- less of which way the experimentation goes. If morphological units turn out to be significantly better fit for human measurements, then cogni- tive researchers using LLMs should be using them which could potentially raise real challenges given the widespread use of off-the-shelf pretrained
"In principle one could be more sophisticated by calculat- ing the uniqueness point within the word a given lexicon (Luce, 1986) and only summing up assigned complexity val- ues for wordpieces that precede that point.
"We discuss another illustrative example in Appendix A and Section 2.1 provides details on the morphological seg- menter we use in our experimentation.

---

guages, including English (Batsuren et al., 2022).² The morphological transducer was based on an encoder-decoder architecture, which used a two- layer stacked LSTM as the encoder and performed greedy decoding. We judged the corpus too small for retraining a GPT-style model, and we did not train an LSTM because Wilcox et al. (2020) conclu- sively showed 5-gram models are stronger predic- tors of results in broad-coverage psycholinguistic experiments."
texts from the behavioral experiments and com- puted each token's surprisal. If a word was split into multiple tokens, its surprisal under the other two models was the sum of the tokens' individual surprisals. 7 We then fit regression models predict- ing reading time based on surprisal.
For each word segmentation method, com- pared the per-token log likelihood (ALggLik un- der two multiple linear regression models, follow- ing previous literature to quantify how much sur- prisal contributes to reading time prediction, inde- pendent of other predictors. One model used the control features of word length and log unigram fre- quency to predict reading times as a baseline model, and the other used these factors in conjunction with surprisal. If the regression model with surprisal- based features generated more accurate predictions, the difference between the log likelihoods would be above zero.
2.2 Psycholinguistic Evaluation
Once we trained the models, we computed their surprisal estimates for words in eyetracking and reading corpora and fit regression mod- els evaluating surprisal as predictor of the reading times from these corpora.
2.2.1 Corpora
We used averaged eye movement data from the Dundee corpus (Kemnedy et al., 2003) and self- paced reading times from the Natural Stories cor- pus (Futrell et al., 2018) made available by Wilcox et al. (2020). Both corpora representative of the material in COCA. In the Dundee corpus, each word's fixation time in milliseconds was averaged across 10 English-speaking participants reading newspaper articles. The Natural Stories corpus consists of sentences from narrative texts that were edited to include syntactic constructions that are rare in spoken and written English for psycholin- guistic analysis. The self-paced reading times were recorded from 181 English speakers who read the texts as they were presented word-by-word. We used the wor-word presentation times that were av- eraged across participants.
Although Lik as the measure of predictive power is standard for this literature, we highlight two specific methodological details from (Wilcox et al., 2020). First, the predictive power normalizes the regression models' aggregate log likelihood since we are comparing this metric on corpora with different sizes. Second, we used 10-fold cross val- idation to report Log on a held-out test set. The training and testing data were consistent across all models for each fold. Reporting the value of cross-validated regression model is important to ensure that the predictive power measures com- puted on the complete dataset are not the result of overfitting (Yarkoni and Westfall, 2017). Due spillover effects from previous words (Smith and Levy, 2013), we also included the surprisal, length, and log frequency of the previous word as predic- tors in the the regression models for the Dundee corpus, and similarly for the previous three words for the Natural Stories corpus.
2.2.2 Measuring Predictive Power of Surprisal
To compare the reaction times to the models, we computed the surprisal for each word under each n-gram model. The model trained on orthographic words generated surprisal for each word, but since the BPE tokenizer and the morphological trans- ducer used subword information, we tokenized the
3 Results
All our results show surprisal improves predictions of reading time relative to the control features, com-
"We excluded data for certain words from our analysis following Goodkind and Bicknell (2018). These were words preceding and following punctuation, words that contained alhaabphaaliaa characters, and words that were out vocab- ulary for the language models. If any token under a language model was not vocabulary, the entire word was excluded,
'This analyzer came second the SIGMORPHON competition. We chose because the first place system's implementation was not publicly available.
'In this study we used a publicly available subset of the COCA corpus for replicability, and it is less than 2/3 the size of the training dataset for the smallest orthographic GPT2 model in Wilcox et al. (2020). A near-term aim for future work is to replicate this experimentation with the full COCA corpus, which requires a license.
"This difference arises because the corpora were used with different psycholinguistic tasks (Wilcox et al., 2020); we re- port results for each corpus separately for this reason, and hecause the corpora have different sizes and material.

---

Tokenization Method
ALog for Dundee
Lig for Natural Stories
Dundee
Natural Stories
0.015
Orthographic
0.01
0.009
BPE
0.01
0.01
Morphological
0.01
0.008
ALogLix
Table 1: Per-token Lik showing surprisal-buce -based
regression models improve over controls.
0.005
parably to the 5-gram models' results in Good- kind and Bicknell (2018) and Wilcox al. (2020). Table I reports the difference per-token log like- lihood between the linear regression models pre- dicting words' reading times using surprisal as feature and the models which simply used length and log frequency features. We also report conventional measure of effect size using Co- hen's f in Table 5. The surprisals of the previous and current word were statistically significant pre- dictors of reading time for all models on all corpora (p < 0.001).²
0.000
Orthagrapiic
Morphaingcallrmaaraanic
Morphologica
Figure Distribution of predictive power surprisal under models trained under each tokenization scheme, evaluated using 10-fold cross-validation for each cor- pus. There is no major difference in predictive power associated with tokenization.
in Table 3). Moreover, the standard linking theory for surprisal suggests that effort for the entire word should be the sum of the subword efforts (Smith and Levy, 2013), and therefore that processing ef- fort should increase incrementally with the number of units word is segmented into. But this appears to be true only for the morphological tokenization: for BPE tokenization there a sharp jump from low surprisal with unsplit words to essentially equal sur- prisal for words split into 2, 3, and tokens (Figure 2). The data therefore suggest that surprisal based on BPE tokenization is less cognitively realistic than surprisal over morphological units. In addi- tion, replicating the entire analysis separately for non-segmented and segmented words, find that the predictive power of the BPE-based model is significantly worse for words that do get split by the tokenizer, and this is not true for the morpheme- based model (Figure 3). We conclude that, despite the aggregate results in Figure 1, LLM-style tok- enization should be viewed with caution in cogni- tive investigations.
We also report ALogLik on held-out test set using 10-fold cross validation (Figure 1). The train- ing and testing data were consistent across all mod- els for each fold. Reporting the value of cross- validated regression model is important to ensure that the predictive power measures computed on the complete dataset are not the result of overfitting. We compared the distribution of these values under a Wilcoxon rank-sum test (Table 6). The predictive power of surprisal under the models using BPE and the morphological transducer's output did not show a statistically significant difference from the model using orthographic words. For the Natural Stories corpus, the predictive power of surprisal was lower than the Dundee corpus, which expected since the Natural Stories corpus contained rare syntactic constructions.
On the face of it, these results seem to show that LLM-style tokenization may not an issue in psycholinguistic modeling. However, finer-graine gnarier analyses suggest otherwise. First, few words in the psycholinguistic corpora were split by the BPE tokenizer in the first place. As it turns out, the BPE tokenizer only split 5% of the tokens in the psycholinguistic corpora (11% when ignoring stop- words), compared 25% and 44% respectively for the morphological analyzer (complete results
Conclusions
This study was small, focused contribution that tackled essential question previously unad- dressed in psycholinguistics research that uses LLMs and their subword tokenizations. Previous work has demonstrated a linear relationship be- tween LLM surprisal and human processing as mea- sured by reading time, but there good evidence
"For Natural Stories, the tokens at u and uy-2 also had some predictive power (p < 0.01 and < 0.1, respectively).

---

BPE
Morphelog.cel call
ies. Perhaps our most important take-away is that cognitive investigations require a careful look the cognitive plausibility of the models' units analysis.
80
Surpricl
Durcee
8
8
Our results also suggest new directions for more cognitively realistic models of prediction in lan- guage comprehension. We view the results from our finer-grained analysis as step in this direction, and they also suggest going more deeply into the role of morphological segmentation on an item-by- item basis; for example, training an LLM on mor- phological units and evaluating it on diagnostics for morphosyntactic generalization. Our work here also introduces morphological surprisal computed automatically using morphological segmenter, and validates its predictive power. This would be natural fit for further work morphological prediction the neural level (Gwilliams, 2020), including looking at the role of morphemes in phoneme prediction (Ettinger et al., 2014), and as a new representational level within integrative pro- cessing models that take phonological, word-level, and sentence-level contexts into account (Brodbeck et al., 2022). This implementation could be refined through inferring hierarchical structure over mor- phological units in the style of Oseki and Marantz (2020) conduct larger-scale analyses.
8
Naturall Stovies
Surpiisa
88 8
Nor ber of Tokern
Number of Tokerss
Figure 2: Distributions of surprisal of words with dif- ferent numbers of subword tokens, split by corpus and segmentation method.
Dundee
Natural Stories
Morphologiaal
BPE
Morphoiogical
0.02
0.01
ALogLi
Mloglk
0.00
-0.01
0.00
-3002
Finally, regarding broader theoretical discussion, wc note that surprisal (as operationalized using an LLM or any other probability estimates) gen- erally contributes to explanations at Marr's "com- putational" level (Marr, 1982). Moreover, L.LM predictions represent black-box combination of categories of information that both theoretical and experimental considerations suggest are processed in distinct ways (Oscki and Marantz, 2020; Oh et al., 2021; : Krauska and Lau, 2023). We would therefore argue that, despite their undeniable conve- nience and power, the widespread use of LLMs as probabilistic predictors deserves drastically more careful consideration than it has received if the field is to move in the direction of deeper insights into human representations and mechanisms in lan- guage processing.
Split
Split
Whole Split
Split
Figure Replicating the cross-validated analysis of predictive power separately for whole and split words under the different word segmentation methods.
that aspects of human processing are mediated by morphological units. For cognitive research, mightn't 't be important to model surprisal using morphological units, rather than distributionally de- rived subword tokens that often deviate drastically from word's morphological decomposition?
On the one hand, our work offers the first com- parison demonstrating that using distributionally derived rather than morphological subwords does not affect aggregate correlations between surprisal and reading times, widely used behavioral mea- surement of human processing. On the other hand, given that psycholinguistics work is increasingly using proprietary pre-trained models with non- morphological subword tokenization, far beyond the scale available for academic model training, our finer-graie grainer analysis indicates that a degree caution is warranted for more grained stud-
Limitations
The study was limited to English, and it is pos- sible different results might obtain in languages
IAAs an interesting and promising exception, Futrell et al. (2020) take a step in the direction of the algorith- mic/cppesennationall level hy bringing memory considcrations into the model.

---

with other morphological structures. However, the morphological transducer can be trained on any language for which a suitable morphologically seg- mented corpus is available, and it has already been evaluated on a multilingual test suite (Batsuren et al., 2022), so this is promising topic for future work. This is an active area of research, especially because not all languages have the same notion of what counts "word." Existing work (de Varda and Marelli, 2022; Wilcox et al., 2023) has evalu- ated predictions on a multilingual eyetracking cor- pus with some typological diversity, but still trains transformer language models on subword tokeniza- tion. More work is also needed to see if results vary across mono- and multi-morphemic words; see Appendix B for an indication that LLM sub- word tokenizations can still be problematic at the level of individual predictions, even for words that do not include much morphological complexity. We also note that the publicly available version of COCA we used was preprocessed by Yang et al. (2022), and this may have led to some small dis- crepancies with the results from previous studies trained on other academically feasible datasets. Perhaps most significant limitation was in using n-gram versus sfate-of-the-art LLM architec- tures for our comparison, which in principle may not generalize to the best models. We would strongly encourage those who are able to train LLMs at scale to consider offering models with morphologically valid segmentation, both to facili- tate more extensive language model comparisons, and to support scientific studies involving morpho- logical representations as articulated in Section 4.
tinger provided invaluable discussion we formu- lated our research question, and Marine Carpuat. Utku Turk, and other members of the Computa- tional Linguistics & Information Processing and Psycholinguistics groups at UMD provided insight- ful feedback on this work during various stages, Last, we thank the reviewers for suggesting im- provements and clarifications to the paper, partic- ulary comments and questions that motivated our finerggrinedd analysis.
References
Khuyagbaatar Batsuren, Gábor Bella, Aryaman Arora, Viktor Martinovic, Kyle Gorman, 7denčk Zabokrt- sky, Amarsanaa Ganbold, Šárka Dohnalová, Magda Ševčíková, Kateřina Pelegrinová, Fausto Giunchiglia, Ryan Cotterell, and Ekaterina Vylomova. 2022. The SIGMORPHON 2022 shared task on morpheme seg- mentation. In Proceedings of the 19th SIGMOR- PHON Workshop on Compulational Research in Pho- netics, Phonology, and Morphology, pages 103-116, Seattle, Washington. Association for Computational Linguistics.
Shohini Bhattasali and Philip Resnik. 2021. Using sur- prisal and fMRI to map the neural bases of broad and local contextual prediction during natural lan- guage comprehension. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3786-3798, Online. Association for Computa- tional Linguistics.
Christian Brodbeck, Shohini Bhattasali, Aura AL Cruz Heredia, Philip Resnik. Jonathan Z Simon, and Ellen Lau. 2022. Parallel processing in speech perception with local and global representations of linguistic context. ELife, 11:070066.
Mark Davies. 2010. The corpus of contemporary ameri- can english as the first reliable monitor corpus of en- glish. Literary and linguistic computing. 25(4):447- 464.
6 Ethical Considerations
All data we used are publicly available. Human ex- perimentation was approved by the institutions who conducted the research, including our own. The software we used was publicly available, and the trained morphological segmenter was distributed by the authors of the paper, so our implementa- tion and data analyses do not require specialized computing hardware.
Andrea de Varda and Marco Marelli. 2022. The effects of surprisal across languages: Results from native and non-native reading. In Findings of the Associa- tion for Computational Linguistics: AACL-IJCNLP 2022, pages 348,144, Online only. Association for Computational Linguistics.
Vera Demberg and Frank Keller. 2008. Data from eye- tracking corpora as evidence for theories of syntactic processing complexity. Cognition, 109(2):193 193-210.
7 Acknowledgements
Peter Donhauser and Sylvain Baillet. 2020. Two distinct neural timescales for predictive speech pro- cessing. Neuron, 105(2):385-393.
This material is based upon work supported by ONR MURI Award N00014-18-1-2670. We would like to thank Silvan Wehrli for providing the English morphological segmentation model. John Hale, Tal Linzen, Brian Dillon, and Allyson Et-
Susan F. Ehrlich and Keith Rayner. 1981. Contextual effects word perception and eye movements dur- ing reading. Journal of Verbal Learning and Verbal Behavior, 20K6):641-6555

---

Allyson Ettinger, Tal Linzen, and Alec Marantz. 2014, The role of morphology in phoneme prediction: Evi- dence from meg. Brain and language, 129: 14-23.
John M Henderson, Wonil Choi, Matthew W Lowder, and Fernanda Ferreira. 2016. Language structure in the brain; A fixation-related fmri study of syntactic surprisal in reading. Neuroimage, 132:293-300.
Richard Futrell, Edward Gibson, and Roger P Levy. 2020. Lossy-context surprisal: An information- theoretic model of memory effects in sentence pro- cessing. Cognitive science, (e(3)x12814.
Alan Kennedy, Robin Hill, and Joël Pynte. 2003. The dundee corpus. In Proceedings of the 12th Earopean conference eye movement.
Richard Futrell, Edward Gibson, Harry J. Tily, Idan Blank, Anastasia Vishnevetsky, Steven Piantadosi, and Evelina Fedorenko. 2018. The natural stories corpus. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Alexandra Krauska and Ellen Lau. 2023. Moving away from lexicalism psycho- and neuro- -iuguuiingssss Frontiers in Language Sciences, 2.
Jiaxuan Li and Allyson Ettinger. 2023. Hcuristic in- terpretation rational inference: A computational model the n400 and p600 in language processing. Cognition, 233:105359.
Ariel Goldstein, Eric Ham, Samuel A Nastase. Zaid Zada, Avigail Grinstein-Dabus, Bobbi Aubrey, Mar- iano Schain, Harshvardhan Gazula, Amir Feder, Werner Doyle, et al. 2022. Correspondence between the layered structure of deep language models and temporal structure natural language processing in the human brain. biokxiv, pages 2022-07.
Paul A Luce. 1986. A computational analysis of unique- ness points in auditory word recognition. Perception & Psychophysics, 3953):15--158
David Marr. 1982 Vision: A Computational Investiga- tion into the Human Representation and Processing of Visual Information. Henry Holt and Co., Inc., USA.
Adam Goodkind and Klinton Bicknell. 2018. Predictive power of word surprisal for reading times is a linear function of language model quality. In Proceedings of the 8th Workshop on Cognitive Modeling and Com- putational Linguistics (CMCL 2018). pages 10-18, Salt Lake City, Utah. Association for Computational Linguisties.
James A Michaelov, Megan D Bardolph, Cyma K Van Petten, Benjamin K Bergen, and Seana Coul- son. 2023. Strong prediction: Language model sur- prisal explains multiple n400 effects. Newrobiology of Language, pages 1-71.
Laura Gwilliams. 2020. How the brain composes mor- phemes into meaning. Philosophical Transactions of the Royal Society B, (75(1791):201
Hermann Ney, Ute Essen, and Reinhard Kneser. 1994. On structuring probabilistic dependences stochas- tic language modelling. Computer Speech & Lan- guage, 8(1):1338.
John Hale. 2001. probabilistic Earley parser as a psy- cholinguistic model. In Second Meeting of the North American Chapter of the Association for Computa- tional Linguisties.
Byung-Doh Oh, Christian Clark, and William Schuler. 1021. Surprisal estimators for human reading times need character models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 3746-3757, Online. Association for Computational Linguistics.
M. Halle and A. Marantz. 1993. Distributed morphol- ogy and the pieces of inflection. In K. Hale and S.J. Keyser, editors. The view from Building 20: Essays in honour of Sylvain Bromberger, volume 20 of Current studies in linguistics series, pages 111-176. Cam- bridge, MA: MIT Press, Cambridge, Massachusetts.
Byung-Doh Oh and William Schuler. 2023. Why does surprisal from larger transformer-base language models provide a poorer fit human reading times? Transactions of the Association for Computational Linguistics, 1:006-350.
Yiding Hao, Simon Mendelsohn, Rachel Stemeck, Randi Martinez, and Robert Frank. 2020. Probabilis- tic predictions of people perusing: Evaluating metrics of language model performance for psycholinguis- tie modeling. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguisties, pages 75 15-86, Online. Association for Computational Linguistics.
Yohei Oseki and Alec Marantz. 2020. Modeling mor- phological processing in human magnetoencephalog- raphy. Proceedings of the Society for Computation in Linguistics, 3(1):209-219;
Martin Haspelmath. 2017. The indeterminacy of word segmentation and the nature of morphology and syn- tax. Folia linguistica, 51 51(11000):11888
Milena Rabovsky, Steven S Hansen, and James I. Mc- Clelland. 2018. Modelling the n400 brain potential as change in a probabilistic representation of meaning. Nature Human Behaviour, 209):693-705.
Kenneth Heafiel.. 2011. KenLM: Faster and smaller language model queries. In Proceedings of the Sixth Workshop Statistical Machine Translation, pages 187-197, Edinburgh, Scotland. Association for Com- putational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2018. Language models are unsupervised multitask learners.

---

Soo Hyun Ryu and Richard Lewis. 2021. Accounting for agreement phenomena in sentence comprehen- sion with transformer language models: Effects of similarity- based interference on surprisal and atten- tion. In Proceedings of the Workshop on Cognitive Modeling and Compulational Linguistics, pages 61- 71, Online. Association for Computational Linguis- tics.
Thomas Wolf. Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- rie Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of ofe--ff-thhee of-the-ar art natural language processing. In Proceedings of the 2020 Conference Empirical Methods Natural Language Processing: System Demonstrations, pages 38 38-45, Online, Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 7715-1725, Berlin, Germany, Association for Computational Lin- guistics.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, al. 2016. Google's neural machine translation system: Bridging the gap between human and machine trans- lation. arXiv preprint 1609:06009.88
Cory Shain, Idan Asher Blank, Marten van Schijndel, William Schuler, and Evelina Fedorenko. 2020. fmnri reveals language-specifee predictive coding during naturalistic sentence comprehension. Neuropsycholo- gia, 138:107307,
Cory Shain, Clara Meister, Tiago Pimentel, Ryan Cot- terell, and Roger Philip Levy. 2022. Large-scale evi- dence for logarithmic effects of word predictability on reading time.
Jinbiao Yang, Antal Van den Bosch, and Stefan L Frank. 2022. Unsupervised text segmentation predicts eye fixations during reading. Frontiers in Artificial Intel- ligence, 5:1
Herbert A. Simon. 1969. The sciences of the artificial. The sciences of the artificial. The MIT Press, Cam- bridge, MA. US. Pages: xii, 123.
Nathaniel Smith and Roger Levy. 2013. The effect of word predictability on reading time is logarithmic. Cognition, 322(3))30--319
Tal Yarkoni and Jacob Westfall. 2017. Choosing predic- tion over explanation in psychology: Lessons from machine learning. Perspectives on Psychological Science, 12(6):1 100-1122.
Miloš Stanojević, Jonathan R Brennan, Donald Duna- gan. Mark Steedman, and John T Hale. 2022. Mod- eling structure-but building in the brain with ceg pars- ing and large language models. arXiv preprint arXiv:2210.1611
A Comparing Word Segmentation Methods
Adrian Stauh. 2015. The Effect of Lexical Predictability on Eye Movements in Reading: Critical Review and Theoretical Interpretation: Predictability and Eye Movements. Language and Linguistics Compass, 9())33113377.
In this illustrative example, the BPE-based tok- enizer fails to split up most multimorphemic words. When it succeeds, the words are not segmented by morpheme (relegated, fringes). The morpholog- ical transducer is able to make cognitively plau- sible choices involving tense (relegates), posses- sives (its), and plurals (fringes). It also splits up more complex words into roots, prefixes, and suf- fixes (coverage, reporters, journalistic, commu- nity). Both tokenizers marked word boundaries in their output, although they are not shown in this example.
Silvan Wehrli, Simon Clematide, and Peter Makarov, 2022. CLUZH at SIGMORPHÖN 2022 shared tasks on morpheme segmentation and inflection generation. Proceedings of the 19th SIGMORPHOO Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 212-219, Seattle, Washing- ton. Association for Computational Linguistics.
Ethan Gotlieb Wilcox, Jon Gauthier, Jennifer Hu, Peng Qian, and Roger Levy. 2020. On the predic- tive power of neural language models for human real-time comprehension behavior. arXiv preprint arXiv:2006.09112
In Tables and 4 we report the number of words that were split by both the BPE tokenizer and the morphological segmenter in the psycholinguistic corpora.
Ethan Gotlieb Wilcox Tiago Pimentel, Clara Meister, Ryan Cotterell, and Roger P Levy. 2023. Testing the predictions surprisal theory 11 languages. arXiv preprint 2307 036677.

---

Segmentation
Sentence
Corpus
Tokens Per
Percent of Corpus
Percent
Method
of Corpus
Orthographic
the sporadic nature of press coverage of the court often relegates its reporters to the fringes of the journalistic community
Word
Excluding
Words
Stopwords
94.4
88.5
4.19
8.68
Dundee
3
1.22
2.53
BPE Tokenizer
the sporadic nature of press coverage of the court often releg ates its reporters the fr inges of the journalistic community
4
0.104
0.217
Output
5
0.005
0.011
95.2
89.9
Natural Stories
3.85
8.01
0.971
2.02
Morphological Transducer Output
the sporadic nature of press cover age of the court often relegate it port er to the fringe of the journal istic commune ity
0.016
0.03
Table 3: Words the psycholinguistic corpora split into different numbers of tokens by the BPE tokenizer.
Table Illustrative example of the same sentence from COCA tokenized orthographically, morphologically, and using BPE.
Corpus
Tokens Per Word
Percent Corpus
Percent of Corpus Excluding
B Examples of Surprisal Differences for Morphemes and BPE tokens
Stopwords
75.7
55
Figures and 5 provide two illustrations of sur- prisal differences between subword segmentations. Note the major difference in the surprisal of bulb when summed over BPE tokens when compared to morphological units bulb In the sentence in Figure 4, the GPT2 tokenizer split tulips into tul ips and did not split bulbs. It is reasonable for human comprehender to expect the word bulb immediately after tulip since they would cooccur frequently text, but less predictable after 1ip. This is reflected in the higher surprisal under the BPE--based model.
21
38.3
Dundee
3
3
6.18
0.218
0.451
5
0.011
0.022
76.9
58.3
Natural Stories
2
20.9
37.1
3
2.05
4.27
4
0.125
0.26
Table Words in the psycholinguistic corpora split into different numbers of tokens by the morphological segmenter.
ShC
255
Figure 5, which is not split up by the BPE tok- enizer. The morphological transducer, on the other hand, split carefully into care ful ly, producing much lower surprisal. This suggests that further item- wise comparisons involving words with more morphologically relevant units may be worth inves- tigating.
Maried
19
collndion
bulla
Tai
Figure 4: Differences in surprisal of plural (bulbs) un- der n-gram models using morphological BPE-bbaed tokenization, from sentence in the Natural Stories cor- pus.
To take a more morphologically complex ex- ample, this difference also occurs for carefully in

---

gooding
Figure 5: Differences in surprisal of adverb (care- fully) under n-gram models using morphological and BPE-based tokenization, from a sentence in the Dundee corpus.
C Statistical Testing for Predictive Power of Surprisal
C.1 Effect Sizes
In Table 5 we report effect sizes for the regres- sion models trained on the full psycholinguistic corpora via the widely used Cohen's f However, since our work replicating subliterature that almost exclusively uses LoggLik, we used that measurement in the main body of the paper. We refer readers to Goodkind and Bicknell (2018) for further statistical justification of that choice.
Tokenization
2 for Dundee
f2 for Natu-
Method
ral Stories
Orthographic
0.021
0.018
BPE
0.021
0.02
Morphological 0.021
0.017
Table 5: Effeet size comparing surprisal feature of regression models relative to controls.
C.2 Statistical Significance Testing
For the aggregate analysis (Figure 1, we used Wilcoxon rank-sum test to compute significance. We find no statistically significant difference be- tween the Lig estimated for the folds for the two subword tokenization methods relative to predictions orthographic words.
Tokenization
Corpus
W
p
BPE
Dundee
43
0.63
Morphological
Dundee
50
BPE
Natural Stories
41
0.53
Morphological
Natural Stories
53
0.85
Table 6: Wilcoxon rank-sum test comparing distribu-
tions of Loo from cross-vas vass-vvlidation results of mor- phological and BPF versus orthographic surprisal.


Reviews:
[{"review_id": "oU6cB3jXet", "reviewer": "Reviewer_aGqM", "summary": "", "questions": "Question 1:\nWhat is the purpose of reporting the per-token log likelihood on a held-out test set using 10-fold cross validation? The contribution/motivation could be stated more clearly.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their interest in this work and their helpful comments and suggestions.\n\nRegarding the judgment criterion, our intent, consistent with short-paper criteria, was a small, focused contribution: since the focus here is on tokenization, we vary only that, and keep everything else, including surprisal as the metric, consistent with the existing methods in the literature. Determining whether incorporating surprisal into a regression model improves its estimates of reading times is standard practice for such analyses (Smith & Levy, 2013, current practices use metrics from Goodkind & Bicknell, 2018 and Wilcox et al, 2020). That said, we completely agree that finer-grained analyses are helpful and motivate future research into the role of morphemes in lexical prediction, as suggested by both you and Reviewer jJ46, as well as being of further interest for cognitive scientists, as Reviewer MAqk notes. As discussed below, we have now done some additional fine-grained analysis as suggested, and although status quo tokenizations for LLM-based psycholinguistic studies look safe in the aggregate based on surprisal, the finer-grained results indicate a more nuanced and interesting story.\n\nAs for the specific metrics we used, both our methodology and results were consistent with previous published literature: Wilcox et al (2020) reported comparable values for Delta(LogLik) on both corpora we tested, this was also the case with Goodkind & Bicknell (2018)’s findings for the Dundee corpus. In addition, we can add brief findings on effect sizes. Using Cohen’s f^2, we found comparable effect sizes of the surprisal feature across models and corpora (around 0.02), when we compared the models using surprisal with models fitted only for the control features. We will report these statistics if the paper is accepted, and please also see Reviewer jJ46’s comment on small effect sizes, as well as our responses, for further discussion. \n\nIn finer-grained analysis, we investigated the extent to which tokenization was actually taking place. As it turns out, the BPE tokenizer only split 5% of the tokens in the psycholinguistic corpora (11% when ignoring stopwords), compared to ~25% and ~44% respectively for the morphological analyzer. Moreover, surprisal theory would suggest processing effort increases incrementally with the number of units a word is segmented into. However, this appears to be true onlyfor the morphological tokenization: for BPE tokenization there is a sharp jump from low surprisal with unsplit words to essentially equal surprisal for words split into 2, 3, and 4 tokens.  In addition, replicating our experiment in a finer-grained way, separately for non-segmented and segmented words, the predictive power of the BPE-based model is worse for words that get split by the tokenizer, but this is not true for the morpheme-based model. Taking all of this together, the finer-grained results suggest that, notwithstanding our aggregate results, there are, in fact, important differences between BPE and morphological segmentation with regard to surprisal. (These were already hinted at by Appendix B in the submission.) If accepted, we will flesh this out, including numbers and figures as appropriate.\n\nRegarding the choice of tokenization schemes, we used orthographic tokenization as a baseline. We chose this because it is how the stimuli were presented to the human participants. N-gram based studies like Smith & Levy (2013) did not consider how words were split, but since LLM-based estimates are now the state of the art, it is important to determine how credible they are because they operate over tokens, which may or may not necessarily correspond to words. We chose the implementation of BPE tokenization used by GPT2, since Oh & Schuler (2022) and Shain et al (2022) showed that larger models were worse predictors of reading times. These details were mentioned in a footnote, but we will promote them to the main text if accepted. Since morpheme-level information has been shown as a component of lexical prediction in humans (Gwilliams, 2020), we used a state-of-the-art morphological transducer that outperformed conventional approaches and LLM tokenizers in the 2022 SIGMORPHON shared task on morphological segmentation. We will make these details clearer in the final version of the paper.\n\nRegarding why we reported cross-validated measurements of the models’ predictive power, we wanted to replicate the findings of Smith and Levy (2013) and Wilcox et al (2020), but only varying tokenization for the n-gram models. This means that we would also need to use the same evaluation metrics. We reported predictive power per token, instead of the aggregate value as in Goodkind & Bicknell (2018), because we were working with multiple corpora as in Wilcox et al (2020). Reporting the value of a cross-validated regression model is important to ensure that the predictive power measures computed on the complete dataset are not the result of overfitting. We will make these motivations clearer in the final version if accepted. \n\nWe appreciate the comments on the presentation of the Methods section. We will prioritize ensuring all the relevant details get discussed in their appropriate subsection, as well as having more appropriate subsection titles, if accepted.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The strength of the paper is that it tapped into the inner workings of language models and tried to find the connection between its inner working and the validity to evaluate language models as psycholinguistic subjects. It is always an interesting question to ask how tokenization affects training, and testing, and how that affects metrics that parallel human sentence processing. I believe that looking at surprisals and the relation to reading time is one of the operational validation strategies. It would be exciting to see more validation works come around.", "reasons_to_reject": "1. The judgment criterion could be improved. It seems that this paper relied on whether the surprisal factor significantly helped with the prediction of the reading time in regression models as a criterion to judge the validity of the surprisal measure. More fine-grained or direct criteria could be checked. For example, was the effect size of the surprisal factor across the various surprisals similar? If not, what conclusions could be generated? \n\n2. It is interesting that the tokenization/subword information is studied, but I would appreciate seeing more discussion about why you chose the three ways of tokenization in addition to other existing ones. It feels like the choice was random but I am sure there was a valid reason.", "questions_to_authors": "Question 1:\nWhat is the purpose of reporting the per-token log likelihood on a held-out test set using 10-fold cross validation? The contribution/motivation could be stated more clearly.", "paper_topic_and_main_contributions": "This paper tests whether the tokenization process in current NLP practices underlyingly affects the probabilities of words to the extent that computational linguists might extract distorted insights from the evaluation of large language models. This research question was operationalized by looking at surprisals composed of different tokenization methods and testing if the surprisals can help improve the predictability of word reading times, as existing research showed. The findings suggest that the three ways of tokenization all contributed to the prediction. The research provided a first validation and indicated that it is safe to proceed with the status quo tokenizations for LLM-based psycholinguistic studies.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The \"Method\" part could be improved with more clarity. One way that helped with my understanding could be separating into clearer sections, eg. Language Models, Data, Comparison, etc. and making sure that each section only discussed the relevant part. Currently, the \"Language Models\" part not only missed the model selection (which was stated in the preceding text) but also included data description. It could be improved further."}}, {"review_id": "vD2gM32QgE", "reviewer": "Reviewer_jJ46", "summary": "", "questions": "If this paper is accepted, I'd like to see the authors address the questions I outlined above.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed comments and concerns about the work, and we appreciate your noting this is the first work to address the question of human/LLM segmentation differences. We agree the question is important and share your hope that it will stimulate further research in this direction. If accepted, we will certainly address both major questions in the final version. \n\nRegarding low predictive power for surprisal, the values we report are consistent with those reported by Wilcox et al (2020), who evaluated 5-gram models on the same corpora using the same metric (also around 0.01), providing some degree of confidence that we have not introduced unexpected methodological glitches. All that said, the reviewer is right to question how much of language processing is accounted for by predictive processing, operationalized as surprisal; this is an active direction of research we are currently pursuing, along with many others in the field. We will include relevant discussion in the final version if accepted.\n\nRegarding broader theoretical discussion, there is good evidence that prediction is one element in a theory of human language comprehension (cf. strong correspondence of human neural recordings with LLM outputs in Goldstein et al 2022), and LLMs are useful estimators of the predictability of upcoming linguistic information; however, to be clear, we do not take LLMs themselves to be cognitive models, and we would not take that position even with extremely strong correlations between surprisal and indices of human effort. Surprisal, operationalized using an LLM or any other probability estimates, is at best part of an explanation at Marr’s “computational” level, with no direct connection to algorithms/mechanisms or representations, things that we take to be essential elements of a cognitive model in the sense under discussion here.  In our view, this sub-field has come to rely too much on LLM-based surprisal estimates, especially over words, which do not always correspond to the units humans are predicting (Krauska & Lau, 2023; Brodbeck et al, 2022). This issue is part of what motivated this study to begin with.  Evaluating more explicit models (including algorithms and representations) with behavioral and neural data, as well as developing newer models, are critical directions for future research that we are currently exploring, along with others in the field. One such direction we are pursuing, for example, involves the model of Futrell, R., Gibson, E., & Levy, R. P., 2020,. Lossy‐context surprisal: An information‐theoretic model of memory effects in sentence processing. Cognitive science, 44(3), e12814, within which the incorporation of memory effects presents an opportunity for including explicit psychological models of memory and/or attention. \n\nEven with regard to the present paper, our finer-grained analysis (see response to reviewer aGqM) demonstrates that, despite the results indicating the BPE-based surprisal is solidly correlated with reading times in the aggregate, it differs in an potentially important way from morphological segmentation when we look more closely at non-segmented versus segmented words. We view the results from this finer-grained analysis, in response to our reviews, as a step in the direction of the kind of cognitive model in which we are interested.  Those results also suggest going more deeply into the role of morphological segmentation on an item-by-item basis; for example, training an LLM on morphological units and further following Wilcox et al. (2020) in running on test suites that evaluate (morpho)syntactic generalization. \n\nAgain, we very much appreciate your careful attention to the paper and your recommendations with regard to the theoretical and cognitive implications of the work. The discussion above offers a sense of how we are thinking about these issues and we agree with you that it should be possible to incorporate relevant discussion given the additional space in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is, to my knowledge, the first to address the apparent differences between segmentation between humans and LLMs. This is an important question to be asked, and I hope that if the paper is accepted, it will stimulate further research in this direction.", "reasons_to_reject": "Given the track the work was submitted to, I'd have liked to see the authors take a firmer stance on the cognitive implications of their findings. Certainly, just because subword tokenization approaches predict similarly well to morphological segmentation approaches doesn't mean that they are equally cognitively plausible. It's unclear to me where this leads us: if we know that LLMs are doing processing differently than humans, why are we using them to predict reading times at all? Do the authors think that incorporating morphological segmentation would solve these problems, and if so how? Would an LLM making use of morphological segmentation be a plausible cognitive model? I realize that these are challenging questions to answer but the nature of a paper exploring such questions is that it is significantly weakened by the authors failing entirely to address any of them.\n\nI'd also like to see the authors address the overall predictive power of surprisal that they report, which seems extremely low across the models -- low enough that I wonder if there may have been methodological issues at play. If the authors are confident in these numbers, then it seems that there are implications more broadly for surprisal theory, and I'd like to see these discussed. Are we really confident in a theory with a predictive power of 0.01?\n\nI think the concerns I raised could be addressed in the additional page with extra discussion, but my main hesitations about this paper at the moment are (1) there may be methodological issues leading to the low predictive powers reported and (2) it lacks discussion of theoretical implications of the findings.", "questions_to_authors": "If this paper is accepted, I'd like to see the authors address the questions I outlined above.", "paper_topic_and_main_contributions": "This paper aims to address the question of whether it is a problem for the use of LLMs in surprisal theory that LLMs are trained on subword token prediction while humans do morphological segmentation. The authors compare the predictive power of surprisal in 5-gram models trained on segmented morphemes and on subword tokens and show that there is no significant overall difference in the predictive power of the two models, though they give some examples in the appendix in which the morphological segmentation approach fares better. Though the authors conclude that the use of subword tokenization doesn't have catastrophic implications for the uses of LLMs in psycholinguistic research, they argue that morphological segmentation is still important and should be further explored.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}, {"review_id": "bA2zzNxzHf", "reviewer": "Reviewer_MAqK", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We very much appreciate your positive comments, particularly on the relevance of this work to both cognitive scientists and the NLP community. That said, we would refer you to our responses to the other reviewers. The additional, finer-grained analysis we have done, a closer look at what’s happening at the level of segmented versus non-segmented words, turns out not to be an overall vindication of LLM-style tokenization for cognitive researchers.\n\nIn our view, however, this additional result does not undermine the main message of the paper, which is that LLM-style tokenization, with its differences from morphological segmentation, should not simply be accepted blindly as people study the potential value of LLMs in cognitive research and their relationship to human processing. Indeed, we believe the results are now rather more important and interesting: even if there are more superficial or aggregated correspondences, one cannot simply “check the box” and we believe that compared to our original result, these more nuanced results (obtained in response to reviewer feedback) will encourage more rather than than less of the important discussion that should be taking place about LLM-related work and its relationship to cognitive modeling.\n\nFinally, we agree wholeheartedly that extending this work to languages with different morphological systems is an important direction for future research. We will make sure to reinforce this point in the final version if accepted.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper have several merits that would benefit both cognitive scientists and the NLP community. On the cognitive science/psycholinguistic side, the authors are testing whether morphological units are a significantly better fit for human measurements. If this turns out to be the case, this would be a strong result, making cognitive researchers/psycholinguistics using pre-trained LLMs aware of the real challenges. \n\nOn the NLP side, the authors have used state-of-the art language models (e.g. models using BPE, and an n-gram model based on the output of a morphological transducer), presented their method very clearly, and compared the results with eye movement data. \n\nOne result of the work is that surprisal improves predictions of reading time (relative to the control features), which is consistent with previous research. This is shown under each tokenization scheme (evaluated using 10-fold cross-validation for each corpus). The main finding over both corpora is that the predictive power of surprisal in the models using BPE and the morphological transducer is not statistically significant different from the model using orthographic words. \n\nAll in all, the authors have shown, quite convincingly, that there's no difference in the predictive power of surprisal associated with tokenization. This is a good result particularly for cognitive scientists. \n\nAs the authors themselves report, similar tests should be run over typologically different languages that are morphologically richer than English. Given the clarity of the method, the work can be replicated by future researchers over new languages, given that there's publicly available human data for those languages, such as eye movement data. ", "reasons_to_reject": "I don't have any reasons to reject this paper. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper sets out to investigate an assumption that comes with using LLMs over psycholinguistic data. It specifically examines whether LLM-based predictions based on subword tokenization, rather than decomposition of words into morphemes matters.  The authors compare\nsurprisal estimates using orthographic, morphological, and BPE tokenization against reading time data. Various publicly available resources are used, such as the COCA (for training a 5-gram model), the Dundee corpus and the Natural Stories corpus (for eye movement data). ", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper in question investigates the impact of tokenization methods on the predictability of word reading times in large language models (LLMs). It compares the use of subword tokenization to morphological segmentation and assesses their effectiveness in psycholinguistic research. They provide evidence that predictions using BPE tokenization do not suffer relative to morphological and orthographic segmentation. While the paper addresses an important question, there are notable concerns regarding methodology, theoretical implications, and the overall quality of the results.

Pros from the reviews:

- Relevant Research Question: The paper addresses a pertinent question about the influence of tokenization methods on the predictability of reading times in LLMs, which is of interest to both cognitive scientists and the NLP community.

- Clear Methodology: One reviewer praises the clarity of the paper's methodology and its presentation of results, which contributes to the ease of replication and future research in this area.

- Important Implications: The study's findings have implications for understanding the cognitive processes underlying LLMs' language understanding and their utility in psycholinguistic research.

Cons from the reviews:

- Methodology and Results Concerns: A reviewer raises concerns about the judgment criteria used in the paper, suggesting that more nuanced criteria could be employed to evaluate the impact of tokenization methods. Another reviewer expresses doubts about the overall predictive power of the surprisal models and calls for more rigorous exploration of theoretical implications. Both reviewers question the cognitive plausibility of LLMs in predicting reading times.

- Lack of Theoretical Discussion: A reviewer criticizes the paper for failing to address the theoretical implications of its findings. The reviewer questions the use of LLMs if they differ significantly from human cognitive processing and suggests that the paper should delve into these theoretical questions.

- Limited Scope: While a reviewer appreciates the clarity and presentation of the paper, they point out that the study focuses on English and suggests extending the research to typologically different languages with richer morphology. This limitation reduces the generalizability of the findings.

I have read the rebuttals and followed the changes after each acknowledgement. The rebuttals helped to clarify many of the reasons to reject the paper. Overall, the conference could benefit from such a short paper contribution, but this short paper contribution could also benefit from some additional content that has been raised in the rebuttal discussions. Initially, various point have been unclear and although the authors could clarify them, the paper could benefit from some additional content. Overall, all reviewers would like to see the paper as a contribution to the field and I can agree with that.
Decision: Accept-Findings