Analyzing Cognitive Plausibility of Subword Tokenization
Authors: Lisa Beinborn
Yuval Pinter
Venue: EMNLP 2023 Main (2023)
Keywords: subword tokenization
subword segmentation
cognitive signals
cognitive plausibility
lexical decision
vocabulary size
morphological segmentation
TL;DR: We evaluate subword tokenization algorithms using cognitive signals.

Abstract: Subword tokenization has become the de-facto standard for tokenization although comparative evaluations of their quality across languages are scarce. Existing evaluation studies focus on the effect of a tokenization algorithm on the performance in downstream tasks, or on engineering criteria such as the compression rate. We present a new evaluation paradigm that focuses on the cognitive plausibility of subword tokenization. We analyze the correlation of the tokenizer output with the reading time and accuracy of human responses on a lexical decision task. We compare three tokenization algorithms across several languages and vocabulary sizes. Our results indicate that the Unigram algorithm yields less cognitively plausible tokenization behavior and a worse coverage of derivational morphemes, in contrast with prior work.

Analyzing Cognitive Plausibility of Subword Tokenization
Lisa Beinborn
Yuval Pinter
CLTL Lab
Department of Computer Science
Vrije Universiteit Amsterdam
Ben-Gurion University of the Negev
Amsterdam, The Netherlands 1. ww.eenorr@vu....
Beer Sheva, Israel uvpêcs. bgw.t.gu.ccc. ac.
Abstract
Sequence
Tokens
CHUNK
RT
Acc
Subword tokenization has become the de-facto standard for tokenization, although compara- tive evaluations of subword vocabulary quality across languages are scarce. Existing evalua- tion studies focus on the effect of tokenization algorithm on the performance in downstream tasks, engineering criteria such as the compression rate. We present a new evalu- ation paradigm that focuses on the cognitive plausibility of subword tokenization. We an- alyze the correlation of the tokenizer output with the response time and accuracy of human performance a lexical decision task. We compare three tokenization algorithms across several languages and vocabulary sizes. Our results indicate that the UnigramLM algorithm yields less cognitively plausible tokenization behavior and worse coverage of derivational morphemes, in contrast with prior work.
scafood
seafood
0.86
578
0.97
outfoxed
out-fo-x-ed
0.50
734
0.62
"brithbloom
br-ith-blo-om
0.60
693
0.97
#catchwind
catch-wind
0.78
788
0.82
Table I: Examples of lexical decision stimuli, the aver- age response time (RT) and accuracy (Acc) of human responses, and the output of a WordPiece tokenizer with vocabulary size of 50,000. Non-words are marked with asterisk (*). Chunkability (CHIun)) is calcu- lated based on the rate of tokens per character (1).
sequences as single tokens (e.g., seafood), and split less frequent ones into multiple tokens com- posed of frequently occurring character sequences (e.g., seabirds + seab-ird-s). Subword splits might coincide with morpheme boundaries (the plural marker s), but not necessarily (seab).
Previous comparisons of tokenization algorithms focused on engineering-oriee
1 Introduction
desiderata such processing speed and encoding efficiency, and on the performance of models on downstream NLP tasks (Rust et al., 2021). In this paper, we evalu- ate subword tokenizers from a cognitive perspec- tive, utilizing lexical decision task measures as a proxy for the processing complexity of individual words. We evaluate the split rates of three tok- enization algorithms various languages. We find significant correlations line with cognitive expectations, allowing systematic analyses of the influence of parameters such as vocabulary size. We observe that the UnigramL.M tokenization al- gorithm (Kudo, 2018) produces less correlative splits than BPE (Sennrich et al., 2016) and Word- Piece (Schuster and Nakajima, 2012), in contrast with previous evaluations made over corpus statis- ties and downstream tasks (Bostrom and Durrett, 2020), In further experiments, we find that multilin- gual token vocabularies inhibit tokenizers' ability to predict cognitive performance as well as signs that current popular vocabulary sizes are insuffi-
When we develop natural language processing (NLP) models, we first need segment a stream of text into small processable units. This preparatory step is known as tokenization and it is more chal- lenging than the segmentation of continuous sensor signals because human language uses symbolic representations. Traditionally, the space-delimited word considered meaningful basic unit, but the concept cannot intuitively be mapped lan- guages with rich morphological structure such as Turkish or Finnish, or to languages that use a writ- ing system without white spaces such as Chinese. More recently, the focus has shifted to smaller character sequences known as subwords, with the explicit goal of limiting the necessary vocab- ulary size (which affects model size and perfor- mance), and the implicit hope of better approx- imating leaaantcclll-mmannneee
linguistic units below the word level, i.e., morphemes. In prac- tice, today's i dominant subword tokenization algo- rithms are purely data-driven. They treat frequent

---

cient for morphologically-ricc languages, echoing recent findings (Liang al., 2023).
similar vein, Park al. (2021) find that BPE does not properly reflect morphological complexity and that enriching the model with explicit mor- phological information leads to reduced language modeling surprisal. Hofmann et al. (2022) show that vocabulary with better morphological cov- erage leads to better performance in genre classi- fication of English titles, and might lead to better generalization capabilities (Hofmann et al., 2021). Other studies have shown that these consistent results in English do not necessarily generalize to other languages (Mager et al., 2022), particu- larly morphologically-rio ones (Klein and Tsar- faty, 2020).
2 Self-Supervised Tokenization
A tokenizer takes as input a sequence S of charac- ters [cl,..., cn] and splits it into non-overlappng substrings, the tokens [t,,...,fa]] fa., where & S n. Each token 4 consists of a variable number of j consecutive characters (1 S S n), such that the concatenation of the tokens 4 yields the sequence S. A tokenizer consists of a vocabulary consisting of m tokens and algorithm that determines the best splits of the input S into vocabulary items t,. See Mielke et al. (2021) for a detailed survey of tokenization approaches.
Morphological segmentation is related to to- kenization, but it is sensitive to phonotactic variations, e.g., discernible is segmented into di scern and -able (Batsuren al., 2022). Never- theless, the winning system the SIGMORPHON shared task was based on subword tokenization and outperformed character-base approaches, indicat- ing that subwords can approximate morphological boundaries (Peters and Martins, 2022).
Evaluating Vocabularies Comparative evalua- tions of tokenization algorithms commonly focus on downstream performance and on cross-lingual differences. Maronikolakis et al. (2021) calcu- late the tokenization compatibility for pairs of lan- guages and find that the vocabulary size of a tok- enizer needs to be adapted to the characteristics of the language. Multilingual language models use single shared vocabulary for a large number of languages facilitate cross-lingual transfer, how- ever, Rust et al. (2021) find improvements when these are replaced with targeted monolingual to- kenizers. Liang et al. (2023) propose to increase the vocabulary size for multilingual models and assign per-language budgets in a dynamic man- ner, in order to mitigate effects on the splitting ratio for languages less represented in the vocabu- laries. They de-emphasize token sharing between languages with little lexical overlap, in line with Chung et al. (2020). Yehezkel and Pinter (2023) propose to incorporate context sensitivity in order to generate more cohesive tokenization and show that their approach leads to increased downstream performance for both English, and the morphologi- cally more complex language Turkish.
Cognitive Plausibility From cognitive perspec- tive, it remains an open question to which extent lexical processing is driven by morphological units. One of the most robust effects in lexical decision tasks (Amenta and Crepaldi, 2012) is that mor- phologically structured non-words cause longer re- sponse times and lead to decreased accuracy in word detection. Beyersmann et al. (2020) find that this effect is stronger for German than for French, and suggest that this is due its larger degree of morphological productivity. Dawson et al. (2021) compare lexical decision times for English words and find that priming with morphological compo- nents (teach) leads to faster responses (teacher) even if the prime has no semantic relation (corn corner). (Yang et al., 2022) show that the pre- diction of eye fixations is facilitated for English and Dutch readers by operating on subtokens deter- mined by unsupervised tokenizers instead of word units. Stevens and Plaut (2022) claim that effects attributed to morphological decomposition cannot be easily disentangled from frequency effects, and urge NLP researchers to integrate response times into the evaluation of distributional approaches.
Morphological Evaluation More linguistically motivated evaluations of subword tokenization fo- cus on morphological plausibility. Bostrom and Durrett (2020) compare the BPE and UnigramLM al algorithms for English and Japanese and find that the segmentation produced by the latter aligns more closely with morphology and leads to better results on downstream tasks, especially for Japanese. In
3 Experimental Setup
We train three tokenization algorithms on 100,000 sentences from the news domain of the Leipzig
All analyses are available on github: https://gtt thub. com/ hom/cclpp-aa/ccage lab/cogtok

---

Language
Participants
Words
Non-Words
their first language. The datasets contain infor- mation about the average response time (i.e., the number of milliseconds it took the participants to make a decision)* and accuracy for each stimu- lus. Table 2 provides an overview of the number of participants and stimuli for each dataset. Each participant only saw subset of the stimuli; further details about the data collection are available in the original references. Since the Spanish study was a crowd-soureing project, we removed outliers with a reported response time in the first and last percentiles (<484 and >7,753 ms, respectively).
English
78
28,730
27,137
Dutch
81
14,089
14,089
French
975
38,335
38,807
Spanish
209.351
45,223
56,861
Table
Summary statistics of the lexical decision data.
corpus (Biemann et al., 2007), 2 and introduce the chunkability metric to evaluate tokenizer output against cognitive data from lexical decision task.
Tokenization Models We use the Huggingface implementations of three corpus-based based subword algorithms: byte-pair encoding (BPE), WordPiece (WPC), and Unigram (UNT).²
Metric A sequence of characters [c1,.... cn] is Ga]
split into tokens [t...... [t]] We base our metric the intuition that fewer splits generally indicate better fit and that longer sequences are more likely to be split. Our metric therefore takes both n and into account to measure how well the tokenizer handles the sequence. We define the chunkability of a sequence as a value that decreases as the ratio of tokens over characters increases:
BPE originated as a compression algorithm (Gage, 1994; Sennrich et al., 2016) and has been used in large pre-trained language models such as GPT-2 (Radford al., 2019). BPE vocabularies are built bottom-up, starting with an initial vocab- ulary of all characters. The algorithm then itera- tively merges sequences of characters frequent in the corpus and adds them as tokens to the vocab- ulary until reaching the maximum size. During inference, an input sequence is greedily split into tokens aiming for a minimum number of splits, see Gallé (2019) for more detailed description. WordPiece (Schuster and Nakajima, 2012) is variant of BPE which adds tokens to the vocabu- lary when they maximally increase the likelihood of an n-gram-based language model in the corpus, and is decoded greedily left-to-right to find the locally longest token available at each step. The UnigramLM (Kudo, 2018) algorithm, in contrast, takes top-down approach starting with an overly large vocabulary of all possible tokens, followed by iteratively pruning those that lead to minimal loss of likelihood over the corpus when they are removed from token-unigram language model.
# #tokens
k
chunkability 1-
=
(1)
#chars
n
If a token is split into individual characters, chunk- ability is zero. For tokens that are not split, chunk- ability approaches one the number of characters increases, reflecting the increasing challenge of fitting long words into the vocabulary."
Results
We test the hypothesis that sequences with higher chunkability are easier to process for humans and more likely to be considered words. Figure visualizes Pearson's correlation between the chunk- ability of a sequence and the response time and ac- curacy observed from humans rating the sequence, and Table illustrates some typical examples. We can see that for a sequence that qualifies as a word, like seafood, a higher chunkability score (i.e., cas- ier processing by the tokenizer) is likely to co-occur with higher accuracy and a lower response time. For non- words, observe the reverse tendencies: non-word sequences with high chunkability such as catchwind require longer response times and
Cognitive Data We use data from lexical deci- sion tasks in British English (Keuleers et al., 2012), Dutch (Keuleers et al., 2010), French (Ferrand al., 2010), and Spanish (Aguasvivas et al., 201 8). In these tasks, participants are presented with sequence of characters, e.g., thornier, and de- cide whether the sequence forms valid word in
*While this duration is usually characterized as reading time in the resources, we agree with the observation a re- viewer that it remains unclear how much time the participants spend reading and use the term response time instead.
iIn pilot experiments, we explored larger training sizes but did not observe relevant differences. tht thub. com/huggingfacee wttgginnnaaeeeeeeeeeeeeeeeee tokenizers. We focus on character-segment models, while other approaches operate on the single character, byte, or pixel level (Clark et al., 2022; Xue et al., 2022; Rust et al., 2023).
'For comparison, we also ran our experiments by using the number of splits (without normalization) as our metrie, see Appendix B.

---

Accurnty
Recoonse Time
Words
foo: Verds
04
Langgaee
English
Peunson' Condaation
French
en
-0.2
Engish Dutthh Trench Spanish
Engiihh Dutch Trench Sparish
X.Unguul X0n
Moro XiLingual
02
Figure 2: Differences between pre-trained monolingual and multilingual tokenizers measured Pearson cor- relation between the chunkability words and the ob- served accuracy and response time of human partici- pants in lexical decision task. All differences between the tokenizers' performance are statistically significant (p<0.01), except for comparisons of the multilin- gual tokenizers (correlation with accuracy for French and correlation with reading time for Spanish).
Con
Figure 1: Pearson correlation between the chunka- bility of sequence and the observed accuracy and response time of human participants in a lexical de- cision task. All tokenizers are trained on the same data (per language) with a vocabulary size of 50,000. In all cases, model results are significantly differ- ent from the length baseline (p<0.01) according to the Fisher z-transformation (using an implementation by Philipp Singer: https:// /ii thub. cwm/csoocc/icceeee com/psinger/ / CorrelationStats)) The comparisons between UNI and BPE/WPC are significant 22 out tions. The differences between BPE and WPC are not significant in most comparisons.
5.
Eeglihh Sparish
Pessoonin Comelato
Duisi French
12
condi-
20000
20000
00000
00000
Viccabubrry Sizw
Figure 3: Pearson correlation between the WordPiece chunkability a word the observed accuracy hu- man responses to lexical decision task, vocabulary size grows.
are less accurately identified as nondwords than an unusual sequence such as brithbloom. These patterns are consistent across algorithms and lan- guages, whereas baseline that only considers char- acter length cannot capture the effeet (longer se- quences generally lead to longer response times). The correlation for the UnigramLM algorithm is systematically lower than for the other two algo- rithms, suggesting a contrast with morphology- and corpus-based measures considered by Bostrom and Durrett (2020). For French, chunkability seems to be less correlated with human responses, echo- ing findings related to cognitive performance in morphologically periiiioogccllyy-imeedddd primed environment (Beyers- mann al., 2020).
Cross-lingual Vocabulary It has been shown that the "curse of multilinguality" reduces the per- formance of multilingual models compared to their monolingual counterparts (Conneau et al., 2020). Rust et al. (2021) show that this difference in per- formance is strongly related to the tokenizer. We compare cognitive plausibility of monolingual and cross-lingual tokenizers of pretrained models Figure 2, and affirm that the monolingual tokenizer aligns much better with human responses than the lingull ones.? 7 See also Appendix D.
While the absolute correlation scores might be of limited explanatory value, we find the relative differences between conditions a relevant point of information for further development.
Vocabulary Size and Morphology The chunk- ability values vary with the size of the vocabulary of the tokenizers. Figure 3 shows how the cor- relation with human responses increases with the vocabulary size of the WordPiece tokenizer until
'For a complementary perspective, we ran a linear regres- sion analysis on the words to compare to a frequency measure which can be found in Appendix C. We are also considering alternative correlation metrics such as Spearman's s, Kendall's ss and Goodman-Knnn Kruskal. Initial analyses indicate that they cap- ture similar tendencies.
"We use the following huggingface models: GroNLP/bert- base-dutch-cused (Dutch). bert-buse-umcused (English). camembert- base (French), decuchile/bert- waseeet---b-------s---aas wwmm- uncased (Spanish). www.-mmltillnguallnmee roberta-hae baberta-base (crosslingual). BERT-based models use Word- Piece tokenization, M-B-RoBRTTa uses BPE.
and xlm-

---

essarily align with previous work evaluating tok- enizers on morphological segmentation and down- stream performance, suggesting that our framework provides a complementary perspective to tokenizer vocabulary evaluation. Our analyses on vocabulary size and morphological coverage provide initial in- sights towards the development of cognitively and linguistically more plausible tokenizers.
Sank
Denor
Limitations
Our cognitive analyses are limited to two Romance and two Germanic languages. The response times were collected as separate experiments with slight variations in the data collection procedure (i.e., number of stimuli per participant, background of participants) and might not be directly com- parable. We average over the responses, which may conceal individual differences between respon- dents (Plank et al., 2014; Kidd et al., 2018; Pavlick and K Kwiatkowski, 2019). Pearson's p has a ten- dency to pick up spurious correlations (Aldrich, 1995), which is why we abstract from absolute val- ues and focus on relative differences between ditions. The quality of the selection of derivational morphemes is determined by the characteristics of the SIGMORPHON datatset.
Figure 4: Coverage of derivational morphemes in the vocabulary WordPiece (A) and UnigramL.M (x) for 13 languages.
it platcaus for most languages at 50,000. To test whether this effect may be related to morpholog- ical coverage, we use morphological annotations for 13 languages from the SIGMORPHON shared task (Batsuren al., 2022) and extract an inventory of derivational morphemes for each language. We only used derivational morphemes which occur in at least 0.1 of the annotations (to avoid rare mor- phemes such as oculo). yielding between 80 and 140 derivational morphemes per language. From the coverage curves in Figure 4, we see that deriva- tional coverage increases with vocabulary size, but morphologically--icc languages such as Russian, Polish, and Mongolian remain unsaturated even with vocabulary size of 50,000. This suggests that previous work on morphological segmenta- tion which used significantly smaller vocabulary sizes (c.g., Peters and Martins, 2022) did not un- cover the full potential of the approach. We also see that WordPiece tokens overall provide better cover- age of morphemes than UnigramI.M reinforcing our findings from the previous experiments.
Ethics Statement
We use datasets that have been fully anonymized and adhere to ethical guidelines for data collection. Our analyses do not reveal metadata of the par- ticipants that would enable identification. Claims about cognitive plausibility need be made with caution because the procedural patterns underlying human language processing still remain an open research question. We have therefore paid spe- cial attention to a realistic interpretation of our re- sults and avoid overpromising messages (Lipton and Steinhardt, 2019).
5 Conclusion
We propose a new evaluation paradigm for compar- ing subword tokenization algorithms using cogni- tive data. We introduce novel metric to capture the chunkability of sequence that correlates with cognitive phenomena of lexical recognition. The overall trends suggest that the connection between plausibility tasks and segmentation is mcaningful enough to be used as benchmark. We find a lower cognitive correlation for the UnigramLM algorithm than for WordPiece and BPE, which does not nec-
Acknowledgements
Lisa Beinborn's 's work was supported by the Dutch National Science Organisation (NWO) through the VENI program Ven. 2tt..111..3399 Yuval Pinter's work was supported by a Google gift intended for work Meaningful Subword Text Tokenization.
We thank the reviewers for their thoughtful com- ments. We thank Joshua Snell and Omri Uzan for comments on early drafts.
"The tendencies for BPE and UNI are comparable.

---

References
cross looss-lingual representation learning at scale. In Pro- ceedings of the 58th Anmual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451, Online. Association for Computational Lin- guistics.
Jose Armando Aguasvivas, Manuel Carreiras, Mare Brysbuert, Pawel Mandera, Emmanuel Keuleers, and Jon Andoni Duñabeitia. 2018. Spalex: A spanish lexical decision datahase from massive online data collection. Frontiers in Psychology.
Nicola Dawson, Kathleen Rastle, and Jessie Ricketts. 2021. Finding the man amongst many: A develop- mental perspective on mechanisms of morphological decomposition. Cognition, 21 211:104605.
John Aldrich. 1995. Correlations genuine and spurious pearson and yule. Statistical science, pages 364- 376.
Ludovic Ferrand, Boris New, Marc Brysbaert, Em- manuel Keuleers, Patrick Bonin, Alain Méot, Maria Augustinova, and Christophe Pallier. 2010. The french lexicon project: Lexical decision data for 38.840 french words and 38,840 pseudowords. Be- havior research methods, 42:488-496,
Simona Amenta and Davide Crepaldi. 2012. Morpho- logical processing as we know An analytical re- view of morphological effects in visual word identifi- cation. Frontiers in psychology, 3:232.
Khuyagbaatar Batsuren, Gábor Bella, Aryaman Arora, Viktor Martinovic, Kyle Gorman, Zdenčk Zabokrt- sky, Amarsanaa Ganbold, Šárka Dohnalová, Magda Ševěíková, Katerina Pelegrinová, Fausto Giunchiglia, Ryan Cotterell, and Ekaterina Vylomova. 2022. The SIGMORPHON 2022 shared task on morpheme seg- mentation. In Proceedings of the 19th SIGMOR- PHON Workshop on Compulational Research in Pho- netics, Phonology, and Morphology, pages 103-116, Seattle, Washington. Association for Computational Linguistics.
Philip Gage. 1994. A new algorithm for data compres- sion. C Users Journal, 12(2):23-38.
Matthias Gallé. 2019. Investigating the effectiveness of BPE: The power of shorter sequences. In Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Inter- national Joini Conference on Natural Language Pro- cessing (EMNLLP-IJCN) Kong, China. Association for Computational Linguis- tics.
pages 1375-1381, Hong
Blisabeth Beyersmann, Petroula Mousikou, Ludivine Javourey- -Drevet, Sascha Schroeder. Johannes C Ziegler, and Jonathan Grainger. 2020. Morphological processing across modalities and languages. Scien- rific Studies of Reading, 2406)1500-519
Valentin Hofmann. Janet Pierrehumbert, and Hinrich Schütze. 2021. Superbizarre is not superb: Deriva- tional morphology improves BERT's s interpretation of complex words. In Proceedings of the 59th Annual Meeting of the Association for Computationl Lin- guistics and the 11th International Joint Conference Natural Language Processing (Volume 1: Long Papers), pages 3594-3608, Online, Association for Computational Linguistics.
Chris Biemann, Gerhard Heyer, Uwe Quasthoff, and Matthias Richter. 2007. The leipzig corpora Dollection-monolinge corpora of standard size. Pro- ceedings of Corpus Linguistic, 2007.
Valentin Hofmann, Hinrich Schuetze, and Janet Pierre- humbert. 2022. An embarrassingly simple method to mitigate undesirable properties of pretrained lan- guage model tokenizers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume 2: Short Papers), pages 385-393, Dublin, Ireland. Association for Computational Lin- guistics.
Kaj Bostrom and Greg Durrett. 2020. Byte pair encod-
ing is suboptimal for language model pretraining. In Findings of the Association for Computational Lin- guisties: EMNLP 2020, pages 4617-4624, Online. Association for Computational Linguistics.
Hyung Won Chung, Dan Garrette, Kiat Chuan Tan, and Jason Riesa. 2020. Improving multilingual models with language-clusteree vocabularies. In Proceed- ings the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pages 4536-4546, Online. Association for Computational Linguistics.
Emmanuel Keuleers, Kevin Diependaele, and Mare Bryshaert. 2010. Practice effects in large-scale visual word recognition studies: A lexical decision study on 14,000 dutch mono- and disyllabic words and nonwords. Frontiers in Psychology. 1.
Jonathan H. Clark, Dan Garrette, Iulia Turc, and John Wieting. 2022. Canine: Pre-training efficient tokenization-frre encoder for language representa- tion. Transactions of the Association for Computa- tional Linguistics, 10:73-91.
Emmanuel Keuleers, Paula Lacey, Kathleen Rastle, and Mare Bryshaert. 2012. The british lexicon project: Lexical decision data for 28,730 monosyllabic and disyllabic english words. Behavior research methods. 44:287-304.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised
Evan Kidd, Seamus Donnelly, and Morten H Chris- tiansen. 2018. Individual differences in language acquisition and processing. Trends in cognitive sci- ences, 2242):154-169.

---

Stav Klein and Reut Tsarfaty. 2020. Getting the ##life out of living: How adequate are word piddep for mod- elling complex morphology? In Proceedings of the 17th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 204-209, Online. Association for Computa- tional Linguisties.
and Morphology, pages 131-138, Seattle, Washing- ton. Association for Computational Linguistics.
Barbara Plank, Dirk Hovy, and Anders Søgaard. 2014. Linguistically debatable or just plain wrong? Proceedings of the 52nd Annual Meeting of the As- sociation for Computational Linguistics (Volume 2: Short Papers), pages 507-511, Baltimore, Maryland. Association for Computational Linguistics.
Taku Kudo. 2018. Subword regularization: Improv- ing neural network translation models with multiple subword candidates. In Proceedings of the 56th An- nual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pages 66-75, Melbourne, Australia. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever. et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9.
Phillip Rust, Jonas F. Lotz, Emanuele Bugliarello, Eliz- abeth Salesky, Miryam de Lhoneux, and Desmond Elliott. 2023, Language modelling with pixels. In The Eleventh International Conference on Learning Representations.
Davis Liang, Hila Gonen, Yuning Mao, Rui Hou, Na- man Goyal, Marjan Ghazvininejad, Luke Zettle- moyer, and Madian Khabsa. 2023. Xlm-v: Overcom- ing the vocabulary bottleneck in multilingual masked language models. arXiv preprint 2311:23311.1472
Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. 2021. How good is your tok- enizer? on the monolingual performance of multilin- gual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguisties and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 3118-3135 Online. Association for Computational Linguistics.
Zachary C Lipton and Jacob Steinhardt. 2019. Trou- bling trends in machine learning scholarship: Some ml papers suffer from flaws that could mislead the public and stymie future research. Queue, 17(1):45- 77.
Manuel Mager, Arturo Oncevay, Elisabeth Mager, Katharina Kann, and Thang Vu. 2022. BPE vs. mor- phological segmentation: A case study on machine translation of four polysynthetic languages. In Find- ings of the Association for Computational Linguistics: ACL 2022, pages 961 -971, Dublin, Ireland. Associa- tion for Computational Linguistics.
Mike Schuster and Kaisuke Nakajima. 2012. Japanese and korean voice search. In 2012 IEEE international conference on acoustics, speech and signal process- ing (ICASSP), pages 5149-5152. IEEE
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 1715-1725, Berlin, Germany. Association for Computational Lin- guistics.
Antonis Maronikolakis, Philipp Dufter, and Hinrich Schützee 2021. Wine is not v n. on the compatibility of tokenizations across languages. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2382-2399, Punta Cana, Dominican Re- public. Association for Computational Linguisties.
Patience Stevens and David C Plaut. 2022. From de- composition to distributed theories of morphological processing in reading. Psychonomic Bulletin & Re- view, 1773
Sabrina J. Mielke, 7aid Alyafeai, Elizabeth Salesky, Colin Raffel, Manan Dey, Matthias Gallé, Arun Raja, Chenglei Si, Wilson Y. Lee, Benoît Sagot, and Sam- son Tan. 2021. Between words and characters: A brief history of open- modeling and tok- enization in nlp. ArXiv, 12..
Linting Xue, Aditya Barua, Noah Constant, Rami Al- Rfou, Sharan Narang, Mihir Kale, Adam Roberts. and Colin Raffel. 2022. Byt5: Towards a token-free future with pre-trained byte-to-byte models. Transac- tions of the Association for Computational Linguis- tics, 10:291-306
Hyunji Hayley Park, Katherine J. Zhang, Coleman Ha- ley, Kenneth Steimel, Han Liu, and Lane Schwartz. 2021. Morphology matters: A multilingual language modeling analysis. Transactions of the Association for Computational Linguistics, 9:261-276.
Jinbiao Yang, Antal van den Bosch, and Stefan L. Frank. 2022. Unsupervised text segmentation predicts eye fixations during reading. Frontiers in Artificial Intel- ligence, 5.
Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent disagreements in human textual inferences. Transac- tions of the Association for Computational Linguis- tics, 7:677-694.
Shaked Yehezkel and Yuval Pinter. 2023. Incorporating context into subword vocabularies. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 623-635, Dubrovnik, Croatia. Association for Com- putational Linguistics.
Ben Peters and Andre F. T. Martins. 2022. Beyond char- acters: Subword-leve' morpheme segmentation. In Proceedings of the 19th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology,

---

A Language Codes
frequency obtained similar low mean squared ror (0.00-0.06) on the test data for both response time and accuracy. However, all explained vari- ance scores are negative (with systematically higher scores when predicting with frequency and partic- ularly low scores for French and Dutch response times). We assume that the low explained variance is related to the finegrained cognitive signal and to individual differences in the responses. Prediction would probably be easier when predicting classes (e.g.. high/low) instead of absolute values. We are interested in diving deeper into these pilot analyses in cooperation with cognitive scientists. Finally, we note that while frequency is only available for true words, chunkability can be proxy for frequency effects in non-words as well.
The abbreviations used in Figure 4 correspond ISO 639-3 codes and are mapped as fol- lows: swe = Swedish, deu= German, eng= En- glish, cat= Catalan, spa= Spanish, fra= French. ita= Italian, por= Portuguese, hbs= Serbo-Croatian, ces= Czech, rus= Russian, pol= Polish, hun= Hun- garian, fin= Finnish, mon= Mongolian.
B Number of Splits
In the chunkability metric, we normalize by the length of the sequence. For comparison, also ran experiments by simply using the number of splits as metric. The tendencies in the results in Figure 5 are comparable for the two metrics except for the correlations with the response time for non- words which are substantially less consistent with cognitive phenomena for the oumber-of-spling mea- sure. We assume that this can be explained through the finer range of values afforded by chunkability.
MSE
EV
Lang
Signal
CHUNK
FREQ
CHUNK
FREQ
RT
.01
.01
-14.73
-0.54
eng
Ace
.06
.05
-3.73
-0.91
RT
.03
.03
-38.28
-2.51
nld
Acc
.04
.04
-5.01
-2.33
Num_Splits
RT
.02
.01
-124.78
-2.28
fra
Acc
.02
.02
-6.96
-4.24
04
Cot. Actunoc
02
spa
Acc
RT
.06
.01
.00
.05
-18.18
-2.88
-0.03
-0.56
Table 3: Linear regression results as mean squared error (MSE) and explained variance (EV) for the words in our dataset with a random 80/20 train-test split. We com- pare the features chunkability (CHUNK) and word frequency (FREQ). Frequency is determined using Zipf frequency scores obtained from the wordfreq package v3.03 (https://pypi org/ wwtggprojec/////eeee (ebecc//oorfrre))) and chunkability is determined using the WordPiece tok- enizer. We predict the two signals response time (RT) and accuracy (Acc) separately using the standard lin- ear regression model from the sklearn package v1.3 (https://scikit t- t-learn. are normalized using min-max scaling.
Diglish
Duich
frerch
Spunssh
English
Dutch
hhocch
Spariahi
04
ore
03
Roading
00
-22
-0.1
Figure 5: Pearson correlation between the number of splits a tokenizer assigns to sequence and the observed accuracy (top) and response times (bottom) in human responses to lexical decision task. All tokenizers are trained on the same data (per language) with a vocabu- lary size 50,000. Note that the correlation directions are reversed compared to Figure 1, as we did not take the inverse.
Both signals
D Four-lingual Vocabulary
In order to better control the effect of vocabulary sharing, also trained tokenizers all the En- glish, Dutch, French, and Spanish training data jointly. Figure 6 illustrates the results for the Word- Piece tokenizer and shows that the correlation is lower for multilingual models but improves when increasing the vocabulary.
C Regression Analysis
In this paper, focus the evaluation of sub- word tokenizers and on predicting cognitive phenomena. However, one of the reviewers in- spired us to run demonstrative linear regression analysis for the words in the dataset. For all four languages, we found that both chunkability and

---

Words
Non Words
0.4
Corr. Accuracy
0.0
English
Dutch
French
Spanish
English
Dutch
French
Spanish
0.4
Corr. Reading Time
0.2
0.0
-0.2
mono 50K nutti S0k mumti 70k
Figure 6: Pearson correlation between the chunkability of sequence and the observed response times and accuracy in human responses to a lexical decision task. To determine the chunkability, we trained the WordPiece (WPC) tokenizer monolingual and four-lingual training data with vocabulary sizes of 50k and 70k.


Reviews:
[{"review_id": "baX5Apdf7U", "reviewer": "Reviewer_ye9p", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your comments, and agree that our evaluation setup can inform further work examining the relationship between segmentation and lexical access.\n\nTo the best of our knowledge, our methodology and metric are novel. If you could point us to prior work evaluating subword tokenizers through cognitive task performance, we would greatly appreciate it.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper is clear and well-written. The cognitive perspective introduced in the task is interesting and provides new lights on the subword tokenisation mechanism. This technique is of course efficient, but nothing indicates that it is used by the human parser. These results could argue in favor of the idea that input segmentation is based on low-level techniques, delaying (or even avoiding in some cases) a lexical access to the mental lexicon.", "reasons_to_reject": "The methodology is not original and the confirm known results.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper adresses  the question of subword tokenization in the perspective of its cognitive plausibility. It is based on the study of the classical subword tokenization techniques (Byte-pair encoding, WordPiece and Unigram) and the correlation of their outputs with the reading time. The hypothesis is that morphologically structured non-words cause longer reading times and lead to decreased accuracy in word detection. The proposed metric is based on chunkability (the inverse of the ratio of tokens over characters) with the idea that  chunkability level can be correlated with processing difficulty: higher chunkability corresponds to an easier process, higher accuracy and lower reading times. Results show a better prediction with BPE and WPC.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "6zOb7bJeI9", "reviewer": "Reviewer_1agx", "summary": "", "questions": "Sometimes you write “reading time” referring to “reaction time”:\nare you sure that the two cognitive processes may be considered as overlapping or as the same?\n\nAt the beginning of the Results section, indirectly, you define “chunkability” as a measure of “wordlikeness”: are you sure of that? Do you think that the two properties can be considered at the same level, invoking the same cognitive processes?\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your constructive comments.\n\n> Sometimes you write “reading time” referring to “reaction time”: are you sure that the two cognitive processes may be considered as overlapping or as the same?\n\nAnswer:  We used the term “reading time” in the text because it was used by the creators of the datasets. We indeed don’t know how much time the participants spend on reading which is why we added the clarification in parentheses in lines 194ff: “The datasets contain information about the average reading time (i.e., the number of milliseconds it took the participants to make a decision) and accuracy for each stimulus.” In order to make this point more clear, we will update the text to consistently use the term “response time” instead.\n\n> At the beginning of the Results section, indirectly, you define “chunkability” as a measure of “wordlikeness”: are you sure of that? Do you think that the two properties can be considered at the same level, invoking the same cognitive processes?\n\nAnswer: Thank you for pointing out our slightly imprecise language. Our analyses are meant to find a relationship between the chunkability assigned by the subword tokenizer and the human reactions. Based on the finding that higher chunkability correlates with longer reading times and lower accuracy for non-words, we assume that high chunkability indicates that participants are more likely to consider the sequence to be a word, but further research is necessary to identify whether this assumption holds. We will phrase our findings more carefully.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The research rationale is well defined and rises an important question: how cognitive plausible are tokenizer algorithm results?\nResults suggest a new evaluation paradigm that deserves an open discussion with the conference community ", "reasons_to_reject": "No reason to reject it", "questions_to_authors": "Sometimes you write “reading time” referring to “reaction time”:\nare you sure that the two cognitive processes may be considered as overlapping or as the same?\n\nAt the beginning of the Results section, indirectly, you define “chunkability” as a measure of “wordlikeness”: are you sure of that? Do you think that the two properties can be considered at the same level, invoking the same cognitive processes?\n", "paper_topic_and_main_contributions": "The short paper focuses on subword tokenization algorithms by comparing their performance with human lexical decision results, in a few languages (two germanic and two romance ones, i.e. German, English and Spanish, French).", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "To the best pf my knowledge, all the most relevant papers are cited.", "Typos_Grammar_Style_and_Presentation_Improvements": "My suggestion is to change those issues related to my questions to the author(s)."}, "extra_text": {}}, {"review_id": "P7rnDFOx9z", "reviewer": "Reviewer_6Wn8", "summary": "", "questions": "As presented, it seems that the chunkability metric is affected by word length in ways that may not be desirable: does a 5-character word with one token really have a lower chunkability than a 10 character one with one token? Is this desirable, and if so, why? I wonder if the authors tried similar metrics that do not factor in length as explicitly? I'd like to see a couple of sentences discussing this.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your constructive comments. \n\n> In correlating with human metrics, the authors don't control (to my knowledge) for potential confounds such as frequency, which is known to affect lexical decision time. A regression analysis including such metrics would be more compelling; I would rate this paper significantly higher were one given and I think this is the most important point for revisions if the paper is accepted.\n\nAnswer:\nThe original papers that present the lexical decision datasets provide analyses related to frequency. However, these analyses are limited to words, as non-words cannot be found in corpora. For non-words, only the frequency of smaller parts, e.g. character bigrams, can be measured. By design, the subword tokenizers considered in our experiments do just this: they **determine splits based on the frequencies of character sequences**. Using subword tokenizers is a more elegant way of accounting for non-word frequencies because the length of the character sequence under consideration does not need to be hard-coded but is determined by the sequences seen in the training data. \n\nIn this paper, we focus on the evaluation of subword tokenizers and not on predicting cognitive phenomena. However, out of curiosity, we ran a small linear regression analysis comparing the predictive power of chunkability scores of the WordPiece tokenizer to Zipf frequency scores obtained from the wordfreq library for the words in the dataset. We split the data into 80% train and 20% test. For all four languages, we found that both chunkability and frequency obtain similar low mean squared error (between 0 and 0.06) on the test data for both reading time and accuracy. Overall, frequency yields better explained variance probably because it is a more finegrained signal. We are interested in diving deeper into these pilot analyses in cooperation with cognitive scientists. \n\n> In Figures 1 and 2, I'd like to see the y-axes rescaled either to -1 to 1 or -0.5 to 0.5 for a more consistent reading of the results; more generally, I'd like to see the authors acknowledge more explicitly that none of the correlations reported are particularly high (which is really fine to acknowledge in this setting, since their contribution is the evaluation metric).\n\nAnswer: \nThank you for pointing out the slight inconsistency in the negative limit of the y-axis scales of Figures 1 and 2. We will adjust this in the next version. Due to our large sample sizes, all reported correlations are significant (p<=0.01). Indeed, our work focuses more on relative differences between tokenizers. We report on significance of pair-wise comparisons below. \n\n> I'd also like to see significance testing where the authors report differences in correlation between models.\n\nAnswer:\nWe agree that significance calculations are a valuable addition to the paper. \n\nFor the results in Figure 1, all comparisons to the length baseline are significant (p<=0.01). The comparisons of the unigram tokenizer to BPE and WPC are significant in 22 out of 32 conditions. The differences between the BPE and the WPC tokenizer are not significant in most comparisons. \n\nIn Figure 2, 22 out of the 24 comparisons are significant. The two insignificant differences correspond to comparisons between the two multilingual tokenizers. \n\nThese observations support the conclusions presented in the paper and we will add them to the next version. \n\n> The authors mention the shortcomings of Pearson's correlation in their limitations section, but I'd like to see another statistic presented because of this -- something like Goodman-Kruskall might be promising to report.\n\nAnswer: We understand your reservation and will consider alternative metrics. Due to the large number of participants in the cognitive benchmarks, and the well-known effects of individual variability, we are not sure a rank-based correlation metric such as Goodman-Kruskall would be suitable for comparing against a comparatively coarse-grained chunkability score.\n\n> I'd also like to see some discussion of the segmentations in Table 1: the proposed tokens for e.g. outfoxed do not correspond to the constituent morphemes of this word, and I'd like to see some discussion of this mismatch. If the subword tokenizer is often finding tokens that don't directly correspond to morphemes, then I think the cognitive plausibility claims are probably weaker. Again, though, this does not detract from the main aim of the paper, which is to provide a novel evaluation procedure.\n\nAnswer: This is a very relevant observation. Recent related work indeed found that subword splits do not always align with morpheme boundaries especially when the vocabulary size is small (e.g., [1], see also Figure 4 in our paper). It has been speculated that the subword tokenization algorithms are overfitted to English and are less useful for morphologically richer languages [2--3]. The relationship between morphological plausibility, performance, and cognitive plausibility remains an open research question and we think that our proposed methodology provides a useful tool for deeper analyses.\n\n[1] Valentin Hofmann, Janet Pierrehumbert, and Hinrich Schütze. ACL 2021. Superbizarre Is Not Superb: Derivational Morphology Improves BERT’s Interpretation of Complex Words\n\n[2] Stav Klein and Reut Tsarfaty. SIGMORPHON 2020. Getting the ##life out of living: How Adequate Are Word-Pieces for Modelling Complex Morphology?\n\n[3] Manuel Mager, Arturo Oncevay, Elisabeth Mager, Katharina Kann, and Thang Vu. ACL-findings 2022. BPE vs. Morphological Segmentation: A Case Study on Machine Translation of Four Polysynthetic Languages.\n\n> As presented, it seems that the chunkability metric is affected by word length in ways that may not be desirable: does a 5-character word with one token really have a lower chunkability than a 10 character one with one token? Is this desirable, and if so, why? I wonder if the authors tried similar metrics that do not factor in length as explicitly? I'd like to see a couple of sentences discussing this.\n\nAnswer: This is a very good question which we also discussed when designing the experimental setup. We ran comparative analyses using only the number of splits without controlling for length but had not added them to the paper due to space limitations. We will add these figures to the appendix for comparison and use the extra page for additional explanations. The tendencies in the results are comparable for the two metrics except for the correlations with the reading time for non-words which are substantially less consistent with cognitive phenomena for the number-of-splits measure. This can be explained through the finer range of values afforded by chunkability. When comparing a 5-character word with a 10-character one, we indeed believe that the latter should present slightly higher chunkability as it is both more challenging for a tokenizer to incorporate it as a single token in its vocabulary and less likely a-priori to be a valid word.\n\n> Typos Grammar Style And Presentation Improvements: I think the presentation of the paper could be improved by presenting chunkability earlier (i.e. before the start of S3, in its own section). In my view, this metric is the main contribution of the paper, and should thus be presented before the details of the experimental setup.\n\nAnswer: The introduction of the new metric at the end of the experimental setup is indeed not ideal. Adding a section before S3 seems like a promising alternative when given an additional page and we will consider it.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I see this as an impactful short paper: it proposes a fundamental rethinking of how we analyze and evaluate subword tokenizers. I think the push to evaluate tokenizers in terms of cognitive plausibility is the most important contribution of the paper, above and beyond the specific method proposed, though I also see that method as being promising.", "reasons_to_reject": "I'd like to see a more thorough analysis of the results along the following dimensions: \n* In correlating with human metrics, the authors don't control (to my knowledge) for potential confounds such as frequency, which is known to affect lexical decision time. A regression analysis including such metrics would be more compelling; I would rate this paper significantly higher were one given and I think this is the most important point for revisions if the paper is accepted. \n* In Figures 1 and 2, I'd like to see the y-axes rescaled either to -1 to 1 or -0.5 to 0.5 for a more consistent reading of the results; more generally, I'd like to see the authors acknowledge more explicitly that none of the correlations reported are particularly high (which is really fine to acknowledge in this setting, since their contribution is the evaluation metric). I'd also like to see significance testing where the authors report differences in correlation between models. \n* The authors mention the shortcomings of Pearson's correlation in their limitations section, but I'd like to see another statistic presented because of this -- something like Goodman-Kruskall might be promising to report. \n\nI think all of these can be easily addressed in the additional page, and aside from the first point, they do not detract from my overall excitement about the proposal. \n\nI'd also like to see some discussion of the segmentations in Table 1: the proposed tokens for e.g. outfoxed to not correspond to the constituent morphemes of this word, and I'd like to see some discussion of this mismatch. If the subword tokenizer is often finding tokens that don't directly correspond to morphemes, then I think the cognitive plausibility claims are probably weaker. Again, though, this does not detract from the main aim of the paper, which is to provide a novel evaluation procedure.", "questions_to_authors": "As presented, it seems that the chunkability metric is affected by word length in ways that may not be desirable: does a 5-character word with one token really have a lower chunkability than a 10 character one with one token? Is this desirable, and if so, why? I wonder if the authors tried similar metrics that do not factor in length as explicitly? I'd like to see a couple of sentences discussing this.", "paper_topic_and_main_contributions": "This paper presents \"chunkability\" as a new evaluation paradigm for subword tokenization. This evaluation metric is meant to evaluate the cognitive plausibility of subword tokenizers as compared to humans' lexical decision times and accuracy, and correlations of these measures are presented for several popular subword tokenization algorithms.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "I think the presentation of the paper could be improved by presenting chunkability earlier (i.e. before the start of S3, in its own section). In my view, this metric is the main contribution of the paper, and should thus be presented before the details of the experimental setup."}}]

Author rebuttal:

Meta-review: The content and results are clear and a great fit for this track. The focus on cognitive aspects is particularly relevant.
Decision: Accept-Main