Greed is All You Need: An Evaluation of Tokenizer Inference Methods (2024)

Abstract: While subword tokenizers such as BPE and WordPiece are typically used to build vocabularies for NLP models, the method of decoding text into a sequence of tokens from these vocabularies is often left unspecified, or ill-suited to the method in which they were constructed. We provide a controlled analysis of seven tokenizer inference methods across four different algorithms and three vocabulary sizes, performed on a novel intrinsic evaluation suite we curated for English, combining measures rooted in morphology, cognition, and information theory. We show that for the most commonly used tokenizers, greedy inference performs surprisingly well; and that SaGe, a recently-introduced contextually-informed tokenizer, outperforms all others on morphological alignment.

heading: 1 Introduction
text: Modern NLP systems, including large language models (LLMs), typically involve an initial step of mapping raw input text into sequences of subword tokens. These tokens are selected from a large vocabulary of candidates that were produced from algorithms such as Byte-Pair Encoding (BPE; Sennrich et al., 2016), WordPiece (Schuster and Nakajima, 2012), or UnigramLM (Kudo, 2018).
This process, which we refer to as the inference method of tokenization, is critical as it determines how all text is represented and subsequently modeled. Each inference method offers distinct mappings, and we assert that it is not well-understood how these methods differ in performance. Furthermore, popular implementation packages such as Huggingface Tokenizers,1 SentencePiece,2 and SubwordNMT3 often obfuscate or even restrict the choice of inference methods, making it unclear if
1 https://huggingface.co/docs/tokenizers 2 https://pypi.org/project/sentencepiece 3 https://github.com/rsennrich/subword-nmt
inference-time decoding is compatible with the algorithm used to learn the tokenizer’s vocabulary. Moreover, it is yet to be determined whether such a match is ideal, or even necessary.
In Table 1 we present examples demonstrating how the prescribed inference methods of BPE, UnigramLM, and SaGe (Yehezkel and Pinter, 2023) do not necessarily provide the best segmentation for complex English words, even when good segments are available in the vocabulary. BPE’s outof-the-box algorithm merges the cross-morphemic am sequence at an early stage, preventing the consideration of ultra and modern and condemning the downstream model to work with a representation learned for the first-person present form of ‘to be’. UnigramLM’s ablative algorithm enabled nprecedented (which crosses morpheme boundaries) to remain in its final vocabulary of tokens, while SaGe’s greedy algorithm masks the boundaries of both the prefix In and the suffix able. In all cases, an alternative inference method provides a more morphologically-aligned segmentation over the same vocabulary.
Previous work regarding subword tokenization mostly concerns developing vocabulary construction algorithms (Sennrich et al., 2016; Schuster and Nakajima, 2012; Kudo, 2018; Mielke et al., 2021; Yehezkel and Pinter, 2023), finding the optimal
813
vocabulary size (Gowda and May, 2020; GutierrezVasques et al., 2021), building multilingual vocabularies (Liang et al., 2023), and using space positioning in the vocabulary tokens (Gow-Smith et al., 2022; Jacobs and Pinter, 2022). Others analyze the effects of vocabularies, finding intricate relations between algorithm or vocabulary and downstream performance (Bostrom and Durrett, 2020; Cognetta et al., 2024a), information theory (Zouhar et al., 2023; Cognetta et al., 2024b), cognitive plausibility (Beinborn and Pinter, 2023), impact on society (Ovalle et al., 2024), or morphological alignment (Klein and Tsarfaty, 2020; Hofmann et al., 2021, 2022; Gow-Smith et al., 2024; Batsuren et al., 2024).
Research concerning inference methods has been more scarce, and includes examination of random effects on BPE merges (Provilkov et al., 2020; Saleva and Lignos, 2023) and application of sophisticated search algorithms (He et al., 2020). As far as we know, there exists no comprehensive study comparing inference methods across a variety of vocabularies and sizes using diverse metrics.
In this work, we conduct a controlled experiment isolating the effects of inference methods over four tokenizers, introducing an evaluation suite aggregating intrinsic benchmarks from various theoretical realms.4 We find that greedy inference methods work surprisingly well for all four vocabularies across morphological and information-theoretic metrics. Furthermore, we demonstrate that SaGe yields state-of-the-art performance according to morphological metrics, and that inference methods that minimize token count perform strongest by cognitive metrics.

heading: 2 Inference Methods
text: Let V denote a vocabulary of subword tokens and w denote a word (or ‘pretoken’), the output of a pretokenizer. We define s(V, w) := (t1, ..., tk) as a segmentation of w into k subword tokens such that ∀i, ti ∈ V and that the concatenation of t1, ..., tk results in w. We use the term segmentation to denote the application of an inference method on a text given a token vocabulary, as well as its result.
Current widely-employed tokenization schedules couple together the tokenizer vocabulary with the inference method. However, we advocate for decoupling them, as they are independent pro-
4We release our code and data at https://github.com/ MeLeLBGU/tokenizers_intrinsic_benchmark.
cesses. Specifically, given a fixed token vocabulary produced from pre-training data, one could subsequently use any applicable inference method for the task at hand. Thus, in our experiments, we use various intrinsic metrics to analyze the impact and performance of the several classes of inference methods:
Greedy inference methods only consider and produce one token at each step. We test three greedy approaches: Longest prefix, which WordPiece uses by default (Wu et al., 2016), selects the longest token in V that is a prefix of w, and then continues to iteratively segment the remaining text. Longest suffix selects the longest token that is a suffix of w and continues iteratively (Jacobs and Pinter, 2022; Bauwens, 2023). Since this strategy diverges from English Morphology, we consider it an intriguing baseline for assessing the impact of linguistic structure on the inference method. Longest token selects the longest token that is contained in w, adds it to the generated segmentation, and then iteratively segments each remaining character sequence. This was proposed by Hofmann et al. (2022) to approximate words by their k longest tokens. They showed that it preserves morphological structure of words and leads to performance gains on some downstream tasks.
Merge rules-based inference methods begin with a word’s character sequence and iteratively apply token-forming merge rules learnt by the tokenizer at the vocabulary creation phase, until none can be applied. This is BPE’s default inference mode.5 In our experiments we test two variants for BPE: The deterministic merge strategy recursively applies the first applicable BPE merge rule by its order in the trained merge list. Dropout (Provilkov et al., 2020) applies each valid merge rule with probability p, leading to a regularization effect where rare tokens surface more often and their embeddings can be better trained. It has been shown to improve machine translation performance.
Likelihood-based inference methods use individual likelihood values assigned to tokens in order to find a segmentation for w where the total likelihood is maximized (Kudo, 2018; He et al., 2020). Default uses likelihood values learned during vocabulary construction and considers the likelihood
5While ostensibly also compatible with WordPiece, we found no implementation of the model that provides an ordered list of its merges.
of a segmentation to be the product of individual likelihoods (from which UnigramLM gets its name). Least tokens assigns a constant likelihood value to all tokens, effectively selecting a segmentation where the number of tokens is minimized. While not suggested so far as a standalone inference method, this obecjtive is proposed for both vocabulary training and inference in the PathPiece algorithm (Schmidt et al., 2024).

heading: 3 Intrinsic Benchmark
text: Some analyses of tokenizers rely on training language models or translation models and evaluating their performance on downstream tasks. Using this process to isolate effects of tokenization hyperparameters, such as inference method, is both time- and resource-consuming, as well as unstable due to the introduction of multiple sources of randomness throughout the LM/TM pre-training and fine-tuning phases. Few measures have been introduced that are intrinsic to vocabularies and their direct application to corpora, and fewer still avoid conflating the measures with the objectives used in the vocabulary construction process itself. As a result, the body of work focused on improving tokenization schemes is still relatively small.
We create and release a benchmark made to intrinsically evaluate subword tokenizers. We collected word-level datasets and information measures which have been shown, or hypothesized, to correlate with the performance of language models on various downstream tasks. Details on these resources are provided in Table 2. At present, the benchmark is focused on the English language, although corresponding datasets exist for others as well.
Morphological alignment It is commonly assumed that, for a given tokenizer, alignment of word segments to morphological gold-standard segmentations is a predictor of the ability of a language
model that uses the given tokenizer to represent words, especially ‘complex’ ones that are made up of several roots or contain multiple morphological affixes (Schick and Schütze, 2019; Nayak et al., 2020; Hofmann et al., 2021; Gow-Smith et al., 2022). We follow Gow-Smith et al. (2022) and evaluate our tokenizers’s alignment with morphological annotations found in LADEC (Gagné et al., 2019), MorphoLex (Sánchez-Gutiérrez et al., 2018), MorphyNet (Batsuren et al., 2021), and DagoBert (Hofmann et al., 2020). We augment these datasets with morpheme segmentation data (Batsuren et al., 2022), novel blend structure detection data (Pinter et al., 2020), and compound separation data (Minixhofer et al., 2023). The number of words in each resource can be found in Table 2. We compare the segmentations generated by the tokenizers with each inference method to gold-standard morphological segmentations using the metric introduced by Creutz and Linden (2004), and report the macroaveraged F1 score over the different resources.
Cognitive Plausibility We use the benchmark and data from Beinborn and Pinter (2023) to measure the correlation of a tokenizer’s output with the response time and accuracy of human participants in a lexical decision task, predicated on the hypothesis that a good tokenizer struggles with character sequences that humans find difficult, and vice versa. We report the average of the absolute value correlation scores across the four linguistic setups (word/nonword × accuracy/response time).
Tokens distribution statistics We report the Rényi efficiency of different segmentations across a corpus (Zouhar et al., 2023). This measure penalizes token distributions dominated by either very high- and/or very low-frequency tokens, and was shown to correlate strongly with BLEU scores for machine translation systems trained on the respective tokenizers. Recent work (Cognetta et al.,
Vocab Inference Morphological Cognitive Rényi Tokens Decoding method alignment plausibility efficiency per word diff
2024b) reveals a misalignment between Rényi efficiency and downstream performance in certain cases, reinforcing the necessity of an evaluation suite grounded in diverse domains and disciplines, as advocated in this work. We also measure the average number of tokens per word over a corpus, as a proxy for compression quality (Gallé, 2019). We omit the popular measure of character-length distribution of the tokens in the vocabulary, as it does not vary with segmentation strategy.
Lastly, we report the proportion of pretokens that are segmented different from the default across our reference corpus.

heading: 4 Experiments
text: We evaluate inference methods for the following tokenizer vocabularies: BPE, UnigramLM, WordPiece and SaGe. We use the train split of the MiniPile (Kaddour, 2023) dataset to construct the tokenizer vocabularies. We train vocabularies of sizes 32,768, 40,960, and 49,152, using the HuggingFace Tokenizers library, with identical pre-tokenization, representing the text at byte level. UnigramLM and SaGe require an initial vocabulary for their top-down algorithms; for the former, we used the default implementation of one million top n-grams,
while SaGe was initialized with a 262K-size UnigramLM vocabulary. This initial vocabulary also provided us with token likelihood scores for inference, although a more exact implementation would also incorporate the contextual SaGe objective.
Token distribution statistics measurements and decoding diff rates were computed over the test split of the MiniPile dataset. We measure the Rényi efficiency using the tokenization-scorer package6 with α = 2.5. For each tokenizer, all experiments ran within several minutes on a personal laptop computer, highlighting the usefulness of our benchmark as an efficient tool for in-loop hyperparamter tuning.
We present the results on our benchmark for the 40K vocabularies in Table 3. Results for other sizes are presented in Appendix A. A breakdown of individual evaluation subsets is provided in Appendix B.
Inference methods Within each tokenizer, we find that the default (‘intended’) strategy is often outperformed by others on some measures. We observe a significant difference in morphological alignment when using merge rules-based inference methods. Qualitative analysis showed the findings
6 https://github.com/zouharvi/tokenization-scorer
illustrated in Table 1, where early merge rules such as ‘i-n’, ‘a-m’, or ‘o-n’ cross morphological boundaries. We notice a similar trend for likelihoodbased inference, where frequently-used tokens possess very high likelihood values, sometimes exceeding those of the gold-standard segments. We find that the least tokens strategy fares well not only on the token count metric, which is mostly by-design, but also on cognitive measures, suggesting an effect of human preference to minimal word segmentation. Finally, we observe that likelihood-based inference performs poorly in terms of Rényi efficieny, contrary to its stated purpose. Dropout, on the other hand, performs well on this measure, in line with its goal. longest suffix performs poorly across the board, possibly due to the suffixing nature of the English language, which has complementarily been shown to affect character-level sequential modeling (Pinter et al., 2019). Notably, all our key observations are consistent across vocabulary sizes, as shown in Appendix A.
Inter-tokenizer results Our results align with Bostrom and Durrett (2020)’s finding that BPE is inferior to UnigramLM on morphology alignment. However, we show that some of this gap can be attributed not to the vocabulary but to the inference method. In addition, we find that SaGe is most aligned to morphology by a substantial margin, indicating that its contextualized objective succeeds in retaining meaningful tokens in the vocabulary during ablation. It is important to note that our evaluation is limited to English, a language with relatively low morphological complexity. Previous studies have identified significant tokenization challenges in non-English languages (Mager et al., 2022). Therefore, any definitive conclusions regarding the effectiveness of tokenization methods should ideally encompass a diverse array of languages. BPE and WordPiece, optimized for compression, unsurprisingly perform well above the likelihood-based vocabularies on the information measures. However, we note that this carries over to the cognitive benchmark as well, supporting Beinborn and Pinter (2023)’s findings.
Finally, we note that the two likelihood-based vocabularies follow the exact same within-vocab trends, and those for the two information-based vocabularies are also very close. This highlights the consistency and robustness of our benchmark, although some results are relatively close to each other, which can be expected considering that some
inference methods do not change much of the token sequences (see rightmost column of Table 3).

heading: 5 Conclusion
text: In this work, we curated an aggregated benchmark for intrinsic evaluation of subword tokenizers and used it to show the importance of selecting an inference method suited for a vocabulary given a task. Given its computational efficiency, we hope the benchmark can be used in LM training efforts as a fruitful first step to improve tokenization schemes, or to select inference methods on-line. Concretely, our findings suggest that greedy inference is a good choice, especially for morphologically-motivated tasks, even for tokenizers trained on other objectives. Considering its ease of implementation and faster inference, this is an encouraging finding.
In the future, we plan to examine the correlation between our benchmark and various downstream tasks, as well as expand our experimentation to other languages and new algorithms.
Limitations
Our paper contains evaluation of models in the English language. This was done mostly in order to focus this short paper’s contribution, and to be able to control for as many possibly-confounding variables such as training data. Nevertheless, a more complete followup would have to include attempts to replicate our findings on other languages, aiming for a set as diverse as possible mostly in terms of typology and script.
Our evaluation is limited to intrinsic measures. While this makes development of tokenizers easier, we acknowledge that the body of work correlating success on these measures with performance of downstream models on end-tasks is incomplete.
Ethical Considerations
Details for human annotation for the cognitive benchmark are documented in the source benchmark’s paper (Beinborn and Pinter, 2023), from which we took the data as-is.

heading: Acknowledgments
text: We would like to thank Charlie Lovering, Varshini Reddy, and Haoran Zhang for comments on early drafts of this paper. We thank the anonymous reviewers for their comments on our submission. This research was supported in part by the Israel Science Foundation (grant No. 1166/23) and by
a Google gift intended for work on Meaningful Subword Text Tokenization.

heading: A Results on Different Vocabulary Sizes
text: Table 4 presents benchmark results on 32K-sized and 49K-sized vocabularies.

heading: B Detailed Results
text: Table 5 breaks down the results (for 40K) on individual morphological datasets composing our benchmark. Table 6 Provides the same for individual cognitive measures.
C Inter-Metric Correlations
Table 7 presents the Pearson correlation coefficients between the various intrinsic metrics used in the benchmark. These correlations are calculated based on the aggregated results across all vocabulary sizes.
Vocab Inference Morphological Cognitive Rényi Tokens Decoding method alignment plausibility efficiency per word diff
Vocab Inference Ladec Morpho- Morphy- Dago- Uni- UnBlend CompoundLex Net Bert Morph Piece

heading: Future Work
text: In the future, we plan to examine the correlation between our benchmark and various downstream tasks, as well as expand our experimentation to other languages and new algorithms. Limitations
Our paper contains evaluation of models in the English language. This was done mostly in order to focus this short paper’s contribution, and to be able to control for as many possibly-confounding variables such as training data. Nevertheless, a more complete followup would have to include attempts to replicate our findings on other languages, aiming for a set as diverse as possible mostly in terms of typology and script. Our evaluation is limited to intrinsic measures. While this makes development of tokenizers easier, we acknowledge that the body of work correlating success on these measures with performance of downstream models on end-tasks is incomplete. Ethical Considerations
Details for human annotation for the cognitive benchmark are documented in the source benchmark’s paper (Beinborn and Pinter, 2023), from which we took the data as-is