Explaining and Mitigating Crosslingual Tokenizer Inequities
Authors: Catherine Arnett
Tyler A. Chang
Stella Biderman
Ben Bergen
Venue: NeurIPS 2025 poster ()
Keywords: tokenizers
subword tokenization
multilingual
Primary area: deep_learning
TL;DR: We identify several factors that lead to token premium effects in monolingual tokenizers and provide two interventions which significantly reduce tokenizer inequities.

Abstract: The number of tokens it takes to encode parallel text in different languages is known to vary. These disparities are called *token premiums*. Having high token premiums leads to less throughput during training and increases costs at inference. 
In this paper, we show that even after controlling for dataset size, vocabulary size, and data content, monolingual tokenizers exhibit a wide range of token premiums across languages. To understand the cross-linguistic differences that cause these token premiums,
we train a suite of approximately 7,000 comparable monolingual tokenizers for 97 languages, manipulating tokenization algorithm vocabulary size, and dataset size. We measure token premiums and test for a relationship between factors such as data similarity (between tokenizer training and evaluation), vocabulary size, and pre-tokenization. We also investigate the role of language-specific features such as writing system and word length. We find that similarity between training and test data does not impact token premiums, but vocabulary size and pre-tokenization do. While simply increasing vocabulary size does not lead to reduced token premium effects, we can determine an "optimal" vocabulary size for each language to achieve significantly reduced token premium effects. We also train superword tokenizers which allow merges over whitespaces, and we find that they both reduce token premium effects and improve compression overall. Thus, intervening on the vocabulary size or the pre-tokenizer significantly reduces crosslingual token premium effects.

Explaining and Mitigating Crosslingual Tokenizer Inequities
Catherine Arnett Tyler A. Chang', Stella Biderman', and Benjamin K. Bergen
EleutherAI, UC San Diego
Abstract
The number of tokens it takes to encode parallel text in different languages is known to vary. These disparities are called token premiums. Having high token premiums leads to less throughput during training and increases costs at inference. In this paper, show that even after controlling for dataset size, vocabulary size, and data content, monolingual tokenizers exhibit a wide range of token premiums across languages. To understand the cross- lissuliig differences that cause these token premiums, we train a suite of approximately 7,000 comparable monolingual tokenizers for 97 languages, manipulating tokenization algorithm, vocabulary size, and dataset size. We measure token premiums and test for relationship between factors such data similarity (between tokenizer training and evaluation), vocabulary size, and pre-tokenization. We also investigate the role of language- specific features such as writing system and word length. We find that similarity between training and test data does not impact token premiums, but vocabulary size and teettekenizaionn do. While simply increasing vocabulary size does not lead to reduced token premium effects, we can determine an "optimal" vocabulary size for each language achieve significantly reduced token premium effects. We also train superword tokenizers which allow merges over whitespaces, and we find that they both reduce token premium effects and improve compression overall. Thus, intervening on the vocabulary size or the pre-tokenizer significantly reduces crosslingual token premium effects.
MonTok Tokenizers
Code and Data
Introduction
Multilingual tokenizers generally require different numbers of tokens to represent parallel text in different languages. In other words, multilingual tokenizers are able to compress text better for some languages than for others. This phenomenon, token premiumm (Petrov et al., 2023 ) leads to increased
inference costs and latency for languages with high premiums (Ahia et al.) 2023 Petrov et 2023). In previous work, these effects have been demonstrated only in multilingual tokenizers. which are trained on different proportions data per language. be driven exclusively by the proportion of training data per language. However, in this paper, we show that monolingual tokenizers trained with exactly the same implementation, dataset size, and vocabulary size demonstrate widely variable token premium effects (Fig. D
possible, therefore, that these effects could
We then ask why tokenizers trained in the same way lead to these different compression rates for different languages. We first determine the effects of byte premiums (variable byte encoding lengths for content-matched text in different languages; Arnett et al. 2024) and tokenizer types (BPE and Unigram) on compression (Section 3). We find that BPE tokenizers have the best compression rates overall, and byte premium scaling of the training dataset size has little effect. We then identify factors that correlate with token premiums (Section 4) including train-eval domain similarity, mean
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Vocab Size
0192
16384
32768
49152
65536
81920
98304
Density
114608
131072
40000
50000
60000
70000
80000
90000
100000
110000
FLORES CTC
Figure I: Distribution of corpus token counts (CTCs) for all languages in our sample as a density plot over CTC scores, for the BP-unscaled BPE tokenizers. Higher CTCs indicate worse compression. The distribution for each vocabulary size is plotted separately. As vocabulary size increases, the distribution shifts left and becomes more compressed, and the right tail becomes less extreme.
token length, and proportion of whitespaces in each language. In total, we train approximately 7000 monolingual tokenizers, which we make available on Hugging Face: htttps://uugingtee
Finally, explore ways to mitigate token premium inequities by training tokenizers on parallel data, adjusting vocabulary sizes per language, and removing whitespace pre-tokenization (Section We find that training on parallel data does not meaningfully reduce token premium effects. However, we can determine an "opttiml"" vocabulary size for each language, which significantly reduces token premiums. Additionally, we find that removing whitespace pre-tokenization (i.e. SuperBPE tokenizers: Liu et al. 2025) also substantially reduces token premium effects.
In order to fully address token premium effects and design more equitable multilingual tokenizers, we need to understand all the factors that lead to token premium effects. In this work, we show that interactions between tokenizer design and inherent features of language largely determine differences in compression. These findings represent a key step in understanding the implicit biases that tokenizer design choices might incur.
Related Work
2.1
Token Premiums
Token premiums are the relative differences in the number of tokens used to encode the same content in different languages (Petrov et al., 2023). higher token premium means that more tokens are needed encode the same content, which corresponds to lower compression. High token premiums have sometimes been referred to as over-segmentation (Rust et al.) (20T)) or over-tokenization (Liang et al. (203)). Ahia et al. (2023) and Petrov et al. (2023) showed that multilingual tokenizers often have variable token premiums for the different languages they are trained on.
Token premium effects have only previously been observed in multilingual tokenizers. Multilingual tokenizers trained on different amounts of data for different languages, and some languages are over- or under-represented in the tokenizer vocabularies (Balhar) (023). In these tokenizers, therefore, differences in token premiums may be due training data amounts vocabulary allocation per language. The current paper, to the best of our knowledge, is the first to systematically document token premiums comparably trained monolingual tokenizers. In this paper, therefore, we make distinction between crosslingual and multilingual. By crosslingual, we mean differences between languages that are observed monolingual settings. We use multilingual to refer to settings where tokenizer model is trained on multiple languages.

---

To reduce token premium inequities, byte-level tokenization (sometimes referred to as "tokenizer- free"; Choe et al., 2019 Clark et al., [022] Xue et al., 2022 inter alia) has been considered as an alternative tokenization approach. However, different languages require different numbers of bytes to encode equivalent content (Arnttt et al. 2024), a phenomenon known as "byte premiums". Therefore byte-level tokenizer, such as that for ByT5 (Xue et al., (022), still has token premiums- in this case, identical to byte premiums. Some languages have extremely high byte premiums (Burmese: 3.51, Dzhongkha: 3.64; Shan: 3.94; Petrov et al. 2023).
There have been at least two proposed methods reduce token premiums in multilingual byte-based tokenizers. First, [imisiewicz et al. (2024) proposed MYTE, which replaces long byte strings with unused bytes that represents meaningful word segments, morphemes. This method leads to significantly reduced token premium, or byte premium, effects, but it requires a predefined dictionary of morphemes. Relatedly, Ahia et al. (2024) proposed MAGNET, which takes byte-level inputs and uses script-specific boundary- preddiccorrrooorrroo modules to dynamically predict segmentation boundaries. Those boundaries are used to pool together byte-level representations before feeding them into the language model input. This leads to more equitable segmentation across languages. By contrast, in this paper, we discuss methods to mitigate token premiums in subword tokenizers, without modifying the tokenization algorithm.
2.2 Measuring Compression
generally desirable for text to be encoded in as few tokens as possible, iee. to be maximally compressed, and therefore have low token premium. In this paper, we measure compression using corpus token count (CTC: Schmidt et al. 2024). which is the total number of tokens in a corpus. We compute CTC over parallel texts, thereby holding information content constant. As result, we can compare CTCs across languages as an operationalization of compression, where lower CTC corresponds to higher compression. CTCs can be converted token premiums in Petrov et al. (2023) by normalizing them by the CTC of reference language (e.g. English).
Two other compression metrics are less well suited to the present work. First, fertility, another common metric of compression, takes the average number of tokens per word (Rust al. (022). Lower fertility corresponds to better compression. This metric requires functional definition of wordhood, which does not exist"] Even if did, different languages encode different amounts of information per word. For example, Eastern Canadian Inuktitut has single word which means "They wanted to catch a walrus" (Gutierrez- Vasques et al. 2023 citing Fortescue, 1992 Dahl 2017). We thus opt not to use fertility, as it is unclear how to fairly calculate this across different languages. Second, Schmidt et al. (2025) use bytes-per per-token as a metric for compression. However, because different languages encode different amounts of information per byte (byte premiums; Arnett et al.) 2024). this measure is not suitable for comparing compression across languages.
2.3
The Relationship Between Compression and Performance
How compression relates to language model performance on downstream tasks remains uncertain.
Higher compression rates mean that sequences of a fixed length contain more information, which may
lead to improved model performance (Deletang al. 2024), and several studies have demonstrated positive relationship between higher compression and task performance (Goldman et al.) [022]] Gallé
2019). However, through careful manipulation of various aspects of the tokenizer, Schmidt et al.)
(2024) recently found that there was no relationship between compression and model performance. These experiments are limited to English, and it is thus unclear the extent to which they generalize to other languages. We argue that the statistical properties of tokenizers are important in and of themselves. Higher compression leads to lower training and inference costs due to shorter sequence lengths for the same content. Therefore, compression dictates the computational and fiscal cost of both inference and training over fixed dataset sizes.
'In English, this often operationalized as whitespace seatesppcc--sprrateed strings. However, there exist cases, e.g. compound words like 'ice cream', which contain interword whitespaces, but otherwise behave like single words (wickyy 1986). Some languages, e.g. Chinese, do not use whitespaces at all. There is not another definition of wordhood that is widely accepted (Haspelmath) 2017).
3

---

3
Tokenizer Training
For the experiments below, we train a set of subword tokenizers to be comparable across languages, matching training dataset size, vocabulary size, and training method. This allows us to fairly evaluate differences in compression across languages. We use the text datasets from Chang et al. (2024) for all tokenizer training, restricting to the 97 languages with at least 300MB of text data and for which the training data was not significantly contaminated with the FLORES dataset (see Appendix C for details). We use the Hugging Face tokenizers (Hugging Face 2020) package to train these tokenizers. For each language, we train a tokenizer for two tokenizer types (BPE and Unigram; Semnrich et al. 2016; Kudo 2018) 300MB of text data, for seven vocabulary sizes ranging from 16384 to 114688. All vocabulary sizes we use are divisible by 128, because this has been shown to lead to better model throughput (Anthony et al. 2024 Groeneveld al., 2024). We also manipulate whether we scale the tokenizer training dataset size by each language's 's byte premium, the ratio of bytes required to encode content-matched text in each language compared to English. We call this byte-premium scaling (henceforth BP-scaling; /Amett et al., 2024). All 97 languages are listed in Appendix A and additional details about tokenizer training can be found in Appendix B
One potential concern about our tokenizers is their relatively small training dataset size. Many tokenizers trained on gigabytes of text, e.g. Schmmidt et al. (2024). Reddy et al. (2025) recently showed that up to training dataset sizes of 180GB, additional data may lead to better tokenization quality measured by morphological alignment and correlations with metrics of human language processing. Our tokenizers are trained on substantially less data, in order to prioritize the inclusion of lower-resource languages. 300MB was the largest dataset size available for the language in our sample with the smallest amount of available data. Still, we find that our tokenizers have comparable compression rates to those of OLMo, Pythia, and SmolLM (Groeneveld et al. 2024 Biderman et al., 2023 Allal et al. 2025) see Appendix DD). This suggests that our dataset size is not too small to draw conclusions about the role of tokenizer type, vocabulary size, and other tokenizer training decisions on compression.
Measuring Compression. We calculate token premiums by calculating CTC parallel text for the corresponding language for each tokenizer. We use the FLORES-206 dataset (Costa-Jussà et al.) 2022), which is high-quality parallel translation dataset and which includes all 97 languages in our sample. The CTC for each language (for a given tokenizer type, BP-scaling, and vocabulary size) is the un- normalzal token premium; our results are equivalent using the token premiums in PPetrov jet al.] (2023), who normalize by the CTC of a reference language (English). In our results below, a higher CTC indicates worse compression.
3.1 Effects of Byte Premium Scaling and Tokenizer Type
Byte Premium Scaling. First, we discuss the impact of BP-scaling the tokenizer training dataset size. We fit linear mixed effects model with the 1me4 package in R (Bates et al., 2014), using CTC as the dependent variable. As predictors, we include a fixed effect of BPP-scllnng (a binary variable that indicates whether or not the training data is BP-scaled) and random effects for vocabulary size and tokenizer type. We find no effect of BP-scaling the tokenizer training data (I(3544)=00.615 p=0.539). We provide further motivation for this analysis and further results in Appendix For the remaining experiments, we use tokenizers trained with unscaled training dataset sizes.
Tokenizer Type. Next, we turn to the role of tokenizer type. Comparing our BPE and Unigram tokenizers, we find that across vocabulary sizes, BPE and Unigram show a wide distribution of CTCs and therefore a wide range of token premiums, with BPE exhibiting slightly lower CTCs (i.e. better compression) overall. We provide further visualization in Appendix
We also compare with the SentencePiece (Kudo and Richardson 2018) implementation of Unigram tokenizers. We use the tokenizers from the Goldfish models (Chang et al. 2024). which were trained on 5MB of BP-scaled dats and with vocabulary size 50000 [ We evaluate all the Goldfish tokenizers for which there exists a corresponding FLORES dataset, for a total of 183 tokenizers.
Though we noted that BP-scaling on compression, we do not expect this to impact the results. We note that our BPE and Unigram tokenizers have vocabulary sizes of 49152, while the SentencePiece tokenizers have vocabulary sizes of exactly 50000.
using non-BP-scall scaled training data throughout, because found effect of

---

Figure 3 (left) shows the distribution of CTCs for the tokenizers of each type. All tokenizer types show a wide spread of CTCs across languages, though the absolute CTCs vary. BPE tokenizers demonstrate the most compact distribution and the lowest overall CTCs. This is consistent with previous results in English contexts showing that BPE offers more compression than other tokenizers (Gallé) 2019; [Schmidt et al.) 2024), but to the best of our knowledge this has not been shown to be true crosslinguistically SentencePiece implementation of Unigram has the worst compression rates of all tokenizer types tested.
3.2 Final Tokenizer Setup
The results in this section suggest that the wide range in token premiums across languages is not unique to any tokenization algorithm, but BPE does exhibit slightly reduced trogs-linguistic differences CTC as well as lower CTCs (better compression) overall. Therefore, for the remaining analyses, use only the BPE tokenizers with BP-unscaled training dataset sizes. We train tokenizers with an even wider range of vocabulary sizes (from 8192 to 2621144 (62144), which we use the experiments in Section[4]
4 Explaining Token Premiums
In the previous section, showed that tokenizers trained on the same amount of data, with the same vocabulary size, and with the same training procedure, there still show a wide range of compression rates across languages. In this section, test various potential factors which may drive these crosslinguistic differences in token premiums. We fit linear regressions for each of the factors and report the amount of variance explained by the top predictors (Table
Predictor
R²
Data Similarity
0,239
Mean Token Length (vocab)
0.168
Proportion Whitespaces
0.157
Combined
0.297
Table 1:
R² values for linear regressions predicting FLORES CTC from different predictors, for tokenizers with a vocabulary size of 65536. Values for all predictors are provided in Appendix H.3
Data Similarity. The datasets we trained our tokenizers on come primarily from web text (Chang et al. (2024). but the domain and quality may vary widely across languages, especially for lower- resource languages (Kreutzer al. 2022). If the training data is low-quality or contaminated with data from other languages. this may affect the ability of the tokenizer to learn a vocabulary that leads to effective compression. Additionally, if the training data differs too much from the evaluation dataset, c.g. as measured by word overlap, the tokenizer may also Iearn a sub-optimal vocabulary (Dagan al. 2024). Here, we measure train-eval data similarity as the number of overlapping tokens between the training and evaluation datasets, normalized by the vocabulary size. Data similarity is the factor that explains the most variance in CTCs, suggesting that mainl data similarity may be important for token premiums. In Section 5.1 we train tokenizers on parallel data and test whether this reduces token premiums.
Mean Token Length. Mean token length, which measure in in characters, is directly related to compression, as in order to represent a text with fewer tokens each token must be longer. We calculate mean token length in two different ways. First, we calculate the mean length of each token in the tokenizer's 's vocabulary (this is the metric reported in Table T). Second, we calculate the mean token length when only including the tokens used tokenize the FLORES dataset for each language. The latter should be closely linked with FLORES CTC, as measures token lengths in the evaluation corpus itself.
We find that mean token length over the vocabulary is not significant predictor of CTC, but mean token length over FLORES has an R² of 0.168. This suggests that differences in CTC are not due to
5

---

Vicio
Deglish Donnale
Evep Derilago
Englin beeide
Palicse tarnalid
Englin bandled
Figure 2: Comparison of CTCs for tokenizers trained non-parallel (solid lines) versus parallel (dashed lines) data for three languages: Luxembourgish, Khmer, Maltese. Figures for the remaining languages are in Appendix
shorter or longer tokens overall in the tokenizer. Instead, differences in CTC are related specifically to the length of tokens that actually used in the evaluation dataset; tokenizers with high CTCs may have longer vocabulary items in their vocabularies, but they are not being used as frequently. To mitigate CTC differences due token length, size should increase mean token length and therefore lower CTC. In Section 5.2 we test not only the effect of increasing vocabulary size, but also of identifying an "optimal" vocabulary size for each language, on token premium effects.
hypothesize that increasing the tokenizer vocabulary
Proportion of Whitespaces. Most BPE tokenizers use a pre- pre-tokenizztion step, which splits text into pre-tokens. During training, the tokenizer cannot learn merges that cross the pre-token boundaries. One of the most common pre-tokenization steps splits texts at every whitespace. Languages differ in how many whitespaces (equivalently, how many space-separated "words") they use to encode the same information (see examples in Appendix G). This means that whitespace pre-tokenization could affect different languages different extents. Indeed, when we measure the proportion of whitespaces in FLORES for each language, we find that it explains substantial amount of variance CTCs (Table Motivated by this finding, Section 5.3 we train "superword" tokenizers (Liu Schmidt et al. 2025) that allow merges across whitespaces.
al.
2025
Other Metrics. We also test various other metrics related to language and its script (writing system), including number of unique phonemes, entropies over characters and bigrams (character pairs), and language's script itself when predicting CTCs. We provide more detailed discussion and report full results for all metrics in Appendix H In the current section, we focused on data similarity, mean token length, and proportion of whitespaces, because these metrics predict significant amounts variance. Next, we propose interventions for each of these factors.
Mitigating Token Premiums
5.1 Data Quality and Data Similarity
If either data quality or train-eval data similarity is driving the token premiums, then we should see that differences in CTC decrease when tokenizers are trained on parallel text. Parallel text is content- content-matched and therefore has identical domain overlap with the evaluation dataset. To test this, we selected seven languages which had the highest CTCs at at least one vocabulary size. We did this in order to select the languages for which there were the greatest token premium effects, provide more opportunity for reduction of the effects. We take the parallel data for each target language and English in NLLB (Costa-Jussà al., (022), and we create 300MB subsets for each language. Because NLLBB is not multi-parallel, we have a different English dataset for each target language. We train new BPE tokenizers for each new parallel dataset using the same vocabulary sizes as described in Section 3.2 We calculate CTC in the same way as previous sections, and we compare the CTCs from the original monolingual tokenizers with those from the tokenizers trained parallel data. We show the results for three languages in Figure
'These languages Cebuano, Khmer, Luxembourgish. Mandarin Chinese, Maltese, Japanese, and Burmese.

---

Optiall Vocab IBPE)
Vocab
40000
80000
FLONES CTC.
Dendty
70000
60000
50000
40000
SPC (MP)
Unignem (HF)
Unigram ISP)
50000
60000
70000
Schenieet Type
FLORES CTC
Figure 3: Left: the distribution of CTCs for BPE and Unigram tokenizers using Hugging Face
(HF) or SentencePiece (SP) implementations, all with vocabulary sizes of approximately 50000. Right: the distribution of CTCs for same-vocabulary and optimal-vocabulary BPE tokenizers. The aptimal-vocabule tokenizers have target CTCs of 56000. The coma-vocabulary tokenizers have vocabulary size 32768.
Comparing the original and parallcl-data tokenizers for the seveb languages in our sample, we find that the parallel-daa tokenizers have statistically significant reduction in token premium effeets (paired t-test; (1152)=2.355) p=0.0197). The difference between the CTCs is quite small, however: on average about 1% of the total CTC. We also find that the majority of this effect is driven by the tokenizers with the smallest vocabulary size (16384 for these tests). Statistical tests per vocabulary size are reported in Appendix [.2] Overall, these results indicate that training on parallel data has small effect on CTC, but it does not meaningfully reduce token premium effects
5.2 Vocabulary Size
Next, we evaluate whether different vocabulary sizes have greater lesser token premium effects. While increasing vocabulary size reduces CTC overall, even the largest vocabulary size has large token premium effects (Figure [p. We test the difference in variance between CTC distributions for the smallest and largest vocabulary sizes with a Fisher-Snedecor test (F-test), and we find no significant difference 1696996=1.11 p=0.565).
Then, we test whether we can find an "optimal" vocabulary size per language such that there is reduction in variation of CTCs. To do this, we fit a power law curve for the relationship between CTC and vocabulary size for each language (e.g. the curves Figure ( Using these curves, for each language, we predict the vocabulary size at which the tokenizer will reach given CTC. We then train new tokenizers the "optimal" vocabulary size for a given target CTC. We provide more detail on the training of these tokenizers and validation of our estimation method in Appendix[]
Tokenizers with optimal vocabulary sizes per language have reduced token premium effects compared to tokenizers with the same vocabulary size across languages (Figure 3 right). We find the optimal- vocabulary tokenizers have significantly less variance in CTCs (F-test; Fs0.387=0.1500 p <0.001). For BPE tokenizers, therefore, different languages need to be allocated different vocabulary sizes to achieve more comparable compression.
5.3 Whitespace Pre- Tre-Tokenization
this, we train superword tokenizers, which allow merges over whitespaces. Therefore, tokens may be composed of multiple whitespace-separated words and the effect of whitespace pretokenization on compression should be reduced. We use the SuperBPE (Liu et al., 2025) implementation. Our tokenizers are the first non-English superword tokenizers to the best of our knowledge. We train only tokenizers for vocabulary sizes over 64000, as
Finally, we intervene on the effect whitespaces.
'We test whether the parallel-date tokenizers have higher data similarity between training and evaluation data (as measured in Section 4. They do (paired f-test: ((770.55) 1=0. This may explain why this intervention did not lead greatly reduced token premium effects.
7

---

Optimal (BpperPP))
Vocab Buserare)
Denstty
5004
5068
6668
4000
1000
40000
60000
80000
100000
120000
62409
FLORES CTC
Figure 4: Left: the distribution of CTCs for BPE and SuperBPE tokenizers for each vocabulary size. Right: the distribution of CTCs for same-vocabulary and predicted CTCs for the optimal-vocabulary SuperBPE tokenizers. The optimal-vocabulary tokenizers have target CTCs of 43000. The same- vocabulary tokenizers have a vocabulary size of 49152.
we predict the superword tokenizers need larger vocabulary sizes to learn longer superword tokens. Thus, we expect to see the biggest differences between BPE and SuperBPE tokenizers the higher vocabulary sizes. We include additional details about training SuperBPE tokenizers in Appendix K
We find that SuperBPE tokenizers demonstrate lower average CTCs and less variance in CTCs every vocabulary size (Table i0 [opendix[]). We plot the distributions in Figure 4 We also run linear regression to test the relationship between proportion of whitespaces and CTC (as Section but find that there is still significant relationship at all vocabulary sizes except 65536 (full results Table in Appendix L..
We then test whether setting optimal vocabulary sizes for each language would lead to reduced token premium effects in SuperBPE tokenizers. We use the same method to determine the optimal vocabulary size for each language as we did for BPE (see Appendix M). We did not train SuperBPE tokenizers for each optimal vocabulary, however demonstrated that CTC estimation method was accurate for BPE (Appendix J.3) We estimate there to be significantly less variance in CTC for the pptimal-vocabulae p<0.001). This result suggests that training optimal vocabulary tokenizers for SuperBPE would lead further reduced token premium effects (Figure right).
set than the noca-vocyyulary set for SuperBPE (F-test; 1tss,,6=3.8777
5.4 Remaining Variance
There are still significant effects of length ratio and whitespace pre-tokenizers and optimal vocabulary sizes (results in [ppendix[N]. These are features which are determined by inherent properties of the languages, the writing system, the way the writing system is encoded in UTF-8, and their interactions. Future work can seek ways to address remaining inequities, especially those introduced by UTF-8. Recent work has introduced novel encoding scheme to replace UTF-8 in the context of tokenization, in which all Unicode characters are represented with the same number of byte-equivalent units (Land and Arnett (005).. This encoding scheme does not address any differences introduced by the length ratios, however.
after the interventions on
Discussion
The Impact of Fre-Tokenization. Whitespace pre-tokenization responsible for large portion of the crosslinguistic differences in compression. This is consistent with recent work that showed that pre-tokenization had larger impact on model performance than any other tokenizer design choice tested, including the training corpus and vocabulary size (Wegmann et al. 2025). Relatedly, Velayuthan and Sarveswaran (2025) show that many common pre-tokenizers segment text below the grapheme level for multiple South Asian languages (Hindi, Tamil, Sinhala). In this case, this is due to regular expression pre-tokenizers, which often pre-tokenize on punetuation and other symbols. This may include common diacritics and sub-graphemic units in non-l atin scripts, leading to catastrophic

---

compression for some languages. The work in this paper contributes to growing body of work that shows the unequal impact of pre-tokenizer design decisions crosslinguistically.
Sanguage-Speciffe Considerations for Language Model Design. We show that there is no vocabulary size at which monolingual tokenizers will achieve similar compression across a wide variety of languages. However, sizes. Vocabulary size, therefore, should perhaps be a design choice that varies according to the target language languages. Relatedly, Chang et al. (2024) argue that it may be necessary to scale language model training data in order to account for byte premiums and achieve comparable performance across languages (Chang al. (024). The same setrings that work well for English will not necessarily work equally well for all languages.
overcome this through setting language-spectin
vocabulary
Too Much Compression? One concern is that some of the tokenizers we train especially the SuperBPE tokenizers provide too much compression. While there has been work discussing the relative benefits (Gallé 2019) (or lack thereof; Schmidt et al. 2024) of increased compression, there has been less work on the negative impacts of increased compression. Perhaps too much compression could be giving the model effectively less time to think, by dedicating fewer FLOOPS to predicting each next token. Goyal et al. (2024) found that adding "pause" tokens allowed models to perform additional computations before providing an answer and found that this increased performance.
Applications Multilingual Tokenization. These findings inform future development of more eq- uitable multilingual tokenizers. As it is increasingly common to train language models on multilingual data, it is more important than ever reconsider decisions that might be disadvantaging performance or increasing cost for some languages more than others. Simply allocating more vocabulary to some languages may drastically improve compression. Vocabulary allocation, or how much of the vocabulary of a multilingual tokenizer is dedicated to a particular language, is correlated with model performance (Limisiewicz et al. 2023). Our results suggest that some languages might benefit from an increase in vocabulary allocation more than others.
Petrov et al. (202) proposed training monolingual tokenizers with similar compression and combining them to achieve more balanced token premiums. Our work shows that if you did this with monolingual tokenizers with the same vocabulary size, this would likely not mitigate token premium effects. tokenizers for each language,
However, this method were employed with might lead to reduced token premium effects.
The Link Between Compression and Performance. We have thoroughly tested the relationship between different language features and compression, but we have not directly linked this to model performance. Token premiums affect the cost of training and inference, they determine the number of computations needed for a text of fixed length. Therefore, we believe the results of this paper are informative, even if they do not impact model performance. While for English, small changes in compression may not be correlated with better performance (Schmidt et al. 2024), monolingual English tokenizers have among the lowest CTCs (Figure 13). Thus, differences in compression may be smaller relative to other languages. For languages with higher CTCs, improvements in compression may be more drastic and may lead to significant performance gains.
Additionally, there likely to be a complex relationship between compression and other tokenizer properties, which may impact performance. We showed that Unigram tokenizers showed the worst compression across languages, but Bostrom and Durrett (2020) showed that Unigram tokenizers segment text into more linguistically-aiinen units than BPE. It remains unclear, however, whether having linuuisically-alignedd tokens clearly linked to better language model performance (Arnett) and Bergen, 2025).
7 Conclusion
This work investigates how different language-specific features interact with common tokenization algorithms. First, we show that training monolingual tokenizers in the same way leads to different compression rates for different languages, i.e. token premium effects. We identify three main factors which might lead to this effect. First, we ask whether these effects are due to differences in the training data. After training tokenizers parallel data, we find that data similarity does not meaningfully

---

explain crosslingual differences in CTC. Second, we test whether manipulating vocabulary size decreases differences in compression. Increasing vocabulary size does not decrease token premium effects; however, we show that training a tokenizer with an "optimal" vocabulary size for each language drastically reduces differences CTC. Finally, we analyze the effect of pre-tokenization. Different languages use varying amounts of whitespaces, meaning that whitespace pre-tokenization affects languages differently. We train superword tokenizers, which allow merges over whitespace boundaries. This not only reduces token premium effects, but overall improves compression. Together, these results help to inform the development of efficient and equitable tokenizers for more languages.
8 Limitations
While we aimed to be comprehensive in our experiments and to carefully manipulate all variables, some of our experiments were limited in breadth. For the tokenizers we trained on parallel data, we could not train on as many languages due to the availability of parallel data. Additionally, for each language, we had to train new baseline tokenizer, doubling the compute and data requirements for that experiment. For the parallcl-data tokenizers, we were also not able to compare across languages, but only between target language and an English baseline. This was due to the availability of parallel data, which is even more limited than multilingual pre-training data. This could potentially also be limitation of the analysis. It is unclear whether comparisons with different languages would lead to different results.
While we showed that we achieved similar compression to pre-trained tokenizers with our small dataset size, may be the case that the dataset sizes we used were too small for the SuperBPE tokenizers. It could be the case that SuperBPE tokenizers need more data than BPE tokenizers, in order to prevent overfitting to the training data, especially for longer and lower-frequency tokens.
We solely used FLORES as the evaluation dataset for all of our experiments. FLORES is a dataset created by translating from English into each of the languages represented in the dataset. Therefore translation artifacts or differences in translation quality may affect our results.
Relatedly, we measured data similarity as the number of overlapping tokens, which could potentially be measuring token diversity in addition to or instead of data similarity. This could explain why we did not see large changes in data similarity between the non-parallel-data and parallel-data tokenizers.
Finally, we did not train any language models with our tokenizers, therefore we cannot make any claims about the connection between our findings and changes in language model performance.
Acknowledgments
We would like to thank the UC San Diego Social Sciences Computing Facility Team for the use of the Social Sciences Research and Development Environment (SSRDE) cluster. Tokenizers were trained and evaluated using hardware provided by the NVIDIA Corporation as part of an NVIDIA Academic Hardware Grant. We thank Alisa Liu and Jonathan Hayase for their assistance with the crosslingual implementation of SuperBPE.
References
Ahia, O., Kumar, S., Gonen, H., Hofmann, V., Limisiewicz, T., Tsvetkov, Y., and Smith, N. A. (2024). MAGNET: Improving the multilingual fairness of language models with adaptive gradient- based tokenization. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
Ahia, O., Kumar, S., Gonen, H.,, Kasai, J., Mortensen, D., Smith, N., and Tsvetkov, Y. (2023). Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models. In Bouamor, H., Pino, J., and Bali, K., editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9904-9923, Singapore. Association for Computational Linguistics.
Allal, L. B., Lozhkov, A., Bakouch, E., Blázquez, G. M., Penedo, G., Tunstall, L., Marafioti, A., Kydlíček, H., Lajarín, A. P., Srivastav, V., et al. (2025). SmolLM2: When Smol Goes Big-Data- Centric Training of a Small Language Model. arXiv preprint arXiv:2502.0273::
10

---

Anthony, Q Hatef, J.. Narayanan, D., Biderman, S., Bekman, S., Yin, J., Shafi, A., Subramoni, H., and Panda, D. (2024). The case for co-designing model architectures with hardware. In Proceedings of the 53rd International Conference on Parallel Processing, pages 84-96.
Arnett, C. and Bergen, B. (2025). Why do language models perform worse for morphologically complex languages? In Rambow, O., Wanner, L., Apidianaki. M., Al-Khalifa, H., Eugenio, D., and Schockaert, S., editors, Proceedings of the s1 International Conference Computational Linguistics, pages 6607-6623, Abu Dhabi, UAE Association for Computational Linguistics.
Amett, C., Chang, T. A., and Bergen, B. (2024). A Bit a Problem: Measurement Disparities in Dataset Sizes across Languages. In Melero, M., Sakti, S., and Soria, C.,, editors, Proceedings of the 3rd Annual Meeting of the Special Interest Group on Under-resourced Languages LREC- COLING 2024, pages 1-9, Torino, Italia. ELRA and ICCI..
Balhar, J. (2023). Improving subword tokenization methods for multilingual models. Master's thesis, Univerzita Karlova, Matematicko-fyzikkaa fakulta, Prague, Czech Republic.
Bates, D., Mächler, M.. Bolker, B. M., and Walker, S. C. (2014), lme4: Linear mixed-effects models using eigen and s4. package.
Biderman, S., Schoelkopf. H., Anthony, Q G., Bradley, H., O'Brien, K., Hallahan, E.,, Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al. (2023). Pythia: A suite for analyzing large language models across training and scaling. International Conference on Machine Learning, pages 2397-2430. PMLR.
Bostrom, K. and Durrett, G. (2020). Byte Pair Encoding is Suboptimal for Language Model Pretraining. In Cohn, T., He, Y., and Liu, Y., editors, Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4617-4624, Online. Association for Computational Linguistics.
Chang, T. A., Arnett, C., Tu, Z., and Bergen, B. K. (2024). Goldfish: Monolingual language models for 350 languages. arXiv preprint arXiv:2408. 10441.
Choe, D., AI-Rfou, R., Guo, M., Lee, H., and Constant, N. (2019). Bridging the gap for tokenizer-free language models. arXiv preprint arXiv: 1908. 10322.
Clark, J. H., Garrette, D., Ture, I., and Wieting, (2022). Canine: Pre-training an efficient tokenization-free encoder for language representation. Transactions of the Association for Compu- tational Linguistics, 10:73-91.
Costa-Jussà, M. R., Cross, J., Çelebi, O., Elbayad, M., Heafill, K., Heffernan, K., Kalbassi, E., Lam, J., Licht, D., Maillard, J., et al. (2022). No Language Left Behind: Scaling Human-Centered Machine Translation. arXiv preprint arXiv:2207.00 2.7.66777..
Dagan, G., Synnaeve, G., and Roziere, B. (2024). Getting the most out of your tokenizer for pre-training and domain adaptation. In Forty-first International Conference on Machine Learning.
Dahl, O. (2017). Polysynthesis and complexity. In Fortescue, M., Mithun, M., and Evans, N., editors, The Oxford Handbook of Polysynthesis. Oxford Handbooks. Oxford University Press. Online edition published 6 Nov. 2017, accessed 9 May 2025.
Daniels, P. T. (1990). Fundamentals of grammatology. Journal of the American Oriental Society, pages 727-731.
Deletang, G., Ruoss, A., Duquenne, P.-.., Catt, E., Genewein, T., Mattern, C., Grau-Moya, J., Wenliang, K., Aitchison, M., Orseau, L., Hutter, M., and Veness, J. (2024). Language modeling is compression. In The Twelfth International Conference on Learning Representations.
Ding, G., Peng, D., and Taft, M. (2004). The nature of the mental representation of radicals in Chinese: priming study. Journal of Experimental Psychology: Learning, Memory, and Cognition, 30(2):530.
Fenk-Oczlon, G. and Pilz, J. (2021). Linguistic complexity: relationships between phoneme inventory size, syllable complexity, word and clause length, and population size, Frontiers in Communication, 6:626032.
11

---

Fortescue, M. (1992). The development of morphophonemic complexity in Eskimo languages. Acta Linguistica Hafniensia, 25(1):5-27.
Gage, P. (1994). A new algorithm for data compression. The C Users Journal, 12(2):23-38.
Gallé, M. (2019). Investigating the effectiveness of BPE: The power of shorter sequences. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1375-1381.
Goldman, O., Caciularu, A., Eyal, M., Cao, K., Szpektor, I., and Tsarfaty, R. (2024). Unpacking tokenization: Evaluating text compression and its correlation with model performance. In Ku, L.-W., Martins, A., and Srikumar, V., editors, Findings of the Association for Computational Linguisties: ACI. 2024, pages 2274-2286, Bangkok, Thailand. Association for Computational Linguisties.
Goyal, S., Ji, Z., Rawat, A. S., Menon, A. K., Kumar, S., and Nagarajan, V. (2024). Think before you speak: Training language models with pause tokens. In The Twelfth International Conference Learning Representations.
Groeneveld, D., Beltagy, I., Walsh, E., Bhagia, A., Kinney, R., Tafjord, O., Jha, A., Ivison, H., Magnusson, I,, Wang. Y., Arora, S., Atkinson, D., Authur, R., Chandu, K., Cohan, A., Dumas, J., Elazar, Y., Gu, Y., Hessel, J., Khot, T., Merrill, W., Morrison, J., Muennighoff, N.. Naik, A.. Nam, C., Peters, M., Pyatkin, V., Ravichander, A., Schwenk, D., Shah, S., Smith, W., Strubell, E., Subramani, N., Wortsman, M., Dasigi, P., Lambert, N., Richardson, K., Zettlemoyer, L., Dodge, J., Lo, K., Soldaini, L., Smith, N., and Hajishirzi, H. (2024). OLMo: Accelerating the science of language models. In Ku, L.-W., Martins, A., and Srikumar, V., editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15789 15809, Bangkok, Thailand. Association for Computational Linguistics.
Gutierrez-Vasquess X., Bentz, C., and Samardžić, (2023). Languages through the looking glass of BPE compression. Computational Linguistics, 4944):943-
Haspelmath, M. (2017). The indeterminacy of word segmentation and the nature of morphology and syntax. Folia linguistica, 51(s1000):31
Hugging Face (2020). tokenizers package. v0.21.1.
Hutong School (2012). How many chinese characters are there? Accessed: 2025-04-26.
Kreutzer, J., Caswell, I., Wang. L., Wahab, A.. van Esch, D., Ulzii-Orshkkh, N., Tapo, A., Subramani. N., Sokolov, A., Sikasote, C., Setyawan, M., Sarin, S., Samb, S., Sagot, B., Rivera, C., Rios, A.. Papadimitriou, L, Osei, S., Suarez, P. O., Orife, I., Ogueji. K., Rubungo, A. N., Nguyen, T. Q., Müller, M., Müller, A., Muhammad, S. H., Muhammad, N., Mnyakeni, A., Mirzakhalov, J., Matangira, T., Leong, C., Lawson, N., Kudugunta, S., Jernite, Y., Jenny, M., Firat, O.,, Dossou, B. F. P., Dlamini, S., de Silva, N., Çabuk Balh, S., Biderman, S., Battisti, A., Baruwa, A., Bapna, A., Baljekar, P., Azime, I. A., Awokoya, A., Ataman, D., Ahia, O., Ahia, O., Agrawal, S., and Adeyemi, M. (2022). Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets. Transactions of the Association for Computational Linguistics, 10:50-72.
Kudo, T. (2018). Subword regularization. Improving neural network translation models with multiple subword candidates. In Gurevych, I. and Miyao, Y., editors, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Lonng Papers), pages 66-75, Melbourne, Australia. Association for Computational Linguistics.
Kudo, T. and Richardson, J. (2018). SentencePiece: simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66-71, Brussels, Belgium. Association for Computational Linguistics.
Land, S. and Arnett, C. (2025). BPE stays on SCRIPT: Structured encoding for robust multilingual pretokenization. In Tokenization Workshop (TokShop) at ICML.
12

---

Liang, D., Gonen, H., Mao, Y., Hou, R., Goyal, N., Ghazvininejad, M., Zettlemoyer, L., and Khabsa, M. (2023). XLM-V: Overcoming the vocabulary bottleneck in multilingual masked language models. In Bouamor, H., Pino, J., and Bali, K., editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13142-13152, Singapore. Association for Computational Linguistics.
Limisiewicz, T., Balhar, J., and Mareček, D. (2023). Tokenization impacts multilinggul language modeling: Assessing vocabulary allocation and overlap across languages. In Rogers, A., Boyd- Graber, J., and Okazaki, N., editors, Findings of the Association for Computational Linguistics: ACL 2023, pages 5661-5681 Toronto, Canada. Association for Computational Linguistics.
Limisiewiez, T., Blevins, T., Gonen, H., Ahia, O., and Zettlemoyer, (2024). MYTE: Morphology- driven byte encoding for better and fairer multilingual language modeling. In Ku, L.-W., Martins, A., and Srikumar, V., editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 15059-15076, Bangkok, Thailand. Association for Computational Linguistics.
Liu, A., Hayase, J., Hofmann, V., Oh, S., Smith, N. A., and Choi, Y. (2025). SuperBPE: Space travel for language models. In Second Conference on Language Modeling.
Moran, S., McCloy, D., and Wright, R. (2014). PHOIBLE online.
Muppalaneni, N. B., Chilukuri, P., and Kumar Raja, (2019). Telugu handwritten vowels.
Nettle, D. (1995). Segmental inventory size, word length, and communicative efficiency. Linguistics, 33(2):359-33
Petrov, A., La Malfa, E., Torr, P., and Bibi, A. (2023). Language model tokenizers introduce unfairness between languages. Advances neural information processing systems, 1..39963-36990
Reddy, V., Schmidt, C. W., Pinter, Y., and Tanner, C. (2025). How Much is Enough? The Diminishing Returns of Tokenization Training Data. arXiv preprint 20222
Rust, P., Pfeiffer. J., Vulić, I., Ruder, S., and Gurevych, I. (2021). How good is your tokenizer? on the monolingual performance of multilingual language models. Zong. C., Xia, F., Li, W., and Navigli, R., editors, Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 31 3118-3135, Online. Association for Computational Linguistics.
Schmidt, C. W., Reddy, V., Tanner, C., and Pinter, Y. (2025). Boundless byte pair encoding: Breaking the pre-tokenization barrier. In Second Conference on Language Modeling.
Schmidt, C. W., Reddy, V., Zhang, H., Alameddine, A., Uzan, O., Pinter, Y., and Tanner, C. (2024). Tokenization Is More Than Compression. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N., editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 678-702, Miami, Florida, USA. Association for Computational Linguistics.
Sennrich, R., Haddow, B., and Birch, A. (2016). Neural machine translation of rare words with subword units. In Erk, K. and Smith, N. A., editors, Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume Long Pupers), pages 1715-1725, Berlin, Germany. Association for Computational Linguistics.
Sun, C. C., Hendrix, P., Ma, J., and Baayen, H. (2018). Chinese lexical database (CLD) large-scale lexical database for simplified mandarin chinese. Behavior Research Methods, 50:2606-2629,
The CLI Team (2025). How many characters are there chinese? Accessed: 2025-04-26.
Velayuthan, M. and Sarveswaran, K. (2025). Egalitarian language representation in language models: It all begins with tokenizers. Rambow, O., Wanner, L., Apidianaki, M., Al-Khalifa, H., Eugenio, B. D., and Schockaert, S., editors, Proceedings of the 3 International Conference on Computational Linguistics, pages 5987-5996, Abu Dhabi, UAF, Association for Computational Linguistics.
13

---

Wegmann. A., Nguyen, D., and Jurgens, D. (2025). Tokenization is sensitive to language variation. In Che, W. Nabende, J., Shutova, E., and Pilehvar, M. T., editors, Findings of the Association for Computational Linguisties: ACI. 2025, pages 11958-10988 Vienna, Austria. Association for Computational Linguisties.
Williams, C. and Bever, T. (2010). Chinese character decoding: a semantic bias? Reading and Writing, 23:589-605. 605.
Xue, L., Barua, A., Constant, N., AI-Rfou, R., Narang, S., Kale, M., Roberts, A., and Raffel, C. (2022). Byt5: Towards a token-free future with pre-trained byte-to-byte models. Transactions of the Association for Computational Linguistics, 10:291-306.
Zwicky, A. M. (1986). Forestress and afterstress. OSU Working Papers in Linguistics, volume 32, pages 46 46-62. The Ohio State University.
A Language Sample
Table lists the 97 languages in sample, along with their language family and ISO codes (comprised of the ISO 639-3, which denotes language, and ISO 15924 codes, which denotes script).
14

---

Table 2: Languages with their Codes and Families
Language Code
Language Name
Language Family
afr latn
Afrikaans
Indo-European
als latn
Tosk Albanian
Indo-Buropean
amb ethi
Amharic
Afro-Asiatic
arb arab
Modern Standard Arabic
Afro-Asiatic
aam_ asm_beng
Assamese
Indo-Eurppean
bak
Bashkir
Turkic
bel_cyrl
Belarusian
Indo-European
ben_beng
Bengali
Indo-Buropean
bos_latn
Bosnian
Indo-Buropean
bul_cyrl
Bulgarian
Indo-Buropean
cat latn
Catalan
Indo-Buropean
ceb latn
Cebuano
Austronesian
ces latn
Czech
Indo-European
ckb_arab
Central Kurdish
Indo-European
sym_latn dan latn
Welsh
Indo-European
Danish
Indo-European
latn
German
Indo-European
ell grek
Greek
Indo-European
eng_latn epo epo_latn est latn cus latn
English
Indo-European
Esperanto
Constructed
Estonian
Uralic
Basque
Language Isolate
fao latn
Faroese
Indo-European
lat_
Finnish
Uralic
fra latn glt gla latn
French
Indo-European
Scottish Gaelic
Indo-European
gle latn sig_latn
Irish
Indo-European
Galician
Indo-European
guj_ hat__atn
Gujarati
Indo-European
Haitian Creole
Creole
hau_latn
Hausa
Afro-Asiatic
heb_ hebr
Hebrew
Afro-Asiatic
hin_deva
Hindi
Indo-European
hrv_latn
Croatian
Indo-European
hun
Hungarian
Uralic
aye_armn
Armenian
Indo-European
ibo latn
Igbo
Niger-Congo
ind_ int_
Indonesian
Austronesian
isl latn
Icelandic
Indo-European
ita lat_
Italian
Indo-European
jav_latn
Javanese
Austronesian
jpn_ipan
Japanese
Japonic
knda
Kannada
Dravidian
kat kat_geor
Georgian
Kartvelian
kaz_cyrl
Kazakh
Turkic
khm khmr
Khmer
Austroasiatic
kir cyrll
Kyrgyz
Turkic
kin_latn
Kinyarwanda
Niger-Congo
kor kor_hang
Korean
Koreanic
15

---

Table 2: Languages with their Codes and Families (continued)
Language Code
Language Name
Language Family
lao laoo
Lao
Tai-Kadai
lit latn
Lithuanian
Indo-Buropean
Itz latn
Luxembourgish
Indo-European
mal_mlym
Malayalam
Dravidian
mar_
Marathi
Indo-European
mkd_cyrl
Macedonian
Indo-European
mlt latn
Maltese
Afro-Asiatic
mri_lan
Mãori
Austronesian
mya_mymr
Burmese
Sino-Tibetan
nid_latn nno latn
Dutch
Indo-European
Norwegian Nynorsk
Indo-European
nob latn
Norwegian Bokmål
Indo-European
nya_lam
Nyanja(Chewa) Chichewa)
Niger-Congo
oci latn
Occitan
Indo-European
gan_guru
Punjabi
Indo-European
pes_ arab
Persian
Indo-European
pltlltn
Plateau Malagasy
Austronesian
pol_latn
Polish
Indo-European
por_latn
Portuguese
Indo-European
ron ron_latn
Romanian
Indo-European
rus rus_cyrl
Russian
Indo-European
sin_sinh
Sinhala
Indo-European
slt_latt
Slovak
Indo-European
sly slt_latn
Slovenian
Indo-European
sna latn
Shona
Niger-Congo
snd arab
Sindhi
Indo-uun
som_latn
Somali
Afro-Asiatic
sot_latn
Sotho
Niger-Congo
spa_latn
Spanish
Indo-Furopean
srp_cyrl
Serbian
Indo-European
sun_latn
Sundanese
Austronesian
swe_
Swedish
Indo-European
tam_taml
Tamil
Dravidian
tat_ eat_cyrr
Tatar
Turkic
tel telu
Telugu
Dravidian
tgk_cyri
Tajik
Indo-European
tat_
Tagalog (Filipino)
Austronesian
tha_thai
Thai
Tai-Kadai
tur tur_latn
Turkish
Turkic
uig_arab
Uyghur
Turkic
ukr_cyrll
Ukrainian
Indo-European
urd_arab
Urdu
Indo-European
vit_latn
Vietnamese
Austroasiatic
Jain
Xhosa
Niger-Congo
yor latn
Yoruba
Niger-Conge
zho hans
Chinese (Simplified)
zsm 7sm_latn
Malay
Austronesian
zul rul_latn
Zulu
Niger-Congo
16

---

B Tokenizer Training Details
Tokenizer Data and Language Sample. We create training datasets of 300MB, sourced from Chang et al. (2024). Links to individual source corpora are included hattps://githu com/ hwapcchhanncuureeeeeee
tylerachang/ curse- of -mu multilingualiy We do not redistributed the compiled datasets, as the original source dataset licenses do not permit us to do so. However, all datasets are publicly available.
For each language, we also create a dataset where the dataset size is scaled according to its byte premium (Arnett ct al., 2024). more comparable information content in datasets across languages. There are 98 languages with at least 300MB of BP-scaled data from Chang et al. (2024). However, we exclude Samoan, because we found the training data to be severely contaminated with portions of FLORES (Appendix C) We list all 97 languages included in our experiments in Appendix A For languages with byte premiums over 1 and therefore BP-scaled dataset of over 300MB, the unscaled dataset is subset of the scaled dataset. For languages with a byte premium less than one, this is not the case, as the unscaled dataset is larger than the scaled dataset. While the datasets are not parallel in content, the datasets generally comprised of text from similar genres and domains.
scaling (BP-scaling) the data is intended to achieve
All tokenizers were trained using the CPUs from one server equipped with an NVIDIA RTX A6000.
Tokenizers. For each language and dataset size, we train tokenizers which differ in vocabulary size and tokenizer type. We use seven vocabulary sizes: 16384, 32768, 49152, 65536, 81920, 98304, 1 14688. These vocabulary sizes were chosen to cover the range of frequently used vocabulary sizes (between 32000 and 64000) as well as additional vocabulary sizes above and below this range. The vocabulary sizes constitute an arithmetic progression, where each vocabulary size is 16384 higher than the previous one, with the exception of the largest size, which is double the next smallest size. The smallest vocabulary size we use 16384, because the smaller vocabulary size we tried, 8192, was too small forthe Unigram tokenizers to converge in many languages, All vocabulary sizes are divisible by 128, following Anthony et al. (2024) and Groeneveld et al. (2024), which makes our tokenizers more suitable for training language models in the future. For each vocabulary size, we train both BPE (Gage 1994; Sennrich et al. 2016) and Unigram (Kudo 2018) tokenizers. For each language (n=98%. we train tokenizer for each possible combination of vocabulary size (7), tokenizer type (2), and BP-scaling (2) for a total of 2, 2,744 tokenizers. We release
all tokenizers and training datasets Hugging Face"] Code for training and evaluation is released: htttp:://iit http:/tgtttuuu... wtt./coooo///aaaee wwtherineaanneteeeeeeeeeeeeeeeee htt.t//tttimarrrtte.r...aaaiaaaa wewplininggnnieeeeeeeee....neeee tokenizzer laarrriieeeiiiisssee
C
Contamination Analysis
Our training data are subsets of the training data from Chang et al. (2024). who claim to exclude the FLORES dataset from their training datasets, but there is still the possibility that FLORES could be contaminated inadvertently in the web data sources. To check for any potential contamination of FLORES in the datasets, we tokenize each FLORES sequence and the entire training dataset from Chang et al. (024) for each language that is in both their dataset and FLORES (208 languages, which is a superset of the languages used in the analyses in this paper). We compute the total number of times that the first 10 tokens of any FLORES example appears in the training dataset for the language. For 72% of languages, no FLORES examples appear in the training dataset at all. For 98% of languages, less than 10 FLORES examples appear in the training dataset (out of over 2000 FLORES examples total). The only language with notable FLORES contamination is smo_ man_latn (Samoan; 7155 occurrences of FLORES examples in the training dataset), As it an extreme outlier in the contamination analyses, we do not include it in the analyses in this paper.
D Comparing CTC to Pre-trained Tokenizers
In order to determine that our training data size is not too small, we compare the FLORES CTC of
existing English-centric tokenizers, OLMo (Groeneveld et al. 2024). Pythia (Biderman et al. 2023
'We then exclude Samoan per Appendix C for a total of 97 languages included in our analyses.
17

---

and SmolLM (Allal et al. 2025), all of which have vocabulary sizes of approximately 50000 These are represented by dashed horizontal lines (Figure 5) We compare these the BPE and Unigram tokenizers we trained on English data (solid lines; Fig. 5). We note that our BPE tokenizers in particular have lower token premiums than all three tokenizers at most vocabulary sizes.
BPE
Unigram
OLMo
Pythia
SmolLM
FLORES CTC
20K
40K
60K Vocabulary Size
80K
100K
Figure 5: The solid curves, fit using a power law, represent the measured CTCs for each vocabulary size, plotted as points along the curve. The horizontal dashed lines indicate the CTCs of the OLMo, Pythia, and SmolLM tokenizers.
E BPE vs. Unigram Tokenizers
We ask whether there is direct relationship between byte and token premiums. We hypothesize that if there is, it would only be for Byte-pair Encoding (BPE) tokenizers. Byte premiums capture the observation that some languages need more bytes convey the same amount of information compared to other languages. BPE tokenizers work by first splitting a sequence into individual bytes. The tokenizer learns merge pairs of bytes iteratively. For a given token, the more bytes is composed of, the more merges the tokenizer has to execute order to create that token. For languages with higher byte premiums, for sequence with fixed amount of information, BPE tokenizer will have to execute more merges in order to create tokens with the same amount of content. Therefore, for languages with higher byte premiums, BPE tokenizers will be less likely to learn the correct number of merges, and thus will exhibit stronger token premium effects across monolingual tokenizers trained on the same amount of data for each language.
Unigram, by contrast, during training creates the vocabulary by first adding each unique whitespace- separated word to the vocabulary. It then iteratively removes items until the vocabulary is the desired size. The final vocabulary maximizes the unigram token probability over the training corpus. Therefore, hypothesize that byte premiums should not directly impact Unigram tokenization.
In Section 3.1]
we tested whether byte premium scaling the training data affects CTC. Here, we test the hypothesis about the effect of byte premiums CTC and its interaction with tokenizer type.
Do tokenizers show a relationship between byte premium and token premium? We fit a linear mixed effects model, predicting token premium with a fixed effect of byte premium and random intercepts for both vocabulary size and tokenizer type. We test this only for tokenizers trained unscaled data. We find extremely small, but significant, negative relationship between byte premium and token premium (3 = -0.0418, SE 0.00542, / /(1362) -7.112, < 0.001), such that as byte premium goes up, token premiums go down. This is the opposite of what was predicted by our hypothesis. The effect can be interpreted such that as byte premium goes up by one (which is
OOLMo's vocabuary size is 50279, Pythia's is 50253, and SmolLM's is 49152.
18

---

relatively large difference, equivalent to different in byte premium between English and Greek), the CTC goes down by 3.7%. Due to this extremely small effect size, this result does not support our hypothesis that there is correlation between byte premium.
Is there difference in the relationship between byte and token premiums between BPE and Unigram tokenizers? Our hypothesis predicts that there is stronger (positive) correlation between byte and token premiums for BPE tokenizers than for Unigram tokenizers. We fit a linear mixed effects model predicting token premiums with both byte premium and tokenizer type fixed effects. We also this model so that we test for the interaction between byte premium and tokenizer type. As in the previous linear mixed effects model, fit random intercept for vocabulary size. While find the effect of byte premium to be significant (3=-0.0350, SE=0.00676, 1( ((1553)=551666 p<0..00), there is significant interaction between byte premium and tokenizer type (3=00.016, SE=0.0102, (15533)-11552) p=0.127). We also note that there is no significant difference between token premiums between the BPE and Unigram tokenizers (1-test; ((1553)=-1.528 p=0.127). Therefore, it does not seem to be the case that BPE more sensitive to byte premium effects terms of token premiums.
Comparison of Tokenizers by Vocabulary Size
We compare the Unigram and BPE tokenizers across all vocabulary sizes (Figure O) Unigram shows similar pattern, where increased vocabulary size does not lead to decreased token premiums across languages.
13,0000
100000
100000
90000
90000
Rores CTC
70000
50000
40000
40000
0392
16384
32186
65536
B2820
903i
114688
131072
26344
8192
16/44
32163
e5536
DRJGA
114688
Vocag Size
Vecab Size
Figure Left: BPE CTC distribution by vocabulary size. Right: Unigram CTC distribution by vocabulary size.
G Example FLORES Texts
Below, we include examples from the FLORES dataset for three languages: Scottish Gaelic, Finnish, and English. Each of these sentences are translation equivalents of each other. We observe that languages like Gaelic have higher occurrences of very short words (1-3 characters) and more whites- paces.
H Analyses of Additional Metrics
In Section we test variety language features. Here, we explain what each of the metrics indexes and how the metrics were caleulated [Appendices[H..) and H.2). We also report the correlation metrics for each vocabulary size. We report full results for each metric at each vocabulary size in Appendix
H.1 Factors Impacting Sequence and Word Length
We hypothesize that word length should be correlated with compression related reason to proportion of whitespaces. Languages with longer words are likely to have fewer words, and-for
19

---

Language
Sentence
Scottish Gaelic
Dh'in iniis luchd-saidheans o Sgoil an Leigheis aig Oilthigh Stanford Diluain gun do dh innlich iad inneal diagnosachd ûr tha comasach air ceallan sheorsachadh: sgealb chlö-bhuailte bheag bhiodach as urrainnear clö-bhualadh air clö-bhualadair ince abhaisteach 's a chosgas
seant Aimeireaganach gach tè.
Finnish
Stanfordin yliopiston lääketieteen laitoksen tutkijat ilmoittivat maanantaina uuden
diagnostikkkatýčkaluu keksimisestä: solut tyypin mukaan lajitteleva pienenpieni tulostettava siru, joka voidaan valmistaa normaaleilla mustesuihkutulostmiil.. mahdollisesti noin yhden Yhdysvaltain sentin kappalchintaan.
English
On Monday, scientists from the Stanford University School of Medicine announced the invention of a new diagnostic tool that can sort cells by type: a tiny printable chip that can be manufactured using standard inkjet printers for possibly about one U.S. cent each.
Table 3: Example from FLORES for three languages: Scottish Gaelic, Finnish, and English.
languages that use whitespaces fewer whitespacc-separated words. This, therefore would also be impacted by whitespace pre-tokenization. Therefore, we expect that languages with longer words will have better compression.
Number of Phonemes. As not all languages use whitespaces to indicate word boundaries and the notion of wordhood a fraught concept in linguistic we calculated four metries, which we believe to be related to word length. The first relates to differences in sound systems between languages. There has been an observed relationship between the number of unique sounds (called phoneme inventory) and the average word length across languages. Nettle (1993) found that in small sample of languages. phoneme inventory was predicted to correlate with word length. More recently, Fenk-OOzloo and Pilz (2021) found that phoneme inventory size negatively correlated with word length as measured by number of syllables, but not by number of phonemes.
We calculated phoneme inventory size for each language in our sample using PHOIBLE (Moran et al. 2014), phonological inventory database. We predicted that there would be an inverse relationship between phoneme inventory and word length, and therefore larger inventory size should correspond to lower CTC.
Number of Characters and Character Distribution. Following this logic, we also hypothesized that languages with smaller inventories of characters would have longer words and sequences, because you need longer sequences to get enough unique sequences to represent all the words in a language. The benefit of using this metric over phoneme inventory size is that it does not assume that longer words in number of sounds will correspond directly to sequence length in characters, as writing systems differ wildly in the information they encode. Different writing systems may encode each sound roughly (alphabets), entire syllables (abugidas), just consonants (abjads) (Daniels) 1990), or semantic meaning (logographies, e.g. Chinese characters) (Williams and Bever] [210 Ding et al.) 2004). We hypothesize that languages with more unique characters will have shorter words, as there unique sequences
Following basic combinatorial principles, with larger set of unique characters, you can make more unique sequences (e.g. words) with shorter sequence length. English has 26 basic letters in its writing system. This means there are 26 (676) possible two-letter words. For a language like Telugu, which has approximately 52 characters (Muppalaneni et al. 2019), therefore there are 522 (2,704) possible two-letter words. Chinese has between 50,000 and 100,000 unique characters (The CLI Team 2025), depending on the dictionary. However, the most common 3,500 characters make up over 99% of text seen in daily life (Hutong School| (2012). Therefore, Chinese may have 12,250.000
unique two-character words. And, indeed (Sun et al. 2018) 70% of the words in the Chinese Lexical Database (Sun et al. 2018) are two-character words.
See Haspelmath 2017 for an overview.
20

---

We calculate the number of character unigrams (i.e. unique characters) and character bigram entropy. If language has fewer unique characters. then expect longer words according to the combinatories described above. Therefore, we predict that the more unique characters language has, the shorter its words will be. If language has fewer unique characters, then we expect higher bigram entropy. because each character should be used in more unique contexts (bigrams) High bigram entropy, therefore, should be correlated with fewer unique characters and longer words. We calculate both metrics over the training corpus for our tokenizers for each language.
Length Ratios. We also take a measure of sequence length from the Byte Premium Tool (Arntt et al., 2024). One of the metries released is the "length ratio", which is the ratio of number of characters needed to represent a text in one language relative to the number of characters needed to express parallel text in English. (also called called the "length ratio" (Arnett et al. 2024h). Higher length ratios corresponds to longer sequences.
Number of phonemes was the only metric relating to sequence length that did not significantly predict FLORES CTC. This is likely due the arbitrary relationship between aspects spoken languages and written language.
Length ratio is the most direct measurement of sequence length and explained the most variance. We tested whether unigram entropy or bigram entropy explain additional variance on top of length ratio. We fit reduced linear mixed effects model with just length ratio size as a random intercept. We then fit two full models: one with length ratio and unigram entropy as fixed effects and one with length ratio and bigram entropy as fixed effects. We compare model fit with an ANOVA. Each unigram entropy and bigram entropy improve model fit (unigram entropy: (1) 9.830, p = 0.002; bigram entropy: x (1) 22.613, p < 0.001). Furthermore, model with length ratio, unigram entropy, and bigram entropy is even better than just one with length ratio and unigram entropy (x (1) 82.736, p < 0.001). general, these effects are small, however.
fixed effect, and vocabulary
H.2 Script
We also wanted to know what the influence of script was on CTC. As mentioned at the end of Section different writing systems encode different things. For instance, abjads are writing systems which represent each consonant with a symbol (Daniels) 1990), but vowels are often not represented. The scripts used for Arabic and Hebrew are examples languages that use abjads.
Flores CTC
Latin
Arabic
Cyrillic
Other
Script
Figure Boxplot showing CTCs across languages according to their ISO 15924 code, which indicates script type.
We first looked at scripts according to their ISO 15924 codes (Figure 7 Under this classification system, scripts using modified version Latin script are classified as 'Latin', The vast majority of the languages in sample use Latin script. The only other scripts used by at least than three
21

---

languages were Arabic and Cyrillic. After grouping the languages using remaining scripts into an 'other' category, we found that script explains some variance (R²= 0. 109 for the tokenizers with vocabulary size of 65536). We report pairwise in V Latin script languages have significantly higher CTCs languages using Cyrillie or 'other' scripts. We believe this is likely another way that we can see the effect of the number of whitespace.
Script 1
Script 2 p-value
Adjusted p-value
Significant
Arabic
Cyrillic
0.3560
2.1359
False
Arabic
Latin
0.0370
0.2222
False
Arabic
Other
0.0172
0.1033
False
Cyrillic
Latin
< 0.000
< 0.001
True
Cyrillic
Other
0.0319
0.1912
False
Latin
Other
<.0001
< 0.001
True
Table 4: Pairwise Mann-Whitnee U test results with Bonferroni correction. Significant comparisons (adjusted < 0.05) are highlighted in bold.
Another possible way that we might see the effect of script with the bytes-per-character (or, byte coefficient) metric. We measure this using the byte coefficients released as part of the Byte Premium Tool (Amett et al. 2024). This metric, however, does not show any significant effects.
H.3 Results for All Predictors at All Vocab Sizes
Tables 5 and 7 report the results of individual linear correlations between these factors and FLORES CTC, as we did Table Some of these predictors are colinear and explain some of the same variance. We report these results it is easier interpret the amount of variance explained by each the predictors.
Vocab Size
16384
32768
49152
Predictor
p-value
R²
p-value
R²
p-value
R
n phonemes wwitespace
0.543
0.004
0.601
0.003
0.747
0.001
0.180
0.019
0.002
0.094
0.000
0.132
unigrams_unigg
0.000
0.130
0.265
0.013
0.107
0.027
bigrams-entropy nospace
entropy_nospace
0.011
0.066
0.043
0.042
0.010
0.067
unigrams ennropy_nnosteeeee nospace
0.001
0.107
0.105
0.027
0.031
0.048
char char_coef coef
0.825
0.001
0.004
0.085
0.003
0.089
byte_coef
0.126
0.024
0.125
0.025
0.107
0.027
byte_premium
0.126
0.024
0.125
0.025
0.107
0.027
vocab jocab_mean__o token len
0.000
0.270
0.128
0.024
0.185
0.018
flree_mean_ttoke wwwmmmanntttt Leaa__token_eee
0.000
0.403
0.000
0.137
0.000
0.150
dima_sim
0.000
0.315
0.000
0.242
0.000
0.193
script
0.683
0.042
0.953
0.128
0.828
0.138
Table 5: Significance is indicated with bolding.
22

---

Vocab Size
65536
81920
98304
Predictor
p-value
R
p-value
R²
p-value
R
p.honemes
0.843
0.000
0.906
0.000
0.951
0.000
proportion_whitespaaee whitespace
0.000
0.157
0.000
0.175
0.000
0.189
digramssunnquu
0.072
0.034
0.051
0.040
0.040
0.044
dggramseentrop__nne
0.005
0.079
0.003
0.089
0.002
0.096
lnggrams_entroyy_nnne
0.017
0.058
0.010
0.067
0.008
0.073
char_coef
0.003
0.088
0.003
0.088
0.003
0.087
syte_coct.
0.090
0.030
0.080
0.032
0.073
0.033
byte_ pyte__remmuum
0.090
0.030
0.080
0.032
0.073
0.033
vocab mean haa__mean__tttee tokeettoeeeeeeeeeoe len
0.166
0.020
0.145
0.022
0.122
0.025
flores_mean mean thean_token__e len
0.000
0.168
0.000
0.182
0.000
0.195
data_sim
0.000
0.239
0.000
0.296
0.000
0.325
script
0.692
0.144
0.612
0.148
0.559
0.151
Table 6: Significance is indicated with bolding.
Vocab Size
1 114688
131072
262144
Predictor
p-value
R²
p-value
R²
p-value
R
n phonemes
0.989
0.000
0.981
0.000
0.871
0.000
0.000
0.199
0.000
0.207
0.000
0.247
unigrams_unine
0.034
0.046
0.030
0.049
0.015
0.061
bigrams-entropy nospace
entropy_nospace
0.002
0.100
0.001
0.104
0.000
0.123
unigrams entropy nospace
nospace
0.006
0.077
0.005
0.081
0.002
0.098
char_coct
0.004
0.085
0.004
0.084
0.005
0.079
byte_coef
0.066
0.035
0.061
0.036
0.043
0.043
byte_premium
0.066
0.035
0.061
0.036
0.043
0.043
vocab _aab___en____ ____amnaatttennnn len
0.102
0.028
0.088
0.030
0.031
0.048
flores mean flores_mean_teeee
@eean_token_ler
0.000
0.206
0.000
0.215
0.000
0.253
data_sim script
0.000
0.347
0.000
0.426
0.000
0.442
0.522
0.154
0.497
0.156
0.414
0.164
Table 7: Significance is indicated with bolding.
I
Parallel Data
L1
Plots
Figure 8 shows the CTCs for all seven outlier languages (Cebuano, Khmer, Luxembourgish, Mandarin Chinese, Maltese, Japanese, and Burmese) for tokenizers trained on parallel data.
1.2
Statistical Tests for Parallel-Data Tokenizers
In Table 8 we report the results from the r-tests between the FLORES CTCs of the parallel and non-parallel tokenizers, by vocabulary size. None the differences are significant.
23

---

PLO1ESCTC
Catuant
Trokeh Irgaen (parale)
(tarste
FLSRTS CTC
CTC
Vocas See
Weulb Sie
Qmee
Ergiin Trotes Jprns
Eng
Omeer (aarate)
Lusembouggen (parsle))
Ing in (tersle)
FLOGSS enC
Vocah
Erglish
FLONES eiC
F.OMES CTC
Jaratese
Engith (oardiid)
Burmese
Engles English
Japansse (parolic)
Burmese (parofict
Figure 8: Cebuano, Khmer, Luxembourgish, Mandarin Chinese, Maltese, Japanese, Burmese
J Optimal-Vocabulayy BPE Tokenizers
Here we describe our method estimating optimal vocabularies, provide detail on training tokenizers with optimal vocabularies, and evaluate our estimation procedure.

---

Vocab Size
1-Statistic
p-value
Mean Difference
16384
-2.075
0.054
-5509
32768
-1.862
0.081
-787
49152
-1.232
0.236
-395
65536
-0.777
0.449
-225
81920
-0.438
0.667
-116
98304
-0.338
0.739
-85
1 114688
-0.195
0,848
-48
131072
-0.034
0.973
-8
262144
0.650
0.525
165
Table 8: j-test results and mean difference between parallel and non-parallel tokenizer CTCs for the seven languages in the sample. Significance is indicated with bolding.
J.1 Estimating Optimal Vocabularies
In order to estimate the vocabulary size at which tokenizer for given language will reach given CTC, we use the CTCs for the existing tokenizers and fit a power law curve, e.g. Figure
FLORES CTC
Vocab Size
Figure 9: Power law relationship fit to the CTC values for the BP-unscaled BPE tokenizer for English.
The goal is to find the vocabulary size for each language at which the tokenizers are predicted have the same CTC. For example. Figure IO (left) shows the curves for English (eng_latn) Georgian (kat_geor)) and Burmese maya_mymr). There exists a vocabulary size for each language such that we predict tokenizer would achieve FLORES CTC of 52000 (indicated with the horizontal dashed line). However, this is not the case for all languages, e.g. Figure (right). There is no vocabulary at which Telugu (tel_ter telu), Khmer (khm khmr), and German (deu latn) can achieve the same CTC.
Instead, for range of CTCs (from 50000 to 74000 in increments of 1000) and estimated the vocabulary size at which every language would achieve the closest CTC to that target CTC. If our curve didn't 't predict tokenizer could ever reach target CTC, used the CTC from the closest existing tokenizer. If the target CTC was below what we predicted could be achieved for given language, set the vocabulary size to the largest size we tested (262 144) and set the predicted CTC to be the observed CTC at that size. the prediction was higher than the worst CTC we observed, we set the vocab size to the lowest vocabulary size tested (8192).
In Figure we show the distribution of predicted CTCs for every target CTC. the target CTC of 69000 there is the least amount of variance. This means if you wanted to select vocabulary size at which most languages had a CTC closest together would be approximately 69000. target CTC gets lower, there are more languages which cannot achieve CTCs low the target and "get stuck"
25

---

eng_latn Kal_geor
tol_tell talu
Khhnr
deu, latn
FLORES 52000
FLORES CTC
FLORES
Vocab Size
Vorab Size
Figure 10: Left: CTCs across vocabulary sizes for English, Georgian, and Burmese. Horizontal black dashed line indicates FLORES CTC of 52000. Intersections with the black line correspond the "optimal" vocabulary size for each language. Right: CTCs for three languages, which do not have any vocabulary size which all three languages can achieve the same CTC.
at much higher CTC. For BPE tokenizers, there is limit to the level of compression that can be achieved. This is reflected in the asymptote we see in the power law curves for each language.
75000
Estirated CTC
70000
11
8
8
09 0000.
60R
00 Loot
00 0000
00
00 0
60000
000 0
00
80
000
(400
100
00
90
00
00
8
0000
800
45000
SHor0
StoNg
52000
$3000
54000
550000
SHopp
Stegg
Gloug
Grokg
edpig
eeoup
7200g
7,000
74000
Drgee
Figure 11: Each box plot shows the distribution of estimated CTCs for given target CTC. In many cases, the distribution is so compressed that there is simply a gold line where the mean CTC is, and no box is visible.
J.2 Tokenizer Training
For each target CTC, trained the tokenizer with the vocabulary size that estimated. If the vocabulary size was estimated at 8192 or 262144, we did not train new tokenizer, and instead used the existing one with that vocabulary size. In total there are 2055 aptimal-vocabul tokenizers, 25 for each language. We used the same BPE implementation to train the tokenizers and trained them on the same training datasets.
JJ3 Estimated Vocab Validation
Next, we test how accurate our CTC estimation method was. We plot the distribution of CTCs for the estimated optimal-vocabulary tokenizers and actual optimal-vocabulary tokenizers in Figure 12 with the sace-locabulary tokenizers as reference. The estimated and actual vocabll-vocabulayy tokenizers have much less variance in their CTCs. The estimated CTCs are not perfectly accurate, but shows greatly reduced token premium effects compared to the same-vocabulary tokenizers.
26

---

Optimal (Estimated) Vocab
Optimal (Actual) Vocab
Same Vocab
Density (log scale)
FLORES
Figure 12: Density plot of same-vocab tokenizer CTCs, estimated optimal-vocab CTCs and actual optimal-vocab CTCs across languages. Y-axis is log- log-scaled to improve readability.
Overall, the RMSE between the actual and predicted values was 3733. We plot the distribution of CTCs across all languages in Figure 12 In Table 9 we report the RMSEs for each target CTC. As target CTC goes up, the RMSE goes down until CTC 59000, then it increases monotonically.
Target CTC
RMSE
Target CTC
RMSE
50000
3536
60000
2913
51000
3185
61000
2992
52000
2975
62000
3059
53000
2938
63000
3144
54000
2982
64000
3283
55000
2933
65000
3445
56000
2921
66000
3563
57000
2880
67000
3758
58000
2870
68000
3981
59000
2870
69000
4384
70000
4601
71000
4892
72000
5083
73000
5350
74000
5651
Table 9: RMSE for estimated CTC method for each target CTC. RMSE decreases target CTC
increases on the left-hand column. Then, after 59000, RMSE increases as target CTC increases.
27

---

K SuperBPE Training
SuperBPE training works in two phases. In the first phase, the tokenizer trains like normal BPE tokenizer. In the second half, the tokenizer learns merges over whitespace boundaries. In the first phase, the tokenizer learns a vocabulary of the desired size. Therefore, to learn superword tokens, you determine the number of tokens you wish to keep from the first phase, remove the rest, and then fill up the rest the vocabulary the second phase. The recommendation by the authors is to keep 180000 tokens from the first phase of training out of a total vocabulary size of 200000. The number of tokens retained from the first phase is called the transition point. In the original implementation of SuperBPE, they do not manipulate vocabulary size to the extent do here. Therefore, in order to scale this appropriately to different vocabulary sizes, we set the transition point as proportion the full vocabulary size. Since the original recommendation was 90% of the full vocabulary, this was the highest transition point we tested. We also trained tokenizers with more aggressive transition points (see Section K.1 below), but in the paper all reported results are for tokenizers with transition point of 0.9. We tested five different transition points: 0.5, 0.6, 0.7, 0,88, and 0.9.
We trained SuperBPE tokenizers for all languages for all the vocabulary sizes we used before over 64000 (six sizes). In total, we train up to 2940 SuperBPE tokenizers. There were come cases where the training failed and additionally, found some cases where there was apparent error with byte fallback, which we describe in the following section.
The decrease in token premium effects is visualized in Figure 13 For random sample of ten languages there is no vocabulary for each language such that a BPE tokenizer can achieve even approximately the same CTC. For SuperBPE. on the other hand, it is possible to achieve almost exactly the same CTC for the ten languages by training a tokenizer with each languages' optimal vocabulary size.
BPE Tokenizer
SUPERBPE Tokenizer
65000
Languages
urd_arab
60000
kaz, cyrl
FLORES CTC
FLORES CTC
55000
mit Jatn
50000
bel_cyrl
Ibo latn
45000
mkdd cyrl
kir cyrc
40000
eng_latn
100000
150000
200000
250000
100000
150000
200000
250000
Vorabulary Size
Vecabulory Size
Figure 13: CTC curves for a random sample of languages for BPE and SuperBPE. This includes only tokenizers with vocabulary sizes greater than 64000.
K.1 Transition Point Analysis
Here report the distribution of CTCs by transition point. None of the transition points show greatly reduced token premium effects, however as transition point decreases, i.e. as the number of tokens retained from the first phase decreases, the compression trends downward, Figure 14 shows the distribution for each transition point.
L Additional Super Statistical Tests
We conduct an F-test to test whether there a difference in the variance between the BPE and SuperBPE tokenizers. We also conduct a paired t-test to test whether the CTC is different between the two tokenizer types. All results are reported in Table TO For all vocabulary sizes, SuperBPE tokenizers have less variance in CTC and have lower CTCs.
28

---

65000
$5000
30000
69536
B1920
11468B
131072
262166
SuperBPE.
SuperSFE, SuperBPE.
SuperBRE,
Distribution of CTCs by vocabulary size and transition point. These results do not include the outlier languages.
Figure 14:
F-test
t-test
Vocab Size
F
DF1
DF2
p-value
p-value
65536
2.187
73
96
0.001
-2.42
0,016
81920
2.006
70
96
0.002
-2.67
0.008
98304
1.660
71
96
0.021
-3.47
0.001
1 114688
1.607
67
96
0.033
-3.36
< 0.001
131072
1.404
73
96
0.119
-4.32
0.001
262144
0.745
65
96
0.207
-7.28
< 0.001
Table 10: F-test and t-test results for SuperBPE tokenizers vocabulary size.
Here report the significance (p-value) and variance explained (R²) for linear regression predicting CTC from proportion of whitespaces for SuperBPE tokenizers. The relationship is not significant for any vocabulary size.
Vocab Size
R
Adj. Ri
p-value
65536
0.057
0.042
0.057
81920
0.086
0.070
0.023
98304
0.069
0.053
0.039
114688
0.079
0.062
0.034
131072
0.082
0.066
0.023
262144
0.111
0.094
0.012
Table 11: R² and p-value for regressions predicting CTC based on proportion of whitespaces for SuperBPE tokenizer with threshold of 0.9.
M Optimal-Vocabulan SuperBPE Tokenizers
We use the same method to estimate CTCs for SuperBPE tokenizers we did for BPE tokenizers (Appendix П). We plot distribution of estimated CTCs for target CTCs (Figure The target CTCs at which there is near-perfect achievement of the same CTC across languages are much lower for SPP tokenizers (48000 to 57000) than for BPE (69000). And there larger range of CTCs for which it is possible to get tokenizer for every language to achieve.
29

---

by
Languages
6300
Estratted
600
800
60000
0000
0
0
00
50000
larget HOREES CTE
Figure Distribution of estimated CTCs for each target CTC for SuperBPE, tokenizers.
N
Remaining Variance
We fit a linear mixed effects model with unigram entropy, bigram entropy, length ratio (character coefficient), and bytes-per-character ratio (byte coefficient) predictors. We included vocabulary size as random intercept. The results of the ANOVA are reported in Table
Predictor
SS
MS df (num)
df (den)
F
p-value
Unigram entropy Bigram entropy
10,084,887
10,084,887
356.05
0.68
.411
1,031,215
1,031,215
356.06
0.07
.793
Length ratio
185,829,300
185,829,300
356.00
12.48
<.001
Byte coef
68,017,284
68,017,284
356.01
4.57
.033
Table 12: ANOVA results for each of the fixed effects
30

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We train and release all the tokenizers mentioned in the paper. We show evidence for token premium effects in Figure LL we identify relevant factors in Section and finally provide interventions which reduce token premium effects in Section 5
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss limitations in Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
31

---

Answer:
[NA]
Justification:
Our results are empirical and any claims made are grounded in those results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We release all training and evaluation code. The datasets used to train the tokenizers are publicly available (see Appendix [B). We use openly available evaluation data. We also release the tokenizers themselves. We release all result datasets, analysis code, and code used to generate the figures in the paper. This allows any and all parts of the paper to be reproduced.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
32

---

reproducible, they 5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification: Code and data are released on OSF. Datasets and tokenizers are released on Hugging Face.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/nips. wwtpcc/pcccccpcccccooooooop hutee/ggguudeeeoeeeuuuubiiiiiiii semeisssinnpp ssinnolicyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Experimental details, e.g. vocabulary sizes, exact tokenizer implementations, are provided in the main body of the paper and/or in the appendices.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Experiments are supported with appropriate statistical tests, which are reported either in the main body of the paper or the appendices.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
33

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
We describe our CPU resources in Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: To the best of our knowledge, the work in this paper conforms the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: This paper aims to contribute to a solution to inequities across languages that are the result of tokenization, which we address in the Introduction and Discussion sections. Therefore, we hope our work has positive societal impacts. We do not believe there are any possible negative societal impacts of the work.
34

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: In Appendix B we discuss that our reason for not releasing our training data is in order to adhere to the permissions from the original data licenses.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes, see Appendix B
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
35

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssippsssii thcode cwwc datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We describe in detail the process for training the tokenizers release. The tokenizers will be released on Hugging Face and will have their own model card, which will re-iterate critical details about the tokenizers, how they were trained, and how to use them.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We did not conduct any human subjects research in this paper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: We did not do any human subjects research, therefore we did not need IRB approval.
Guidelines:
36

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
37


Reviews:
[{"review_id": "9IncmRaDAN", "reviewer": "Reviewer_TBJw", "summary": "This paper investigates token premiums, the extra tokens required to encode text in different languages. By training ~7k tokenizers across 98 languages, the authors find that vocabulary size and pre-tokenization drive these differences. They also investigate Superword tokenizers and find that they reduce token premiums and improve compression.", "questions": "I do not have any questions. The weaknesses are mostly inherent to the topic and its significance. I scored 3 for clarity because the paper is clear enough; however, I think the significance is limited (score 1). The originality is good (score: 3) but not excellent, as the CTC concept has been previously introduced and explored. Overall, the quality is good (score 3).\n\n \nSuggestions:\n- The main suggestions are included in the weaknesses section. You could solve them. OR perhaps if i misunderstood inform me of what do you think is right. These rest of suggestions here are just extra and do not impact your score.\n- In Figure 1 (page 2), the term CTC is introduced without explanation, and only later (on page 3) is it clarified it stands for. Add the acronym in both the text and figure caption for better readability.\n- You could try using more diverse parallel datasets beyond FLORES. I think most paper just use FLORES for token premiums. But for example, UDHR (as shown in Figure 3 of https://arxiv.org/abs/2309.13320) covers additional scripts not present in FLORES. You might also find useful parallels in the [MTEB multilingual bitext mining task](https://github.com/embeddings-benchmark/mteb/tree/main/mteb/tasks/BitextMining/multilingual). I suggest this because your paper's main contribution is extensive comparison; adding this dimension could further deepen your comparisons.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 1, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- To the best of my knowledge, this is the first large-scale systematic study of token premiums across nearly 7k monolingual tokenizers covering 98 languages.\n- The paper offers actionable insights, showing that factors such as vocabulary size, whitespace-agnostic tokenization (SuperBPE), and pre-tokenization strategies can significantly influence token premiums across languages.\n\nWeaknesses:\n- This is not necessarily a weakness of this paper, but a limitation of the broader research area: token premiums may not be highly important in practice. At inference time, users tend to use high-resource languages, so the token inefficiency in low-resource languages—while unfair—may have limited real-world impact. The paper does not account for this.\n- The main metric used is CTC, and the evaluation is based on the FLORES dataset, which is parallel across languages. Both the metric and dataset have already been extensively used in prior work—Ahia et al. (2023) and Petrov et al. (2023)—which inspired this study. As a result, this paer is not new and the strength is only about the high number of the comparisons.\n- The study focuses only on tokenizer experiments, much like other tokenizer studies it does not  evaluate whether these improved tokenization settings actually lead to better model performance. This limits the practical impact of the findings. It's possible that an \"optimal\" tokenizer in terms of paper performs worse during downstream tasks after finishing the model training, but this is never tested.\n- The paper is multilingual and comparative, but not a \"crosslingual\" paper per se; the term \"crosslingual\" appear 4 times: one in title: one in abstract; one in conclusion. I'm pretty sure the paper title needs a change.", "was_revised": true, "final_justification": "I will keep my positive score the same as before.", "rebuttal": ">a limitation of the broader research area: token premiums may not be highly important in practice. At inference time, users tend to use high-resource languages, so the token inefficiency in low-resource languages—while unfair—may have limited real-world impact. \n\nOne of the goals of this work is to work towards greater crosslingual equity for lower-resource languages (which by definition impacts fewer users), and the impact of this work is significant for members of those language communities. In this paper, we consider only monolingual tokenizers. If we were to apply these findings to language models, we would expect that these models would also be monolingual, and therefore at inference time, only the efficiency of the target language should be considered. However, in the future, we hope to adapt these findings to multilingual contexts, in which case we would want to balance the efficiency of several languages.\n\n>The main metric used is CTC [...] Both the metric and dataset have already been extensively used in prior work [...] As a result, this paper is not new and the strength is only about the high number of the comparisons.\n\nWe use the same method as Ahia et al. (2023) and Petrov et al. (2023), however in those papers, the authors study existing multilingual tokenizers. Neither of those papers studies monolingual tokenizers, nor do they train their own tokenizers or manipulate factors such as vocabulary size. The contribution of this paper is demonstrating that identical training in a monolingual context still leads to differences across languages, which to our knowledge has not been shown before. Understanding how to mitigate these differences is an essential step in reducing inequities in multilingual contexts.\n\n>The study focuses only on tokenizer experiments, much like other tokenizer studies it does not evaluate whether these improved tokenization settings actually lead to better model performance. This limits the practical impact of the findings. \n\nWe agree that it would be potentially valuable to train models for each of the tokenizers to understand how differences in tokenizers impact performance. We do not currently have the resources to train at least 7,000 models. The contribution of the paper is to understand the relationship between language features and intrinsic features of tokenizers, which before now has not been demonstrated. Future work could train a small number of models to test whether using SuperBPE tokenizers with optimal vocabulary sizes leads to better performance, but this is outside the scope of the research questions set out in this paper.\n\n> The paper is multilingual and comparative, but not a \"crosslingual\" paper per se\n\nIn the paper, we only train and evaluate monolingual tokenizers, therefore the paper does not address multilingual tokenization (i.e. a single tokenizer trained on multiple languages). We use crosslingual to refer to comparisons across languages in monolingual settings. We will clarify our usage of this terminology in the updated version.\n\n>the term CTC is introduced without explanation\n\nThank you for highlighting this, we will address this in the updated version!\n\n>You could try using more diverse parallel datasets beyond FLORES. \n\nWe thank the reviewer for the suggestion. Arnett et al. [1] compared FLORES and UDHR (and also the Bible) as parallel corpora and found that UDHR was both smaller and noisier than FLORES (because, for example, there are many domain-specific and borrowed terms). But it is possible that UDHR could be good to investigate for future expansions of this work. We will include discussion on this point in the updated manuscript.\n\n[1] Arnett, C., Chang, T. A., & Bergen, B. (2024). A Bit of a Problem: Measurement Disparities in Dataset Sizes across Languages. In Proceedings of the 3rd Annual Meeting of the Special Interest Group on Under-resourced Languages @ LREC-COLING 2024 (pp. 1-9)."}, {"review_id": "iDjh4ociyG", "reviewer": "Reviewer_SFsu", "summary": "This paper presents a large-scale empirical investigation of \"token premiums\", the phenomenon where encoding the same content requires a different number of tokens depending on the language. The authors systematically study this issue in a monolingual context to isolate it from the confounding factor of data imbalance in multilingual tokenizers. By training nearly 7,000 tokenizers across 98 languages while controlling for vocabulary size, dataset size, and tokenizer algorithm (BPE, Unigram), they demonstrate that token premium inequities persist even under these controlled conditions.", "questions": "- Why not taking the \"best\" and \"worst\" (and maybe some in between) tokenizers based on your metrics for a few languages which had tasks available and train some MLM or small LLM and perform downstream tasks that allow to get an idea of the performances gap, between what we can consider a \"good\" or \"bad\" tokenizer after reading this paper ? Languages will not be comparable between each others, but at least we will get an idea of the impact of the tokenizer optimality.", "limitations": "- The paper compellingly demonstrates how to achieve tokenization equity in a monolingual context. The natural extension is to consider the common scenario of a single, unified tokenizer for multilingual text, code, and mathematical formulas. Do the authors have a view on how their findings might translate to this challenge? Given that any unified tokenizer is arguably sub-optimal by design, have the authors considered whether a \"mixture-of-tokenizers\" might be a more promising direction? For instance, a system could dynamically route inputs to different language or domain-specific tokenizers, building directly on this paper's findings to mitigate premiums in complex, mixed-data settings.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The scale and systematic nature of this study are its greatest strengths. Training ~7,000 tokenizers across 98 languages is a massive undertaking that provides robust empirical evidence. To my knowledge, this is the first study to systematically disentangle token premiums from multilingual training effects by focusing on a controlled monolingual setup.\n- Benchmarking against established, publicly-available tokenizers provided a valuable real-world baseline for the compression metrics.\n- The findings have immediate practical implications for anyone building multilingual models, offering clear strategies (SuperBPE, optimal vocabulary sizing) to build fairer and more cost-effective systems.\n- The paper is well-written and structured. It logically moves from establishing the problem (Section 1-3), to investigating its causes (Section 4), and finally to testing solutions (Section 5). The figures, particularly the density plots, are effective at visualizing the core problem (the spread of CTCs) and the success of the proposed interventions.\n\nWeaknesses:\n- My primary concern, is the overlap between the training and evaluation data. The paper uses FLORES-200 for evaluation as mentioned L. 128, but this same dataset is listed as a training source in the accompanying repository for the data collection used tylerachang/curse-of-multi-linguality on GitHub (L. 827). This data leakage means the tokenizers were evaluated on familiar data, likely leading to an overestimation of their true compression performance on unseen text. This issue significantly weakens the paper’s empirical claims. The authors must address this by either demonstrating that the specific FLORES-200 splits used for evaluation were properly excluded from training or by re-evaluating their trained tokenizers on a completely novel, held-out dataset.\n- The recommended \"best\" solution, SuperBPE, fails to train for 14 languages (L. 257), particularly those using South and Southeast Asian scripts. This limits the universality of the proposed solution, even if the results were reliable.", "was_revised": true, "final_justification": "The discussion with the authors clarified some aspects of the paper.", "rebuttal": ">My primary concern, is the overlap between the training and evaluation data. \n\nWe thank the reviewer for highlighting this point. We removed FLORES from the training datasets before training the tokenizers, and we will note this in the updated paper. Additionally, since submitting the manuscript, we conducted a contamination analysis and found that for every language except Samoan, there was a maximum of 1 item from FLORES in our training dataset. Samoan was significantly contaminated. We will address the contamination and re-train the Samoan tokenizers, and update the results. \n\n>The recommended \"best\" solution, SuperBPE, fails to train for 14 languages (L. 257), particularly those using South and Southeast Asian scripts. This limits the universality of the proposed solution, even if the results were reliable.\n\nSince submitting the paper, the authors of the SuperBPE paper have helped us identify and address the solution. In fact, the issue is related to the decoder, which can be fixed without re-training. We will include the previously excluded languages in the analysis in the updated manuscript. \n\n>Why not taking the \"best\" and \"worst\" (and maybe some in between) tokenizers based on your metrics for a few languages which had tasks available and train some MLM or small LLM and perform downstream tasks that allow to get an idea of the performances gap, between what we can consider a \"good\" or \"bad\" tokenizer after reading this paper ? Languages will not be comparable between each others, but at least we will get an idea of the impact of the tokenizer optimality.\n\nIn this paper, we focus only on intrinsic tokenizer evaluation. We agree with the reviewers that downstream evaluations would be informative, and think this is a valuable direction for future work.  \n\n>The paper compellingly demonstrates how to achieve tokenization equity in a monolingual context. The natural extension is to consider the common scenario of a single, unified tokenizer for multilingual text, code, and mathematical formulas. Do the authors have a view on how their findings might translate to this challenge? Given that any unified tokenizer is arguably sub-optimal by design, have the authors considered whether a \"mixture-of-tokenizers\" might be a more promising direction? For instance, a system could dynamically route inputs to different language or domain-specific tokenizers, building directly on this paper's findings to mitigate premiums in complex, mixed-data settings.\n\nIndeed, we hope to extend these findings to multilingual settings. One way in which the findings from the monolingual settings inform work on multilingual settings is that they predict that a multilingual tokenizer should dedicate different proportions of the vocabulary to different languages in order to achieve balanced compression. However, other factors such as script overlap and linguistic similarity will also likely play important roles. We hope to conduct experiments on this in the near future and create more optimal multilingual tokenizers. We thank the reviewer for these suggestions!"}, {"review_id": "3m5Y74t0dK", "reviewer": "Reviewer_ehmG", "summary": "The paper studies how tokenization unfairness arises. It trains several tokenizers (BPE, Unigram, and SuperBPE) and uses corpus token count (CTC) to measure the factors that impact the “premium” cost paid by under-represented languages by LLMs. The identified vocabulary size and the pre-tokenizer are the most impactful factors to reduce unfairness.", "questions": "More like a remark: I personally suspect that the experiments on FLORES do not allow for capturing how tokenizers work in real case scenarios. In other words, the unfairness emerges as LLMs use tokenizers mostly trained to work with English on other high-coverage languages (e.g., in a BPE multi-lingual tokenizer, most merges are for English, while other languages remain closer to character-level tokenization). \n\nDo you think that, for a language with high CTC, if we could train a tokenizer on the same amount of data as the English counterpart, we would achieve a significant reduction in the premium? This seems to me an experiment one can try with an LLM mostly trained on a massive non-English corpus (e.g., Chinese) and check if the premium for English is high. \n\nLine 294, “Perhaps too much compression could be giving the model effectively less time to think, by dedicating fewer FLOPS to predicting 295 each next token.”. This does not seem reasonable to me. How does this hypothesis explain why English LLMs are better than (mostly) any other language? If a token appears so frequently that it is merged, it probably means that its “meaning” can be compressed and that the information in the sentence lies somewhere else.", "limitations": "The paper addresses a relevant problem and conducts the experiments well. I do not see any limitation, given what they claim as a contribution.\n\nOne limitation is that FLORES-200 is not a very large dataset (it contains 3K sentences per-language) and may still be biased (though less than the previous version) towards specific high-resource languages such as English, Spanish, etc.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\nTokenization unfairness is a problem that was recently identified as relevant by the LLMs research community. While most of the literature, including the paper that identifies the issues, does not introduce mitigations, this paper addresses them.\n\nFig. 1 and the relative sentence, “Monolingual tokenizers trained with exactly the same implementation, dataset size, and vocabulary size demonstrate widely variable token premium effects.” This is an interesting finding that contradicts one of the hypothesis in (Petrov et al 2023) that unfairness may disappear with larger multi-lingual training corpora.\n\nLine 245: “Tokenizers with optimal vocabulary sizes per language have reduced token premium effects compared to tokenizers with the same vocabulary size across languages”. I personally find this to be the best contribution of the paper, as it is a step towards mitigating tokenizers unfairness.\n\nLine 283: “there is no vocabulary size at which monolingual tokenizers will achieve similar compression across a wide variety of languages”. This seems a trivial observation, but it is not and tells us we cannot rely on the same tokenization technique (and parameters) for any language, if we aim to achieve tokenization fairness.\n\n\n**Weaknesses**\n\nThe FLORES-200 dataset is small (in the sense that it does not contain too many examples per language). On the other hand, I understand there is a scarcity of multi-lingual corpora, so this is a weakness of the current state of the art in multi-lingual NLP and not of your paper (but results may be affected by this limitation).\n\nI am not sure I agree with all the findings the authors describe in the paper. For example, line 303, “Our work shows that some languages need larger vocabularies than others for comparable compression.”. Is that a finding that is correlated with the nature of FLORES, which starts from sentences in English (mostly, and some Spanish and a few other languages, if I remember correctly), and provides translations for them? Would the results change if there were a FLORES whose original sentences are (e.g.,) more prototypical Chinese sentences (e.g., so that the English translation should adapt to them)?", "was_revised": true, "final_justification": "I think this paper has some merits and I have nothing against seeing it at the conference.", "rebuttal": ">The FLORES-200 dataset is small (in the sense that it does not contain too many examples per language). On the other hand, I understand there is a scarcity of multi-lingual corpora, so this is a weakness of the current state of the art in multi-lingual NLP and not of your paper (but results may be affected by this limitation).\n\nWe agree that this is a limitation of multilingual NLP, but we are not aware of any larger multi-parallel datasets for this many languages.\n\n>I am not sure I agree with all the findings the authors describe in the paper. For example, line 303, “Our work shows that some languages need larger vocabularies than others for comparable compression.”. Is that a finding that is correlated with the nature of FLORES, which starts from sentences in English (mostly, and some Spanish and a few other languages, if I remember correctly), and provides translations for them? Would the results change if there were a FLORES whose original sentences are (e.g.,) more prototypical Chinese sentences (e.g., so that the English translation should adapt to them)?\n\nGiven that the compression differences between languages are so extreme in many cases, it seems unlikely that any potential biases introduced by translation can account for such large effects, but we agree that this is an important consideration. \n\n>I personally suspect that the experiments on FLORES do not allow for capturing how tokenizers work in real case scenarios. In other words, the unfairness emerges as LLMs use tokenizers mostly trained to work with English on other high-coverage languages (e.g., in a BPE multi-lingual tokenizer, most merges are for English, while other languages remain closer to character-level tokenization).\n\nThe experiments in this paper will not capture the biases observed in existing multilingual tokenizers, which indeed often focus on English. In addition to the differences we observe in the paper, we also expect interactions between languages (e.g. due to shared vs. different writing systems and linguistic similarity) to play roles. In future work, we hope to better understand these dynamics in a multilingual context.\n\n>Do you think that, for a language with high CTC, if we could train a tokenizer on the same amount of data as the English counterpart, we would achieve a significant reduction in the premium? This seems to me an experiment one can try with an LLM mostly trained on a massive non-English corpus (e.g., Chinese) and check if the premium for English is high.\n\nWith respect to the first part of this question, we compared our tokenizers to ones trained on much more data. The results suggest that tokenizer training data size alone may not lead to lower CTC (see the paragraph starting on L117 and Appendix C). With respect to the second part of this question, we understand the reviewer to be asking about compression in a multilingual setting. In this paper, we focus on the monolingual setting. We observe that tokenizer compression does not generalize or transfer well to new languages, but in future work, we hope to evaluate the interactions between languages in multilingual tokenizers. We would appreciate clarification from the reviewer as to whether we have addressed your questions!\n\n>Line 294, “Perhaps too much compression could be giving the model effectively less time to think, by dedicating fewer FLOPS to predicting 295 each next token.”. This does not seem reasonable to me. How does this hypothesis explain why English LLMs are better than (mostly) any other language? If a token appears so frequently that it is merged, it probably means that its “meaning” can be compressed and that the information in the sentence lies somewhere else.\n\nCompression is not the only factor that explains why language models may perform better in English than for other languages. In most cases, models perform best in English because they have been primarily trained to optimize performance for English. They are trained on mostly English data. English data quality is usually also higher (see [1]). Furthermore, we do not find that English has the best compression in the monolingual setting. Therefore, we would not predict that all else being equal (including dataset quality) in a monolingual setting, that English would necessarily have the best performance. \n\n>One limitation is that FLORES-200 is not a very large dataset (it contains 3K sentences per-language) and may still be biased (though less than the previous version) towards specific high-resource languages such as English, Spanish, etc.\n\nWe agree that this is a concern. This is the motivation for the experiments in Section 5.1, in which we control for domain similarity and data quality. The results suggest that this does not significantly affect the compression results in this paper. \n\n[1] Kreutzer, J., Caswell, I., Wang, L., Wahab, A., Van Esch, D., Ulzii-Orshikh, N., ... & Adeyemi, M. (2022). Quality at a glance: An audit of web-crawled multilingual datasets. Transactions of the Association for Computational Linguistics, 10, 50-72."}, {"review_id": "b4yp2GbOzr", "reviewer": "Reviewer_kKAc", "summary": "This paper examines the phenomenon of cross-lingual tokenization inequalities, i.e., the observation made in prior work that tokenizers generate significantly longer token sequences for some languages compared to others. To investigate this, the authors train a large number of monolingual tokenizers, controlling for various parameters (e.g., tokenization algorithm, vocabulary size), and analyze their properties. They find that the similarity between tokenizer training and test data does not impact tokenization inequalities, but vocabulary size and pre-tokenization do, thus pointing to potential mitigation strategies.", "questions": "- Do your findings translate to better performance in terms of language modeling and downstream tasks?\n- Can you provide qualitative analyses to better explain your quantitative findings?", "limitations": "The authors do not discuss the potential negative societal impact of their work. In terms of general limitations, the authors should mention that they do not analyze language model performance, which in my view is a key limitation if this paper.", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The paper provides a useful extension of prior work on cross-lingual tokenization inequalities. The finding that these disparities are not (solely) caused by the training data is interesting, as are the other analyses provided by the authors (e.g., that different languages need to be allocated different vocabulary sizes to achieve comparable compression). Furthermore, the 7,000 tokenizers trained and released by the authors will benefit future research on tokenization in general, and tokenization fairness in particular.\n\nWhile these contributions are laudable, I have two main concerns:\n- The paper only analyzes statistical properties of the tokenizers but does not examine how these properties affect language models trained with them. Fairness in terms of performance is a key motivating factor of prior work on tokenization inequalities (e.g., see §4.3 in [Ahia et al., 2023](https://aclanthology.org/2023.emnlp-main.614/)), so this issue, I think, is very much in-scope for the current paper. For example, it is unclear whether the ideal vocabulary sizes in terms of token premiums also result in the best relative performance compared to English. \n- The analyses provided by the authors stay on a somewhat superficial level, mostly because the authors focus on quantitative results. It seems important to also provide _qualitative_ analyses that back up the quantitative findings. For example, linguistic, and in particular morphological, analyses might provide for a better understanding of the quantitative findings.\n\nMinor points:\n- The SentencePiece analysis in §3.1 is unclear. SentencePiece is a library that implements both Unigram and BPE, so is this analysis about implementation differences between HuggingFace and SentencePiece? It seems the authors just compared their tokenizers with the tokenizers trained for the Goldfish models, but this means that practically all aspects of tokenizer training are different, so it is difficult to tell how much we can learn from the comparison.\n- The authors missed a key related work, specifically a NeurIPS paper from last year that proposed a new method to mitigate cross-lingual tokenization inequalities ([Ahia et al., 2024](https://openreview.net/forum?id=1e3MOwHSIX)).", "was_revised": true, "final_justification": "After the rebuttal, I still see critical limitations, so I believe the score of 3 is justified.", "rebuttal": ">The paper only analyzes statistical properties of the tokenizers but does not examine how these properties affect language models trained with them. Fairness in terms of performance is a key motivating factor of prior work on tokenization inequalities (e.g., see §4.3 in Ahia et al., 2023), so this issue, I think, is very much in-scope for the current paper. For example, it is unclear whether the ideal vocabulary sizes in terms of token premiums also result in the best relative performance compared to English.\n\nFirst, we think that the statistical properties of the tokenizers are important in and of themselves. Tokenization affects the cost of training and inference, as it determines the number of computations over a given text or prompt. We will emphasize these points in the updated paper. We agree that it is also important to follow up these findings by linking these results to model performance on specific tasks. The aim of this paper is to offer broad coverage of languages, tokenization algorithms, and vocabulary sizes. Therefore, we are not able to train models for all of the tokenizers we train, due to compute limitations. However, we hope that future work can test whether these findings also link to differences in language model performance. \n\n>The analyses provided by the authors stay on a somewhat superficial level, mostly because the authors focus on quantitative results. It seems important to also provide qualitative analyses that back up the quantitative findings. For example, linguistic, and in particular morphological, analyses might provide for a better understanding of the quantitative findings.\n\nWe agree that it would be interesting to investigate other aspects of these tokenizers. Morphological alignment, for example, is difficult to evaluate comparably across languages, especially languages with very little overt morphology, e.g. Chinese, or languages with non-concatenative morphology, e.g. Arabic (see [2] for discussion on this point), which poses theoretical and methodological challenges for addressing this question. We release our tokenizers, which enables others to investigate this in the future.\n\n>The SentencePiece analysis in §3.1 is unclear. SentencePiece is a library that implements both Unigram and BPE, so is this analysis about implementation differences between HuggingFace and SentencePiece? It seems the authors just compared their tokenizers with the tokenizers trained for the Goldfish models, but this means that practically all aspects of tokenizer training are different, so it is difficult to tell how much we can learn from the comparison.\n\nYes, we agree that we were unclear on this point. Here, SentencePiece refers to the SentencePiece implementation of Unigram, which contrasts with the HuggingFace implementation. As we show, the two offer very different results, despite being based on the same algorithm. This is likely due to implementation differences. We will clarify these points in the updated paper. \n\n>The authors missed a key related work, specifically a NeurIPS paper from last year that proposed a new method to mitigate cross-lingual tokenization inequalities (Ahia et al., 2024).\n\nWe thank the reviewer for highlighting this paper. One difference between our method and that in Ahia et al. (2024) is that our approach does not require modifying tokenization algorithms or model architectures. We use existing tokenization algorithms, and only manipulate vocabulary size. We will include further discussion of the Ahia paper and how it differs from our approach in the updated paper. \n\n>Do your findings translate to better performance in terms of language modeling and downstream tasks?\n\nThis was not evaluated in the current work, but we agree that addressing this would be very valuable!\n\n>Can you provide qualitative analyses to better explain your quantitative findings?\n\nWe provided some qualitative discussion in Appendix F, but will try to include more illustrative examples in the updated manuscript to clarify the points in the paper!\n\n>The authors do not discuss the potential negative societal impact of their work. In terms of general limitations, the authors should mention that they do not analyze language model performance, which in my view is a key limitation if this paper.\n\nWe do not foresee any negative societal impact of the work. In fact, we hope that the outcome of the paper is improved crosslingual equity in tokenization and language models more generally. We agree that analyzing language model performance would strengthen the work, and will include discussion of this in the Limitations section. \n\n\n[1] Kreutzer, J., Caswell, I., Wang, L., Wahab, A., Van Esch, D., Ulzii-Orshikh, N., ... & Adeyemi, M. (2022). Quality at a glance: An audit of web-crawled multilingual datasets. Transactions of the Association for Computational Linguistics, 10, 50-72.\n\n[2] Catherine Arnett, Marisa Hudspeth, and Brendan O’Connor (2025). Evaluating Morphological Alignment of Tokenizers in 70 Languages. Tokenizer Workshop at ICML. Vancouver, Canada."}]

Author rebuttal:

Meta-review: This paper investigates token premiums, defined as disparities in the number of tokens required to encode parallel text across different languages. High token premiums can reduce training throughput and increase inference costs.

The reviewers highly appreciated the large-scale, systematic experiments, which involved training 7k tokenizers for 98 languages. The proposed approaches for mitigating token premiums were regarded as important. One reviewer raised a concern about potential data leakage; however, the author rebuttal clarified that steps were taken to avoid leakage. The reviewer accepted this clarification and indicated they would raise their score, though the change was not reflected in the system. This has been taken into account in the meta-review.

A recurring concern among reviewers is the absence of an end-to-end evaluation to directly assess the downstream impact of reducing token premiums. While this limits the scope of the current study, the work nonetheless provides valuable insights and represents an interesting and worthwhile contribution to the NeurIPS community.
Decision: Accept (poster)