SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Aleksandar Petrov, Emanuele La Malfa, Philip H.S. Torr, Adel Bibi
aleks@robots.ox.ac.uk
Recent language models have shown impressive multilingual performance, even when not explicitly trained for it. Despite this, there are concerns about the quality of their outputs across different languages. In this paper, we show how disparity in the treatment of different languages arises at the tokenization stage, well before a model is even invoked. The same text translated into different languages can have drastically different tokenization lengths, with differences up to 15 times in some cases. These disparities persist even for tokenizers that are intentionally trained for multilingual support. Character-level and byte-level models also exhibit over 4 times the difference in the encoding length for some language pairs. This induces unfair treatment for some language communities in regard to the cost of accessing commercial language services, the processing time and latency, as well as the amount of content that can be provided as context to the models. Therefore, we make the case that we should train future language models using multilingually fair subword tokenizers.
Language models are increasingly important in natural language processing tasks, as they can understand and generate human-like language. They have been deployed in applications such as virtual assistants (Chen et al., 2021; Ouyang et al., 2022), chatbots (Kuhail et al., 2023; Lee et al., 2023), machine translation (Stahlberg, 2020; Ranathunga et al., 2023), and text summarization (Krysciński et al., 2019; Xu et al., 2020). As general-purpose technologies, it is also projected that Large Language Models (LLMs) will have a significant impact on the economy and the labour market (Teubner et al., 2023; Eloundou et al., 2023).
Such LLMs are often trained using large swaths of internet content regardless of language. Hence, these models often end up being multilingual, even if not by design. ChatGPT (OpenAI, 2022) is a prominent recent example (Bang et al., 2023; Jiao et al., 2023; Johnson, 2023). Given the economic benefits of LLMs and LLM-derived technology, it's beneficial that they support multiple languages. Equal access is crucial, and multilingual support is a key component of this.
However, this multilingualism is currently treated as a curious emergent phenomenon rather than a carefully designed, controlled and managed process. The performance of LLMs has been shown to be generally lower in non-target languages, a problem especially pronounced for low-resource languages (Virtanen et al., 2019; Ahuja et al., 2023). Providing access to the same technology in different languages but moderation and safety tools only for some has resulted in dire societal consequences before (Stecklow, 2018; Facebook, 2021; Leung, 2022). Differing cost of access could also reinforce inequality in opportunities for economic mobility and social participation (Lythreatis et al., 2022). Therefore, as LLM multilingualism emerges,
we should pay attention to ensuring comparable performance and accessibility across the supported languages, regardless of whether by design or by chance.
This work demonstrates how the unequal treatment of languages arises at the tokenization stage, $^{1}$  well before the language model sees any data at all. For instance, the tokenizer employed by ChatGPT (OpenAI, 2022) and GPT-4 (OpenAI, 2023) uses about 1.6 times more tokens to encode the same text in Italian as it does in English, 2.6 times for Bulgarian and 3 times for Arabic. For Shan—the native language of people from the Shan State in Myanmar—that difference can be as high as 15 times. Unicode character and byte-level tokenization also result in drastically different encoding lengths across languages: byte-level representation of the same text is over 4 times longer for Burmese or Tibetan than Chinese.
We discuss three fairness implications of these differences in tokenization:
1. Cost: Commercial services charge users per token or Unicode character. In either case, these discrepancies lead to users of some languages paying at least 2.5 times more for the same task as users of English.
2. Latency: The number of tokens has a direct effect on the processing time for a task. Some languages can require twice the time to process the same content as English. This may be critical for real-time applications like emergency services.
3. Long context processing: Many models have a fixed-size context. Users of languages that are more token-efficient can use these systems to process or generate texts that may be more than an order of magnitude longer than users of other languages. This may lead to significant discrepancies in the quality of service.
Therefore, we make the case for multilingual tokenization parity: tokenizers should produce similar encoded lengths for the same content across languages. Hence, we advocate for multilingually fair tokenizers for the next generation of language models.
2 Intriguing Properties of Tokenization Across Languages
Subword tokenization is currently the preferred approach for state of the art language models (Kudo and Richardson, 2018). In this section, we show how artefacts from data collection might result in technical terms or rare words having dedicated tokens, while more commonly used words and non-Latin characters end up requiring multiple tokens.
Using large corpora scraped from the internet results in peculiar choices for tokens. For instance, GPT-2 contains glitch tokens which can be usernames or concepts from games (Rumbelow and Watkins, 2023b; Miles and Riley, 2023). As an example, BuyableInstoreAndOnline, likely coming from an online store backend, has a dedicated token. Another such token is rawdownloadcloneembedreportprint.
While such obscure terms get their own tokens, the frequently used Arabic word "مُلْ" (meaning "why") is broken into letters with each letter having its own token. The same word in Bulgarian ("زَمُ") is not only broken down to letters, but some of the letters require two tokens to be represented, resulting in 6 tokens for this 4 letter word.
One may argue that this is because Arabic and Bulgarian are not target languages for GPT-2. However, glitch tokens also exist for Japanese: there are dedicated tokens for “じウス”, the name of the ancient Greek god Zeus and “サイフン”, the name of an ice cream chain (Rumbelow and Watkins, 2023a). At the same time, GPT-2 requires 3 tokens to represent the much more commonly used kanji character for “to say”:
In fact, more than half of the Japanese kanji characters require three tokens.
<sup>1</sup>We offer a summary of the relevant tokenization approaches in Appendix A.
The existence of glitch tokens like "Biz" and "s-ien" despite the lack of a dedicated token for "言" shows that tokenizers are heavily influenced by the biases of the corpus source. If one uses non-natural inputs, log files, or specialist forums, the tokenizer vocabulary would reflect this. While c1100k_base, the tokenizer used for the newer ChatGPT and GPT-4, may not have glitch tokens it still requires two tokens to represent some Cyrillic letters and three tokens for more than  $65\%$  of kanji characters. Therefore, to place all languages on an equal footing, it is important to have the tokens balanced across languages.
3 Measuring Tokenizer Parity
To demonstrate that the above examples are not anecdotal evidence, we introduce the notion of tokenizer parity to systematically assess how fairly tokenizers treat equivalent sentences in different languages. Parity occurs when a tokenizer exhibits similar tokenized lengths for the same sentence in different languages. Take a sentence  $s_A$  in language  $A$  and its translation  $s_B$  to language  $B$ . Then, a tokenizer  $t$  achieves parity for  $A$  with respect to  $B$  at  $s_A$  and  $s_B$  if  $|t(s_A)| / |t(s_B)| \approx 1$ , where  $t(s_A)$  is the tokenization of the sentence  $s_A$  and  $|t(s_A)|$  represents its length. The ratio  $|t(s_A)| / |t(s_B)|$  is the premium for  $A$  relative to  $B$ .<sup>2</sup>
4 Tokenization Length Differences Across Languages
Languages vary significantly in the number of tokens required to encode the same content, as demonstrated in the examples in Section 2. Hence, following Section 3, we measure the tokenization premium of different tokenizers. To this end, we use the FLORES-200 parallel corpus, comprising of the same 2000 sentences taken from Wikipedia and human-translated to 200 different languages (Guzmán et al., 2019; Goyal et al., 2021; Costa-jussa et al., 2022). We look at subword tokenization models which target English, languages other than English, language varieties, multi-lingual tokenizers, as well as tokenizer-free (byte-level) modelling.
4.1 Parity for English-centric Models
As most models target English, we report in Table 1 the tokenization parity for a subset of languages in FLORES-200. The parities for all 200 languages are in Appendix C.  ${}^{3}$  GPT-2 (Radford et al., 2019), RoBERTa (Liu et al., 2019), and the r50k_base, p50k_base and p50k_edit tokenizers (OpenAI, 2022) have close  ${}^{4}$  tokenization lengths so we report them together. ChatGPT and GPT-4 share the same c1100k_base tokenizer and are also reported together. Some models, such as FlanT5 (Chung et al., 2022), use a special UNK token to model unknown symbols not encountered during training. Hence, to ensure a fair comparison, we report only languages where no more than  ${10}\%$  of the input characters are mapped to UNK tokens (marked with -).
Table 1 shows large variations in the tokenizer parity for all tokenizers. For GPT-2 and RoBERTa, Pangasinan, the language with shortest tokenization, is already  $66\%$  more expensive to process than English. ChatGPT and GPT-4 are slightly closer to parity, likely
Table 1: Premiums with respect to English on FLORES-200 for several English-centric models. The languages in the top or bottom three for any tokenizer, as well as the ones discussed in the text, are shown.
GPT-2 RoBERTa	ChatGPT GPT-4	FlanT5
Bulgarian	5.51	2.64	—
Burmese	16.89	11.70	—
Chinese (Simplified)	3.21	1.91	—
Dzongkha	16.36	12.33	—
English	1.00	1.00	1.00
French	2.00	1.60	1.60
German	2.14	1.58	1.37
Italian	2.01	1.64	2.18
Japanese	3.00	2.30	—
Jingpho	2.65	2.35	3.41
Maori	2.45	2.35	3.28
Norwegian Bokmål	1.86	1.56	2.24
Odia	13.38	12.48	—
Pangasinan	1.66	1.57	2.18
Portuguese	1.94	1.48	2.21
Romanian	2.48	1.88	1.50
Santali	12.86	12.80	—
Shan	18.76	15.05	—
Spanish	1.99	1.55	2.23
Standard Arabic	4.40	3.04	—
Tumbuka	2.78	2.57	3.29
Vietnamese	4.54	2.45	—
2The concurrent work by Ahia et al. (2023) also evaluates the tokenization premiums for different languages and reaches similar conclusions.
An interactive table of all the languages and tokenizers is also available on the project website.
4The largest tokenizer parity difference between them is less than 0.005.
Table 2: Tokenizer premiums on the FLORES-200 dataset for non-English centric models. The premium is computed with respect to the target language (Modern Standard Arabic was used for Arabic BERT and Simplified Chinese for RoCBert). The languages that are in the top or bottom two for any tokenizer as well as the ones discussed are shown.
Arabic BERT (Chinese)	RoCBert (French)	CamemBERT (French)	GottBERT (German)	BERT Japanese	PhoBERT (Vietnamese)
Belarusian	4.74	—	—	5.62	—	3.46
Bulgarian	4.30	—	—	4.73	—	3.09
Catalan	2.36	2.86	1.59	1.89	1.95	1.57
Chinese (Simp.)	—	1.00	—	3.95	0.82	—
Chinese (Trad.)	—	0.94	—	3.82	0.84	—
Dutch	2.52	2.92	1.68	1.73	1.98	1.58
Dzongkha	—	—	—	16.12	—	—
English	1.83	2.60	1.20	1.35	1.49	1.20
French	2.42	3.10	1.00	1.99	2.03	1.66
Friulian	2.33	2.79	1.66	1.98	1.92	1.59
German	2.63	3.12	1.85	1.00	2.04	1.67
Greek	4.93	3.00	—	6.73	—	3.73
Italian	2.58	3.10	1.63	1.93	2.04	1.60
Japanese	1.85	1.34	—	4.35	1.00	—
Jingpho	3.12	3.12	2.13	2.55	2.47	1.84
Luxembourgish	2.56	2.97	1.82	1.75	1.96	1.72
N. Lev. Arabic	1.00	—	—	6.52	—	—
Shan	—	—	—	16.88	—	—
Standard Arabic	1.00	—	—	7.03	—	—
Tagalog	2.84	3.28	2.00	2.20	2.39	1.74
Tosk Albanian	2.66	2.90	2.17	2.39	—	2.02
Tsonga	3.01	3.09	2.03	2.29	2.46	1.76
Tumbuka	3.27	3.49	2.21	2.61	—	2.00
Vietnamese	2.52	2.55	—	4.12	—	1.00
Yue Chinese	—	0.92	—	3.75	—	—
Table 3: Tokenizer premiums on the FLORES-200 dataset for the MuRIL model focusing on 16 Indian languages and English. The premium is computed with respect to English.
MuRIL
English	1.00
Nepali	1.01
Bengali	1.01
Tamil	1.06
Marathi	1.06
Kannada	1.06
Hindi	1.16
Malayalam	1.18
Gujarati	1.19
Sanskrit	1.21
Telugu	1.21
Odia	1.21
Sindhi	1.22
Assamese	1.24
Urdu	1.26
Eastern Panjabi	1.35
Kashmiri (Arabic)	1.75
Kashmiri (Devanagari)	1.75
due to their larger vocabulary size. However, the cheapest languages, Portuguese, Pangasin and German, still see a premium of  $50\%$  when compared to English. Shan has the worst tokenizer parity for all four models. Take as an example “ $\frac{C}{C}$ ”, one of the Shan words for “you”. It is tokenized by ChatGPT and GPT-4 as:
This word is constructed from one consonant and three diacritics. As the diacritics are encoded separately, there are four Unicode codepoints for this Shan character, resulting in 9 tokens. The English "you" has three characters but a single token.
FlanT5 has more than  $10\%$  UNK tokens for  $42\%$  of languages (— in Table 1). It has a higher premium than the other tokenizers for all other languages except German and Romanian.
Summary. All four English-centric tokenizers we consider are far from tokenization parity. Portuguese is closest to parity with English for the ChatGPT and GPT-4 tokenizer but still requires about  $50\%$  more tokens for the same content. Shan is furthest from parity for this tokenizer with 15 times longer encodings compared to English. FlanT5 is closer to parity with its premium range 1.37-3.41 but it encodes only  $54\%$  of the languages, so we cannot say that it is more multilingually fair than the other tokenizers.
4.2 Parity for Models with Other Target Languages
There are models targeting languages other than English as well. Table 2 shows six such models based on the BERT architecture (Devlin et al., 2019): ArabicBERT (Safaya et al., 2020), RoCBert for Chinese (Su et al., 2022), CamemBERT for French (Martin et al., 2020), GottBERT for German (Scheible et al., 2020), BERT Japanese (Tohoku NLP Group, 2019) and PhoBERT for Vietnamese (Nguyen and Nguyen, 2020).
Table 4: Tokenizer premiums with respect to English on FLORES-200 for multilingual models. The languages that are in the top or bottom two for any tokenizer, as well as the ones discussed in the text, are shown.
XLM-R	NLLB	mT5	M2M100	BLOOM
Bulgarian	1.16	1.31	1.28	1.23	2.49
Central Kanuri	2.60	2.54	2.43	2.49	2.10
Chinese (Simp.)	0.97	1.11	0.92	1.05	0.95
Dzongkha	—	1.48	4.24	—	7.36
English	1.00	1.00	1.00	1.00	1.00
Indonesian	0.94	0.93	1,08	0.98	0.96
Italian	1.19	1.25	1.34	1.25	1.62
Japanese	1.11	1.01	0.90	1.20	1.81
Kabiyè	2.98	1.56	2.83	2.71	3.34
Santali	—	2.49	—	—	12.71
Shan	4.43	1.94	3.28	4.63	12.06
Std. Arabic	1.18	1.40	1.35	1.29	1.14
Std. Tibetan	—	1.44	3.68	—	6.66
Uyghur	1.41	1.40	2.57	3.00	3.67
Yue Chinese	0.93	1.05	0.95	1.03	0.93
Table 5: Tokenizer premiums with respect to English on FLORES-200 for byte-level models. The languages that are in the top or bottom two for any tokenizer, as well as the ones discussed in the text, are shown.
CANINE
UTF-32 bytes	ByT5
UTF-8 bytes
Bulgarian	1.04	1.89
Burmese	1.24	3.51
Chinese (Simplified)	0.34	0.93
Chinese (Traditional)	0.32	0.89
Dzongkha	1.25	3.64
English	1.00	1.00
Italian	1.18	1.19
Japanese	0.44	1.27
Shan	1.42	3.94
Standard Arabic	0.88	1.60
Standard Tibetan	1.13	3.31
Tok Pisin	1.28	1.28
Tumbuka	1.30	1.32
Yue Chinese	0.31	0.87
The English premium for GottBERT (1.35) is lower than those for Dutch (1.73) and Luxembourg (1.75), which are more linguistically similar to German. CamemBERT is similar: English has the lowest premium (1.20), while Catalan (1.59) and Friulian (1.66) have higher premiums. PhoBERT also has English with the lowest tokenizer premium (1.20). Thus, even models targeting other languages exhibit a preference for English tokenization.
RoCBert and BERT Japanese differ by having the other target language as the one closest to parity, possibly due to the partially shared script. ArabicBERT demonstrates a similar behaviour, with Central Kanuri (1.27) and Acehnese (1.73), both written in Arabic script, and with English at 1.82. Sharing writing systems seems to improve tokenization parity.
Across all tokenizers, the premium for English relative to the respective target language is significantly lower than the premium of RoBERTa for that target language. This asymmetry between English and all other languages likely stems from the extensive incorporation of English in documents written in other languages (Zhang et al., 2022).
We also consider MuRIL, a BERT-based model trained on 16 Indian languages and English (Khanuja et al., 2021). Despite the model's focus on Indian languages, it remains most token-efficient for English (see Table 3).
Unequal treatment of dialects or linguistic varieties can lead to social and economic disadvantages making it important to also study the tokenization differences between the "standard" language and its varieties. For Swiss German and the Mauritian and Haitian Creoles, there are large differences in tokenization lengths compared respectively to High German (on GottBERT) and French (on CamemBERT). English is much closer to parity for both models than these language varieties. Therefore subword tokenizers might not be able to generalize to language varieties, such as dialects and creoles. The tokenizers of ArabicBERT and BERT Japanese, however, are close to parity across various dialects of both languages and have lower premiums for the dialects than for English. This is likely due to the good representation of the dialects in the dataset as well as the dialects being linguistically closer to the respective standard languages. The detailed analysis is deferred to Appendix B.
Summary. We observed that the tokenizers targeting French, German and Vietnamese have English as the language closest to parity, rather than more linguistically close languages. On the other hand, tokenizers for Arabic, Chinese and Japanese have lower premiums for languages they share a script with. Notably, despite targeting Indian languages, MuRIL still has the shortest tokenizations for English. Finally, across all tokenizers, the premium for English is lower than the premium for the same language for the English-centric RoBERTa. Hence, we conclude that tokenizers for other languages give English preferential treatment.
Figure 1: Comparison of the tokenization premiums for XLM-R and RoBERTa for the subset of languages that XLM-R encodes with less than  $10\%$  to the UNK token.
Figure 2: Average processing time and length of the tokenized inputs of RoBERTa. Each FLORES-200 sentence is processed for 20 independent runs. The script family designation is only for illustration purposes.
4.3 Parity for Multilingual Models
There has been a growing interest in multilingual language models, particularly for translation (Dabre et al., 2020). As these models are intended to support a variety of languages, one would expect them to be close to tokenizer parity. We compare several such multilingual models: XML-R (Conneau et al., 2020), NLLB (Costa-jussà et al., 2022), M2M100 (Fan et al., 2021) and mT5 (Xue et al., 2020). All of these models use the SentencePiece tokenizer with upsampling for rare languages. The final model, BLOOM (Scao et al., 2022), uses byte-level BPE instead of SentencePiece and is designed to maintain similar ratios of tokens per word for each language as reference monolingual tokenizers.
BLOOM and NLLB encode all languages with less than  $10\%$  UNK tokens, respectively thanks to byte-level BPE tokenization and being trained on the same 200 languages as FLORES-200 (see Table 4). The other three models fail to encode at least one language. All five models have languages with premiums of more than 2.5. Still, all models are better than the English-centric models in Table 1. Figure 1 shows how XLM-R is much closer to parity than RoBERTa (on which it is based), over all languages it can encode. However, none of the models uniformly reaches parity across all languages. Therefore even models which are intentionally designed to be multilingual suffer from a lack of tokenization parity.
Summary: Multilingual models can improve the tokenization parity for different languages but challenges remain in achieving tokenization parity across all languages.
4.4 Parity for Byte-level Tokenization Models
Byte-level representation is crucial for multilingual support, as it encodes any Unicode codepoint, even if unseen during training. One can also bypass vocabulary construction and directly employ the 256 byte values, enabling end-to-end training (byte-level tokenization). CANINE (Clark et al., 2022) is a large model that operates at the Unicode codepoint level rather than the byte level. The CANINE tokenizer is thus equivalent to the UTF-32 encoding, resulting in an implicit tokenizer with a vocabulary of 1,114,112. ByT5 (Xue et al., 2022), on the other hand, uses the UTF-8 encoding: an implicit vocabulary of 256 tokens. $^{5}$
To be consistent, we will refer to the characters and bytes in the encoding of the CANINE and ByT5 tokenizers as tokens as they fulfil a similar role.
These byte-level models can represent any Unicode codepoint without an explicit tokenization step but there are still significant tokenization disparities. For CANINE, Shan has a premium of 4.58 relative to Yue Chinese. This can be attributed to the fact that CANINE provides a single token for each Unicode codepoint, which results in Chinese being more token-efficient (with a premium range 0.31-0.34 relative to English for the three Chinese languages) as each character is treated as a single token. This encoding also puts Shan at a disadvantage, as its encoding relies on diacritics represented as separate Unicode codepoints. Other languages, such as Tok Pisin and Tumbuka, which use the Latin script but require more characters than English for the same text, also face similar challenges.
Tokenization disparity is also present in the ByT5 model. The tokenization premium for ByT5 ranges from 0.87 (for Yue Chinese) to 3.94 (for Shan). The introduction of the variable-width UTF-8 encoding of Unicode characters in ByT5 creates another issue of unequal treatment. ASCII characters, which are sufficient for English, require only one byte. Other Latin script characters, as well as Greek, Cyrillic, Coptic, Armenian, Hebrew, Arabic and Syriac, require two bytes, while Chinese, Japanese and Korean characters require three bytes. Therefore, the tokenization of Chinese and Japanese is about three times as long for ByT5 as it is for CANINE (Table 5). Shan's premium of 3.94 is due to the fact that all its consonants and diacritics require three bytes. For example, the word “ $\frac{c}{c}$ ” is encoded by ByT5 as 12 tokens, whereas the corresponding “you” requires 3 tokens. The situation is similar for other languages like Dzongkha, Tibetan and Burmese.
Summary. Byte-level models also fail to achieve parity among the languages from FLORES-200 exhibiting a premium of over 4 times for some language pairs. There are two sources of multilingual tokenizer disparities. First, there are natural differences in the number of characters used in different languages to communicate the same content. Second, the UTF-8 standard uses different numbers of bytes to encode codepoints of different scripts.
5 Fairness Implications of Tokenization Length Differences
We showed that no matter whether one uses subword, multilingual, or byte-level tokenization, none of the tokenizers gets close to parity for all languages in FLORES-200. This lack of tokenization parity is not merely a curiosity: it leads to unfairness in the cost to access language models, the latency of the service and the amount of data that can be processed.
5.1 Cost