Language Model Tokenizers Introduce Unfairness Between Languages
Authors: Aleksandar Petrov
Emanuele La Malfa
Philip Torr
Adel Bibi
Venue: NeurIPS 2023 poster (2023)
Keywords: LLM
language model
tokenizer
multilingual
language
fairness
TL;DR: We show singificantly longer tokenized length for languages other than English and argue that this casuses unfair treatment of some language communities.

Abstract: Recent language models have shown impressive multilingual performance, even when not explicitly trained for it.
Despite this, there are concerns about the quality of their outputs across different languages.
In this paper, we show how disparity in the treatment of different languages arises at the tokenization stage, well before a model is even invoked.
The same text translated into different languages can have drastically different tokenization lengths, with differences up to 15 times in some cases.
These disparities persist even for tokenizers that are intentionally trained for multilingual support.
Character-level and byte-level models also exhibit over 4 times the difference in the encoding length for some language pairs.
This induces unfair treatment for some language communities in regard to the cost of accessing commercial language services, the processing time and latency, as well as the amount of content that can be provided as context to the models.
Therefore, we make the case that we should train future language models using multilingually fair subword tokenizers.

Language Model Tokenizers Introduce Unfairness Between Languages
Alcksandar Petrov, Emanuele La Malfa, Philip H.S. Torr, Adel Bibi University of Oxford aleksšrobots. ox ac
Abstract
Recent language models have shown impressive multilingual performance, even when not explicitly trained for it. Despite this, there are concerns about the quality of their outputs across different languages. In this pa- per, we show how disparity in the treatment of different languages arises the tokenization stage, well before a model is even invoked. The same text translated into different languages can have drastically different tok- enization lengths, with differences up to 15 times in some cases. These disparities persist even for tokenizers that are intentionally trained for mul- tilingual support. Character-level and byte-level models also exhibit over 4 times the difference in the encoding length for some language pairs. This induces unfair treatment for some language communities cost of accessing commercial language services, the processing time and la- tency, as well as the amount of content that can be provided as context to the models. Therefore, make the case that should train future language models using multilingually fair subword tokenizers.
regard the
1 Introduction
Language models are inereasingly important in natural language processing tasks, as they can understand and generate human-like language. They have been deployed in applications such as virtual assistants (Chen et al., 2021; Ouyang et al., 2022), chatbots (Kuhail et al., 2023; Lee et al., 2023), machine translation (Stahlberg, 2020; Ranathunga et al., 2023), and text summaaiiatiin (Kryáciískii et al., 2019; Xu al., 2020). As general-purpose technologies, it is also projected that Large Language Models (LLMs) will have significant impact the economy and the labour market (Teubner et al., 2023; Eloundou et al., 2023).
Such LLMs are often trained using large swaths of internet content regardless of language. Hence, these models often end up being multilingual, even if not by design. ChatGPT (OpenAI, 2022) is a prominent recent example (Bang et al., 2023; Jiao et al., 2023; Johnson, 2023). Given the cconomic benefits of LLMs and LLM-derived tcchnology, it's beneficial that they support multiple languages. Equal access is crucial, and multilingual support is key component of this.
However, this multilingualism is currently treated as a curious emergent phenomenon rather than a carefully designed, controlled and managed process. The performance of LLMs has been shown to be generally lower in non-ta non-target languages, problem especially pronounced for low-resource languages (Virtanen al., 2019; Ahuja al., 2023). Providing access the same technology in different languages but moderation and safety tools only for some has resulted in dirc socictal consequences before (Stecklow, 2018; Facebook, 2021; Leung, 2022). Differing cost of access could also reinforce incquality in opportunities for economic mobility and social participation (Lythreatis et al., 2022). Therefore, as LLM multilingualism emerges,
37th Conference on Neural Information Processing Systems (NeurIPS 2023).

---

we should pay attention to ensuring comparable performance and accessibility across the supported languages, regardless of whether by design or by chance.
This work demonstrates how the unequal treatment of languages arises at the tokenization any data at all. For instance, the tokenizer
stage, well before the language model employed by ChatGPT (OpenAI, 2022) and GPT-4 (OpenAI, 2023) uses about 1.6 times more tokens encode the same text in Italian it does in English, 2.6 times for Bulgarian and 3 times for Arabic. For Shan the native language of people from the Shan State in Myanmar- that difference can be as high as 15 times. Unicode character and byto-level tokenization also result in drastically different encoding lengths across languages: byte-level representation of the same text is over 4 times longer for Burmese or Tibetan than Chinese.
We discuss three fairness implications of these differences in tokenization:
1. Cost:
Commercial services charge users per token or Unicode character. In either case, these discrepancies lead to users of some languages paying at least 2.5 times more for the same task as users of English.
2. Latency: The number of tokens has a direct effect on the processing time for task. Some languages can require twice the time to process the same content as English. This may be critical for real-time applications like emergency services.
3. Long context processing: Many models have fixed-size context. Users of languages that are more token-efficient can uase these systems to process or generate than an order of magnitude longer than users of other
texts that may be languages. This may lead to significant discrepancies in the quality of service.
Therefore, we make the case for maltilingual tokenization parity: tokenizers should produce similar encoded lengths for the same content across languages. Hence, we advocate for multilingually fair tokenizers for the next generation of language models.
2 Intriguing Properties of Tokenization Across Languages
Subword tokenization currently the preferred approach for state of the art language models (Kudo and Richardson, 2018). In this section, we show how artefacts from data collection might result in technical terms or rare words having dedicated tokens, while more commonly used words and non-Latin characters end up requiring multiple tokens.
Using large corpora scraped from the internet results in peculior choices for tokens. For instance, GPT-2 contains glitch tokens which from games (Rumbelow and Watkins, 2023b; Miles and Riley, 2023). As an example, www.blleInntorrenndnn11n.1
be usernames concepts
likely coming from online store backend, has dedicated token. Another such token is raudownloadcl www.aa.oooeeeeeeeoorrr......
While such obscure terms get their own tokens, the frequently used Arabic word "öUl" (meaning "why")) is broken into letters with each letter having its own token. The same word in Bulgarian ("samo") is not only broken down to letters, but some of the let- ters require tokens to be represented, resulting in 6 tokens for this letter word.
5821 56434
5821 10386 8700
140 115 16142
141 231 15166
One may argue that this is because Arabic and Bulgarian are not target languages for GPT-2. However, glitch tokens also exist. for Japanese: there are dedicated tokens for " x", the name of the ancient Greek god Zeus and - 7 >", the name of an ice cream chain (Rumbelow and Watkins, 2023a). At the same time, GPT-2 requires tokens to represent the much more commonly used kanji character for "to say":
164 101 222
In fact, more than half of the Japanese kanji characters require three tokens.
We offer summary of the relevant tokenization approaches in Appendix A.

---

The existence of glitch tokens like " t X and - 7 >" despite the lack of a dedicated token for "hows that tokenizers are heavily influenced by the biases of the corpus source. If one uses non-natural inputs, log files, specialist forums, the tokenizer vocabulary would reflect this. While c1100k base, the tokenizer used for the newer ChatGPT and GPT-4, may not have glitch tokens it still requires two tokens to represent some Cyrillic letters and three tokens for more than 65% of kanji characters. Therefore, to place all languages on an equal footing, it is important to have the tokens balanced across languages.
3 Measuring Tokenizer Parity
To demonstrate that the above examples are not anecdotal evidence, we introduce the notion of tokenizer parity to systematically assess how fairly tokenizers treat equivalent sentences in different languages. Parity occurs when a tokenizer exhibits similar tokenized lengths for the same sentence in different languages. Take a sentence 8A in language A and its translation sy to language B. Then, a tokenizer t achieves parity for A with respect to B at SA and sg if [ z 1, where t(Sa) is the tokenization of the sentence 9 and [t(a)) represents its length. The ratio {(a))/[((sn)] is the premium for A relative to B.
Tokenization Length Differences Across Languages
Languages vary significantly in the number of tokens required to encode the same content, as demonstrated in the examples in Section 2. Hence, following Section 3, we measure the the FLORES-22 200 parallel
tokenization premium of different tokenizers. To this end, corpus, comprising of the same 2000 sentences taken from Wikipedia and human-tramm translated to 200 different languages (Guzmán et al., 2019; Goyal et al., 2021; Costa-jussà et al., 2022). We look at subword tokenization models which target English, languages other than English, language varicties, multi-lingual tokenizers, as well as tokenizer-free (byte-level) modelling.
4.1 Parity for English-centric Models As most models target English, we report Table 1 the tokenization parity for subset of languages FLORES-200. The parities for all 200 languages are in Appendix C. 3 GPT- 2 (Radford et al., 2019), RoBERTa (Liu et al., 2019), and the r50k_ base, p50k base and p50k_edit tokenizers (OpenAI, 2022) have close* tokenization lengths so report them together. ChatGPT and GPT-4 share the same c1100k base tokenizer and are also reported together. Some models, such as Flan (Chung et al., 2022), use special UNK token to model unknown symbols not encoun- tered during training. Hence, to ensure fair comparison, we report only languages where no more than 10% of the input characters are mapped to UNK tokens (marked with ).
Table 1: Premiums with respect to En- glish on FLORES-200 for several English- centric models. The languages in the top or bottom three for any tokenizer, as well as the ones discussed in the text, are shown.
GPT-2
ChatGPT GPT-4
FlanT5
RoBERTa
Bulgarian
5.51
2.64
Burmese
16.89
11.70
Chinese (Simplified)
3.21
1.91
Dzongkha
16.36
12.33
English
1.00
1.00
1.00
French
2.00
1.60
German
2.14
1.58
1.37
Italian
2.01
1.64
2.18
Japanese
3.00
2.30
Jingpho
2.65
2.35
3.41
Maori
2.45
3.28
Norwegian Bokmål Odia
1.86
1.56
2.24
13.38
12.48
Pangasinan
1.66
1.57
2.18
Table 1 shows large variations in the tokenizer parity for all tokenizers. For GPT-2 and RoBERTa, Pangasinan, the language with shortest tokenization, is already 60% more ex- pensive to process than English. ChatGPT and GPT-4 are slightly closer to parity, likely
Portuguese
1.94
1.48
2.21
Romanian
2.48
1.88
1.50
Santali
12.86
12.80
Shan
18.76
15.05
Spanish
1.99
1.55
2.23
Standard Arabic
4.40
3.04
Tumbuka
2.78
2.57
3.29
Vietnamese
4.54
2.45
The concurrent work by Ahia et al. (2023) also evaluates the tokenization premiums for different languages and reaches similar conclusions.
3 "n interactive table all the languages and tokenizers is also available on the project website. 'The largest tokenizer parity difference between them is less than 0.005.
3

---

Table 2: Tokenizer premiums the FLORES-200 datasct for non-English centric models. The premium is computed with respect to the target language (Modern Standard Arabic used for Arabic BERT and Simplified Chinese for RoCBert). The languages that are in the lop or bottom two for any lokenizer well as the ones discussed are shown.
Table 3: premiums on the FLORES-200 dataset for the MuRIL model locusing on 16 Indian languages and En- glish. The premium is computed with respect to English.
Tokenizer
Arabic RoCBert CamenBERT GuLLBERT BERT
PhoBERT (French) (German) Japanese (Vietnamese)
BERT (Chinese)
Belarusian
4.74
5.62
3.16
Bulgarian
4.30
4.73
MuRIL
Catalan
2.36
2.86
1.59
1.89
1.95
1.57
Chinese (Simp..
1.00
3.95
0.82
English
1.00
Chinese (Trad.)
0.94
3.82
0.84
Nepali
1.01
Dutch
2.52
2.92
1.68
1.73
1.98
1.58
Bengali
1.01
Dzongkha
16.12
Tamil
1.06
English
1.83
2.60
1.20
1.35
1.49
1.20
Marathi
1.06
French
2.12
3.10
1.00
1.99
2.03
1.08
Kannada
Friulian
2.33
1.98
1.92
1.59
Hindi
1.16
German
2.63
3.12
1.85
1.00
2.04
1.67
Malayalam
1.18
Greek
4.93
3.00
6.73
3.73
Gujarati
1.19
Hallan
2.58
1.63
2.04
1.60
Sanskrit
Japanese
1.85
1.34
4.35
1.00
Telugu
1.21
Jingpho
3.12
3.12
2.13
2.55
2.47
1.84
Odia
1.21
Luxembourgish
2.56
1.82
1.75
1.96
1.72
Sindhi
1.22
N. Lev. Arabic
1.00
6.52
Assamese
1.24
Shan
Standard Arahic
1.00
16.58
7.03
Urdu
1.26
Eastern Panjabl
1.35
Tagalog
2.84
3.28
2,00
2.20
2.39
1.74
Took Alhamian
2.66
2.39
Kashmiri (Arabie)
1.75
Tsonga
3.01
3.09
2.03
2.29
2.46
1.76
Kashmiri (Devanagari)
1.75
Tumbuka
3.27
3.19
2,21
2.61
2.00
Vietuamese
2.52
4.12
1.00
Yue Chinese
0.92
3.75
due to their larger vocabulary size. However, the cheapest languages, Portuguese, Pangasi- nan and German, still see premium of 50% when compared to English. Shan has the worst tokenizer parity for all four models. Take as example "5.", one of the Shan words for "you". It is tokenized by ChatGPT and GPT-4 as:
25870 247 157 224
224
25870
118 25870 116
This word is constructed from consonant and three diacrities. As the diacritics encoded separately, there are four Unicode codepoints for this Shan character, resulting in 9 tokens. The English "you" has three characters but single token.
FlanT5 has more than 10% UNK tokens for 42% of languages (- Table 1). It has a higher premium than the other tokenizers for all other languages except German and Romanian.
Summary. All four ceglish-centric tokenizers consider are far from tokenization parity, Portuguese is closest to parity with English for the ChatGPT and GPT-4 tokenizer but still requires about 50% more tokens for the same content. Shan is furthest from parity for this tokenizer with 15 times longer encodings compared to English. FlanT5 closer to parity with its premium range 1.37-3.41 but it encodes only 54% of the languages, cannot. say that it is more multilingually fair than the other tokenizers.
4.2 Parity for Models with Other Target Languages
There are models targeting languages other than English as well. Table 2 shows six such models based on the BERT architecture (Devlin al., 2019): ArabicBERT (Safaya al., 2020), RoCBert for Chinese (Su et al., 2022), CamemBERT for French (Martin et al., 2020), GoLBBRRT for German (Scheible et al., 2020), BERT Japanese (Tohoku NLP Group, 2019) and PhoBERT for Vietnamese (Nguyen and Nguyen, 2020).

---

Table 4: Tokenizer premiums with respect Table 5: Tokenizer premiums with respect to English on FLORES-200 for byte-level
to English on FLORES-22 for multilingual models. The languages that are in the top models. The languages that are in the top bottom two for any tokenizer, as well as the ones discussed in the lext, are shown.
or bottom two for any tokenizer, as well as the ones discussed in the text, are shown.
XLM-R NLLB m55 M2M100 BLOOM
CANINE UTF-32 bytes UTF-8 bytos
ByT5
Bulgarian
1.16
1.31
1.28
1.23
2.49
Central Kanuri
2.60
2.54
2.43
2.49
2.10
Bulgarian
1.04
1.89
Chinese (Simp.)
0.97
1.11
0.92
1.05
0.95
Burmese
1.24
3.51
Dzongkha
1.48
4.24
7.36
Chinese (Simplified)
0.34
0.93
English
1.00
1.00
1.00
1.00
1.00
Chinese (Traditional)
0.32
0.89
Indonesian
0.94
0.93
1,08
U.98
0.96
Dzongkha
1.25
3.64
Italian
1.19
1.25
1.34
1.25
1.62
English
1.00
1.00
Japanese
1.11
1.01
0.90
1.20
1.81
Italian
1.18
1.19
Kabiyè
2.98
1.56
2.83
2.71
3.34
Japanese
0.44
1.27
Santali
2.49
12.71
Shan
1.42
3.91
Shan
4.43
1.94 3.28
4.63
12.06
Standard Arabic
0.88
1.60
Std. Arabic
1.18
1.40
1.35
1.29
1.14
Standard Tibetan
1.13
3.31
Std. Tibetan
1.44
3.68
6.66
Tok Pisin
1.28
1.28
Uyghur
1.41
1.40
2.57
3.00
3.67
Tumbuka
1.30
1.32
Yuc Chinese
0.93
1.05
0.95
1.03
0.93
Yue Chinese
0.31
0.87
The English premium for GottBERT (1.35) is lower than those for Dutch (1.73) and Luxem- bourgish (1.75), which are more linguistically similar to German. CamemBERT is similar: English has the lowest premium (1.20), while Catalan (1.59) and Friulian (1.66) have higher premiums. PhoBERT also has English with the lowest tokenizer premium (1.20). Thus, models targeting other languages exhibit preference for English tokenization.
RoCBert and BERT Japanese differ by having the other target language as the one closest to parity, possibly due to the partially shared script. ArabicBERT demonstrates a similar behaviour, with Central Kanuri (1.27) and Acehnese (1.73), both written in Arabic script, and with English at 1.82. Sharing writing systems seems to improve tokenization parity,
Across all tokenizers, the premium for English relative to the respective target language is significantly lower than the premium of RoBERTa for that target language. This asymmetry between English and all other languages likely stems from the extensive incorporation of English in documents written in other languages (Zhang et al., 2022).
We also consider MuRIL, BERT-based model trained 16 Indian languages and English (Khanuja et al., 2021). Despite the model's focus on Indian languages, it remains most token-efficient for English (sce Table 3).
Unequal treatment of dialects or linguistic varieties can lead to social and economic disadvan- tages making it important to also study the tokenization differences between the "standard" language and its varieties. For Swiss German and the Mauritian and Haitian Creoles, there are large differences in tokenization lengths compared respectively to High German (on Got- tBERT) and French (on CamemBERT), English much closer to parity for both models than these language varieties. Therefore subword tokenizers might not be able to gener- alize to language varieties, such as dialects and creoles. The tokenizers of ArabicBERT and BERT Japanese, however, are close to parity across various dialects of both languages and have lower premiums for the dialects than for English. This is likely due to the good representation of the dialects in the dataset well as the dialects being linguistically closer to the respective standard languages. The detailed analysis is deferred to Appendix B.
Summary. We observed that the tokenizers targeting French, German and Vietnamese have English as the language closest to parity, rather than more linguistically close languages. On the other hand, tokenizers for Arabic, Chinese and Japanese have lower premiums for languages they share a script with. Notably, despite targeting Indian languages, MuRIL still has the shortest tokenizations for English. Finally, across all tokenivers, the premium for English is lower than the premium for the same language for the English-centric RoBERTa. Hence, we conclude that tokenizers for other languages give English preferential treatment.
5

---

Durmae
Cartra laya
(744 11 (62
nuaseni accec Lime [6]
23
Suntai
Sango
1.-
Padres
Script family:
Newser
hrarch
(Gamman
Standard
Rulgeiin
Armenian
Arati
Patuguor
(sponene
V Vetrumos
Greek
Tys VerC
XLM RoBERb
RuBeRTa
14
Sparten
nee Chincc
Sisppri
Georgan
Tokan zatio
Tokenisation length for FLORES-200 paralle
100
Figure 1: Comparison of the tokenization pre- miums for XLM-R and RoBERTa for the sub- set of languages that XLM-R encodes with less than 10% to the UNK token.
Figure 2: Average processing time and length of the tokenized inputs of RoBERTa. Each FLORES-200 sentence is processed for 20 in- dependent runs. The script family designa- tion is only for illustration purposes.
4.3 Parity for Multilingual Models
There has been a growing interest in multilingual language models, particularly for transla- tion (Dabre et al., 2020). As these models are intended to support a variety of languages, one would expect them to be close to tokenizer parity. We compare several such multi- lingual models: XML-R (Conneau et al., 2020), NLLB (Costa-jussà et al., 2022), M2M100 (Fan et al., 2021) and mT5 (Xue et al., 2020). All of these models use the SentencePiece tokemizer with upsampling for rare languages. The final model, BLOOM (Scao et al., 2022), uses byte-level BPE instead of SentencePiece and is designed to maintain similar ratios of tokens per word for each language as reference monolingual tokenizers.
BLOOM and NLLB encode all languages with less than 10% UNK tokens, respectively thanks to byte-level BPE tokenization and being trained on the same 200 languages as FLORES- 200 (see Table 4). The other three models fail to encode
least one language. All five models have languages with premiums of more than 2.5. Still, all models are better than the English-centric models in Table Figure 1 shows how XLM-R much closer parity than RoBERTa (on which it is based), over all languages it can encode. However, none of the models uniformly reaches parity across all languages. Therefore even models which are innall designed to be multilingual suffer from a lack of tokenization parity.
Summary: Multilingual models can improve the tokenization parity for different lan- guages but challenges remain in achieving tokenization parity across all languages.
4.4 Parity for Byte-level Tokenization Models
Byte-level representation is crucial for multilingual support, as it encodes any Unicode codepoint, even if unseen during training. One can also bypass vocabulary construction and directly employ the 256 byle values, enabling end-to-end training (byle-level lokenization). CANINE (Clark el al., 2022) rather than the byte level. The CANINE tokenizer is thus equivalent to the UTF-32 en- coding, resulting in an implicit tokenizer with a vocabulary of 1,114,112. ByT5 (Xue et al., 2022), the other hand, uses the UTF-8 encoding: implicit vocabulary of 256 tokens.5 5
large model that operates al the Unicode codepoint level
$To be consistent, we will refer to the characters and bytes in the encoding of the CANINE and ByT5 tokenizers as tokens as they fulfil similar role.

---

These byte-level models can represent any Unicode codepoint without an explicit tokeniza- tion step but there are still significant tokenization disparities. For CANINE, Shan has premium of 4.58 relative to Yue Chinese. This can be attributed to the fact that CANINE provides a single token for each Unicode codepoint, which results Chinese being more token-efficient (with a premium range 0.31-0.34 relative to English for the three Chinese languages) as each character is treated as a single token. This encoding also puts Shan at a disadvantage, as its encoding relies diacritics represented as separate Unicode codepoints. Other languages, such as Tok Pisin and Tumbuka, which use the Latin script but require more characters than English for the same text, also face similar challenges.
Tokenization disparity is also present in the ByT5 model. The tokenization premium for ByT5 ranges from 0.87 (for Yue Chinese) to 3.94 (for Shan). The introduction of the variable- width UTF-8 encoding of Unicode characters in ByT5 creates another issue of unequal treatment. ASCIT characters, which are sufficient for English, require only one byte. Other Latin script characters, as well as Greek, Cyrillic, Coptic, Armenian, Hebrew, Arabic and Syriac, require two bytes, while Chinese, Japanese and Korean characters require three bytes. Therefore, the tokenization for ByT5 as it is for CANINE (Table 5). Shan's premium of 3.94 is due to the fact that all its consonants and diacritics require three bytes. For example, the word "G:" is encoded by ByT3 as 12 tokens, whereas the corresponding "you" requires 3 tokens. The situation is similar for other languages like Dzongkha, Tibetan and Burmese.
Chinese and Japanese is about three times as long
Summary. Byte-level models also fail to achieve parity among the languages from FLORES-200 exhibiting a premium of over 4 times for some language pairs. There are two sources of multilingual tokenizer disparities. First, there are natural differences the number of characters used in different languages to communicate the same content. Second, the UTF-8 standard uses different number of bytes to encode codepoints of different scripts.
5 Fairness Implications of Tokenization Length Differences
We showed that no matter whether one uses subword, multilingual, or byle-level tokeniza- Lion, none of the tokenizers gets close to parily for all languages in FLORES-200. This lack of tokenization parity is not merely curiosity: it leads to unfairness in the cost to access language models, the latency of the service and the amount of data that can be processed.
5.1 Cost
is incrcasingly common to access LLMs as paid API services. One pricing approach, em- ployed by OpenAI at the time of writing, is to charge per token. Therefore, the tokenization premiums discussed Section directly map to cost premiums. For ChatGPT and GPT-4, the cost to process text in German or Italian is about 50% higher than to process the same text. in English (Table 1). Using them Dzongkha, Odia, Santali or Shan, the most expensive languages for these services, costs more than 12 times more than in English.
Another pricing strategy is per Unicode character: the approach currently taken by the Google Cloud Natural Language service. However, as we showed in Section 4.4, the same content can have very different lengths when mcasured in Unicode characters. Burmese, Dzongkha, Shan, Tok Pisin or Tumbuka require more than 4 times more characters than Yue Chinese for the same text, resulting in a proportional cost difference. Therefore, both the per-token and the per-character approaches result in large disparilies in the cost for users of different languages to use the exact same service.
5.2 Latency
High latency of real-time interactions for users of certain languages can result in a suboptimal experience and commuication breakdowns. For customer support emergency services, delays in response time can lead to miscommunication or delayed assistance.
htarpe://openai..om//pricc "nttps://cloud, www.coooo.oooggl...oooooo wwwggoo..ooooooooraaaaaaaaaaaaaa waw.urraaalannggggooao pricing

---

As some languages have significantly longer tokenized inputs, they would also experience longer processing times. The transformer attention mechanism has a quadratic complexity in the number of input tokens (Keles et al., 2023). However, the full model architecture contains other submodules and therefore the overall complexity might be different.
To assess the effect of the tokenization length on the latency, in Figure 2 we plot the com- putation time of RoBERTa against the tokenization lengths. It appears that the processing time is linear in the tokenization length rather than quadratic, showing a strong correla- tion between sequence length and execution time. Therefore, tokenization disparities across languages also affect the latency and processing time for text in these languages.
As expected, English is the left lower corner, having the shortest tokenization and one of the fastest. processing times. Shan is on the other extreme with the longest tokeniza- tion length and execution time (almost twice that of English). We can also observe clear trends dependent on the script used. Latin script and other Greek-derived scripts show the shortest tokenization lengths and processing times followed by the CchineseJappanes Korean (CJK) and Arabic languages. Other predominantly Asian and African scripts have longer tokenization lengths and processing times.
The latency implications of tokenization disparity are not limited to text models. Speech recognition models often produce series of tokens their output sequentially. Similarly, speech syuthesis takes as an input tokenized text (Latif et al., 2023). Therefore, differences in tokenization affect speech models too.
5.3 Long context processing
Transformers models have difficulty processing long inputs (Liu al., 2023). Given that the size of the input is contingent upon the tokenization process, inputs of greater length may impose a challenge for language models to adequately reason over. Such predicament may result in reduced abilities or limited applicability for languages with high tokenization premiums. For example, RoBERTa has a fixed block size of 512, GPT-2 has 768, 1024, 1280, or 1600 Radford et al. (2019), GPT-4 comes in 8,000 and 16,000 context variants." These models cannot process inputs longer than that. Therefore, one can process less than a tenth of the content in languages like Burmese and Dzongkha than they can in English.
Alongside inconveniencing the users of these languages, this can also result in diminished performance on automated systems, such as content moderation. Reliable content mod- eration is crucial for tackling hate speech and diminished performance has already been shown to fail to prevent its spread (Stecklow, 2018; Facebook, 2021). Therefore, reduced impacts.
long context capabilities for some languages could have severe
6 Towards Multilingual Tokenization Fairness
Section showed that high values of tokenization parity for a language lead to increased cost and latency and decreased capacity for long context processing. In this section, argue that training language models from scratch with a multilingually fair subword tokenizer is the only approach that can effectively address all these aspects of tokenization unfairness.
Subword tokenization is necessary achieve parity. In Section 4.4, we showed that neither character-level nor byte-level input representation can achieve tokenization parity. Therefore, variation of subword tokenization necessary. For example, Chinese characters could be individual tokens, Latin characters might be represented as tokens with average length of about characters while pairs of Burmese characters and their diacritics being assigned single tokens. Such approach would account for Chinese requiring one-third the characters English does (as shown in Table 5).
A separate tokenizer for determining the processing cost is not sufficient. An casy patch for existing models to use a separate tokenizer for calculating how much a user should be charged. Using one tokenizer for computing the cost and another to process
Pattps://openaa.com//rrr
8

---

80000
78000
7600C
FLORES"
10-fold reduction in the vocabulary would result in only 30% longer sequences for English.
72000
70000
Figure 3: How much longer will English language tokenization be if we dedicate fraction of the cl100k base vocab-
08000
66000
64000
one third vocabulary, English sequences will become just 10% longer for ChatGPT/GPT-4
ulary to other languages? This plot shows how many tokens will be neces- sary to encode the English language cor- pus of FLORRSS-200 for different sub- sets of the c1100k_ _asoe vocabulary.
62000
58000
50000
54000
52000
10000 20000 so0co 40000 s000a coooo 70000 s0c0a 10000000000
Vocabulary size
the input can easily be applied to existing systems without the need to retrain the LLM itself. However, the tokenizer for the language model is unchanged, this approach would still suffer from latency and inability to process long contexts. Therefore, to ensure similar processing times and long context capabilities across languages, the language model has to be trained with a multilingually fair tokenizer.
The tokenization needs to support all Unicode codepoints. Amongst all lokenizers we examine in this paper, the ones which encode all FLORES-200 languages all have one thing in common: they build their tokenization on top of Unicode representation, allowing them them to represent all characters. Therefore, a multilingually fair tokenizer should also start from Unicode (or equivalent) encoding. Considering that subword tokenization is necessary, building the vocabulary from UTF-8 would likely result in a smaller dictionary than building it top of UTF-32. Hence, UTF-8 is likely the more appropriate choice.
Building a multilingually fair parallel corpus. Building and evaluating multilingually fair lokenizers requires attention to the parallel corpus used. One must ensure a balanced representation of topics, otherwise, the resulting tokenizer might end up being multilingually fair only for subset of topics. The presence of named entities must also be balanced. For example, in FLORES-200, there are many English-centriy names and institutions, which might skew the results in favour of English. Additionally, the same sentence can have difforent translations with varying tokenization lengths. To account for this, a diversity of translations could ensure tokenization fairness across languages. These limitations also hold for the results in this paper. Hence, developing a well-curated and diverse parallel corpus is crucial for the development and evaluation of multilingually fair tokenizer.
Building a multilingually fair tokenizer from monolinugal tokenizers. As dis- cussed in Section 4, byte-level, character-level and word-level tokenizers cannot achieve tokenization parity and subword tokenization is needed. However, simply training a sub- word tokenizer balanced dataset is also not sufficient as languages can share tokens. For example, "hotel" is written the same way in English, Spanish, Italian, Portuguese, Dutch, Danish, Hungarian, Polish, etc. Hence, languages from more numerous language families will also witness shorter lokenization lengths while more isolated languages and scripts, e.g. Korean, would see larger language premiums: "hotel" in Korean is "ğ e and no other language has the same spelling as no other language uses the Korean script.
To address this issue, suggest two-stage process towards building multilingually fair tokenizer. First, train individual monolingual tokenizers for all target languages. Then, merge them while maintaining parity. The merging can be done by starting with the 256 tokens corresponding to each value a byte can take and then repcatedly adding the most frequently used token for the language with the highest premium.
While a multilingually fair tokenizer would lead to more tokens being needed for the dom- inant language, this additional cost would likely be much smaller than the benefit for the rest of the languages. The vocabulary size has diminishing returns: the additional tokens correspond to increasingly rare (parts of) words. For example, with only third of the vocab-
9

---

ulary, English sequences will become just 10% longer for ChatGPT/GPT-4 (see Figure 3). Therefore, by removing rarely used tokens of the dominant language and replacing them with frequently used tokens in other languages, we would likely see an overall net benefit.
7 Related Works
Fairness and bias in language models. The rapid increase in the size of language models has raised concerns regarding their biases and unfairness (Bender et al., 2021). For example, Bolukbasi et al. (2016), May et al. (2019) and Nadeem et al. (2021) showed that stereotypes and biases exist in language models, while Magee et al. (2021) identified the presence of intersectional biases which may be resistant to debiasing techniques. Language models were also shown to rely on social biases in question answering (Parrish et al., 2022). Another challenge the generation of toxic content which can occur even without prompting (Gchman ct al., 2020). Interestingly, Gururangan et al. (2022) point out that datasets consider one type of English as a higher quality depending on the location of the writer rather than on factuality or literary acclaim. Moreover, Ramesh et al. (2023) and Levy el al. (2023) highlighted the need to consider fairness issues of languages other than English, as they may have distinct sources of bias and solutions for English may not be applicable.
Multilingual performance. One approach towards similar multilingual performance is to frame languages as entities as recently proposed by Choudhury and Deshpande (2021). Another method is to separately train vocabularies for different language clusters to balance cross-lingual and language-specific tokens (Chung et al., 2020). Still, multilingual models struggle to deliver on the promises of deep transfer learning for lower-resouredd languages (Virtanen et al., 2019) and perform differently depending on the script and resource level of the language (Bang et al., 2023). Ahuja et al. (2023) found that generative models perform better on higher-resource languages and languages that use the Latin script, possibly due to the context length restrictions for some languages. Zhang et al. (2022) show that a balanced tokenizer corpus results in better translation performance. Separately, Hofmann et al. (2021, 2022) show that the BPE results in suboptimal token choices even for English and demonstrate that addressing this issue boosts performance. Similarly, Rajab (2022) and Oladipo et al. (2022) discuss how tokenization affects performance for African languages.
Measuring tokenization lengths. Zhang et al. (2022) suggested using the ratio of the average sentence length in tokens to the length in characters as measure of closeness to the character level. However, this method may not be suitable for comparing languages due to differences in sentence length across languages. On the other hand, Åcs (2019) and Scao et al. (2022) measure the number of tokens created per word, but this method may not be effective for comparing languages due to differences in semantic content per word and the lack of word delincation in some languages. Rust et al. (2021) show that mBERT (Devlin al., 2019) breaks down English words the least, in line with our findings of English receiving special treatment. However, to the best of our knowledge, we are the first to leverage a parallel corpus to compare tokenization lengths across languages.
8 Conclusion
This paper highlights the significant disparities in lokenization across different languages which can lead to unequal treatment and disadvantages for certain language communities. The findings reveal that even tokenizers explicitly trained for multilingual support exhibit tokenization lengths that vary by up to a factor of 13. Furthermore, character-level and byte-level models also demonstrate encoding length discrepancies that are more than 4 times longer. These disparities have important real-world implications including increased costs for accessing commorcial language services, longer processing times and limitations on the amount of contextual information provided to language models. To address these issues, we propose the development of multilingually fair tokenizers for future language models emphasizing the importance of ensuring comparable performance and accessibility across supported languages. By achieving tokenization parity, we can mitigate inequalities and promote fair access to language technologies across diverse linguistic communities.
10

---

Acknowledgements
We would like to thank Puyu Wang, Francisco Firas, Ambre Bertrand and Carmen Scheide- mann for their linguistic advice. Janet Pierrehumbert introduced us to many relevant prior works. We also extend special gratitude to Shinnosuke Takamichi and Hiroshi Saruwatari for open-sourcing the CPJD corpus for this project. Finally, we thank the reviewers; their leedback greatly improved this manuscript.
AB has received funding from the Amazon Research Awards. This work is supported by a UKRI grant Turing AI Fellowship (SP/W022811/1) and the EPSRC Centre for Doctoral Training in Autonomous Intelligent Machines and Systems (EP/S024050/1) 1). We also thank the Royal Academy of Engineering and FiveAI.
References
Ahmed Abdelali, Francisco Guzman, Hassan Sajjad, and Stephan Vogel. 2014. The AMARA corpus: Building parallel language resources for the educational domain. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC 14). European Language Resources Association (ELRA).
Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David R. Mortensen, Noah A. Smith, and Yulia Tsvetkov. 2023. Do all languages cost the same? Tokenization in the era of commercial language models. arXiv preprint KKi:2305513701
Kabir Ahuja, Rishav Hada, Millicent Ochicng, Prachi Jain, Harshita Diddco, Samuel Maina, Tanuja Ganu, Sameer Segal, Maxamed Axmed, Kalika Bali, and Sunayana Sitaram. 2023. MEGA: Multilingual evaluation of generalive AI. aixiv preprinl ar orxi::2303.129 125228
Jinze Bai, Shuai Bai, Shusheng Yang, Shijic Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-VL: A versatile vision-language model for understanding, localization, text reading, and beyond. arXie preprint ar Stw::2288.1266.
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung, 2023. multitask, multilingual, multimodal evaluation of ChatGPT on reasoning, hallu- cination, and interactivity. Xr preprint arXiv:2302.4408.
Loic Barrault, Yu-An Chung, Mariano Cora Meglioli, David Dale, Ning Dong, Paul- Ambroise Duquenne, Hady Elsahar, Hongyu Gong, Kevin Heffernan, John Hoffman, et al. 2023. S4T massively multilinguaa & multimodal machine translation. arXiv preprint arXiw:2308. 11596.
Reem Bassiouney. 2009. Arabic Sociolinguistics Edinburgh University Press.
Emily M. Bender, Timnit Gebru, Angelina McMillan-MMjjor and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedingg of the 2021 ACM Conference on Fairness, Accounlability, and Trunsparency,
Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. 2000. A neural probabilistic language model. Advances in Neural Information Processing Systems.
Tolga Bolukbasi, Kai- Chang, James Y Zou,, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. In Advances in Neural Information Processing Systems.
Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama. Obeid, Salam Khalifa, Fadhl Fryani, Alexander Frdmann, et al. 2018. The MADAR Arabic dialect corpus and lexicon. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018).
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. aiv preprint ar arXi::2107.033
11

---

Monojit Choudhury and Amit Deshpande. 2021. How linguistically fair are multilingual pre-trained language models? In Proceedings of the AAAI Conference on Artificial Intel- ligence.
Hyung Won Chung, Dan Garrette, Kiat Chuan Tan, and Jason Riesa. 2020. Improving multilingual models with language-clusteed vocabularies. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP).
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahmna, et al. 2022. Scaling instruction- fineluned language models. arXiv preprint or Xic:2210. 11116.
Junyoung Chung, Kyunghyun Cho, and Yoshua Bengio. 2016. A character-level decoder without explicit segmentation for neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa-
pers).
Jonathan H. Clark, Dan Garrette, Iulia Turc, and John Wieting. 2022. Canine: Pre-training Efficient Tokenization-Free Encoder for Language Representation, Tronsactions of the Association for Computational Linguistics.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wen- zek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoy- anov. 2020. Unsupervised cross- lerss-lingulllaa representation learning at scale. In Annual Meeting of the Association for Computational Linguistics.
Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint ar orXim:2207.0400
Raj Dabre, Chenhui Chu, and Anoop Kunchukuttan. 2020. A survey of multilingual neural machine translation. ACM Computing Surveys.
Raj Dabre and Aneerav Sukhoo. 2022. MorisienMT: A dalaset for Mauritian Creole machine translation. Xiv preprint ar arXiv:2206.022 02421.
Michel DeGrafl. 2007. Kreyôl Ayisyen, or Haitian Creole (Creole French).
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre- training of deep bidirectional transformers for language understanding, In Conference of the North American Chapler of the Associalion for Compulational Linguistics: Human Language Technologies.
Pelin Schöneerh Julian Mäder, and Thomas Hofmann. 2021. SwissDial: Parallel multidialectal corpus of spoken Swiss German. arXiv preprint 2:X.::2008.1140..
Tyna Eloundou, Sam Manning, Pamela Mishkin, and Daniel Rock. 2023. GPTs are GPTs: An carly look at the labor market impact potential of large language models. arXiv preprint arXiv:2303.100 10130.
Facebook. 2021. Sri Lanka human rights impact assessment. Accessed on April 11, 2023.
Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El- El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. 2021. Be- yond English-centric multilingual machine translation. The Journal of Machine Learning Research.
Philip Gage. 1994. A new algorithm for data compression. C Users Journal.
Yinggiang Gao, Nikola I. Nikolov, Yuhuang Hu, and Richard H.R. Hahnloser. 2020. Character-level translation with self-attention. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.
12

---

Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. RealToxicityromptts Evaluating neural toxic degeneration in language models. In Find- ings of the Association for Computational länguislies: EMNLP. Association for Compu- tational Linguistics.
Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Mare' Aurelio Ranzato, Francisco Guzmán, and Angela Fan. 2021. The FLORES- evaluation benchmark for low-rcsource and multilingual machine translation. Transactions of the Association for Computational Linguistics.
Suchin Gururangan, Dallas Card, Sarah K. Dreier, Emily K. Gade, Leroy Z. Wang, Zeyu Wang, Luke Zettlemoyer, and Noah A. Smith. 2022. Whose language counts as high quality? Mcasuring language idcologies in text data selection. arXiv preprint arXiv:2201. 104444001.044
Francisco Guzmán, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, and Marc' Aurelio Ranzato. 2019. Two new evaluation datasets for and Sinhala-Engliste In Proceedings of low-resource machine translation: Methods in Natural Language Processing and the 9th the 2019 Conference Empirical
International Joint Conference on Natural Language Processing (EMNLP-IJCNLP))
Shiro Hattori. 1973. Japanese dialects. In Diachronic, areal, and typological linguistics.
Valentin Hofmann, Janet Pierrehumbert, and Hinrich Schütze. 2021. Superbizarre is not superb: Derivational morphology improves BERT's interpretation of complex words. In Proceedings of the 59th Anreual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers).
Valentin Hofmann, Hinrich Schuctze, and Janct Pierrchumbert. 2022. An embarrassingly simple method to mitigate undesirable propertics of pretrained language model tokenizers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).
Michael A. Hogg, Nicholas Joyce, and Dominic Abrams. 1984. Diglossia in Switzerland? A social identity analysis of speaker evaluations. Journal of Language and Social Psychology.
Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, and Zhaopeng Tu. 2023. Is ChatGPT a good translator? Yes with GPT-4 as the engine. arXiv preprial trXXt:2301.087755
Johnson. 2023. ChatGPT is a marvel of multilingualism. The Economist.
Alan S. Kaye. 2001. Diglossia: The state of the art. International Journal of the Sociology of Language.
Feyza Duman Keles, Pruthuvi Mahesakya Wijewardena, and Chinmay Hegde. 2023. On the computational complexity of self-attention. In International Conference on Algorithmic Learning Theory.
Simran Khanuja, Diksha Bansal, Sarvesh Mchtani, Savya Khosla, Atreyee Dey, Balaji Gopalan, Dilip Kumar Margam, Pooja Aggarwal, Rajiv Teja Nagipogu, Shachi Dave, Shruti Gupta, Subhash Chandra Bose Gali, Vish Subramanian, and Partha Taluk- dar. 2021. MuRIL: Multilingual representations for Indian languages. arXiv preprint arXio:2103.107330
Wojeiech Kryšciñski, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Neural text. summarization: A critical evaluation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th In- lernational Joint Conference on Natural Language Processing (FMNLP-IJCNIP)) Hong Kong, China.
13

---

Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Compulational Linguisties (Volume 1: Long Papers).
Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural lext processing. In Proceedings of the 2018 Conference Empirical Melhods in Natural Language Processing: System Demonslra-
tions.
Mohammad Amin Kuhail, Nazik Alturki, Salwa Alramlawi, and Kholood Alhejori. 2023. Interacting with educational chatbots: A systematic review. Education and Information Technologies.
Siddique Latif, Aun Zaidi, Heriberto Cuayahuitl, Fahad Shamshad, Moazzam Shoukat, and Junaid Qadir. 2023. Transformers in speech procossing: A survey. arXiv preprint arXio:2308.1100
Jason Lee, Kyunghyun Cho, and Thomas Hofmann. 2017. Fully character-level neural machine translation without explicit segmentation. Transactions of the Association for Computational Linguistics.
Peter Lee, Sebastien Bubeck, and Joseph Petro. 2023. Benefits, limits, and risks of GPT-4 as an Al chatbot for medicine. New England Journal of Medicine.
Heather Lent, Emanuele Bugliarello, Miryam de Lhoneux, Chen Qiu, and Anders Søgaard. 2021. On language models for creoles. In Proceedings of the 25th Conference on Compu- Intional Natural Language Learning. Association for Computational Linguistics.
Heather Lent, Kelechi Ogueji, Miryam de Lhoneux, Orevaoghene Ahia, and Anders Sogaard. 2022. What creole wants, what creole needs. In Proceedings of the Thirleenth Language Resources and Evaluation Conference.
Janny Leung. 2022. Shortcuts and shortfalls in Meta' s content moderation practices: A glimpse from its oversight board's first year of operation. Comparative Law and Language.
Sharon Levy, Neha Anna John, Ling Liu, Yogarshi Vyas, Jie Ma, Yoshinari Fujimuma, Miguel Ballesteros, Vittorio Castelli, and Dan Roth. 2023. Comparing biases and the impact of multilingual training across multiple languages. arXic preprint or XXxx:2305.11222
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023. Lost in the middle: How language models use long contexts. Xix preprint or arxiv22307.0311
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly opti- mized bert pretraining approach. arXiv preprint orXiv:1907.11690
Georges Lüdi. 2007. The Swiss model of plurilingual communication. Receptive multilin- gualism: Linguistic analyses, language policies and didactic concepts.
Sophie Lythreatis, Sanjay Kumar Singh, and Abdul-Nasser Fl-Kassar. 2022. The digital divide: A review and future rescarch agenda. Technological Forecnsting and Social Change.
Liam Magee, Lida Ghahremanlou, Karen Soldatic, and Shanthi Robertson. 2021. Intersec- lional bias in causal language models. Xrviv preprint ur mxXx::2177.00000
Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric Villemonte de La Clergerie, Djamé Seddah, and Benoît Sagot. 2020. CanemBERT: tasty French language model. Tn Annual Meeling of the Association for Compulational Linguisties.
14

---

Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, and Rachel Rudinger. 2019. On measuring social biases in sentence encoders. In Proceedings of the 2019 Conference of the North American Chopter of the Association for Compulational Linguislies: Human Language Technologies, Volume (Long and Short Papers).
Sabrina J Mielke, Zaid Alyaleai, Elizabeth Salesky, Colin Raffel, Manan Dey, Matthias Gallé, Arun Raja, Chenglei Si, Wilson Y Lee, Benoît Sagot, et al. 2021. Between words and characters: A brief history of open-vocabulary modeling and tokenization in NLP. arxiv preprint arXiv:2112.1050
Rob Miles and Sean Riley. 2023. Glitch tokens Computerphile. Accessed on April 11, 2023.
Robert Munro. 2010. Crowdsourced translation for emergency response in Haiti: the global collaboration local knowledge. In Proceedings of the Workshop on Collaborative Trans- lation: technology, crowdsoureing, and the translator perspective. Association for Machine Translation in the Americas.
Picter Muysken and Norval Smith. 1994. The study of pidgin and creole languages. In Pidgins and creoles: An introduction.
Moin Nadeem, Anna Bethke, and Siva Roddy. 2021. StereoSet: Mcasuring stercotypical bias in pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers).
Dat Quoc Nguyen and Anh-Tuan Nguyen. 2020. PhoBERT: Pre-trained language models for Vietnamese. In Findings of the Association for Compulational Linguislies: EMNLP.
Akintunde Oladipo, Odunayo Ogundepo, Kelechi Ogueji, and Jimmy Lin. 2022. An explo- ration of vocabulary size and transfer elfects in multilingual language models for African languages. In 3rd Workshop on Africon Nalural Language Processing.
OpenAl. 2022.
Introducing ChatGPT. Accessed on April 11, 2023.
OpenAl. 2022. tiktoken. Git commit: 82facf9.
OpenAl. 2023. GPT-4 technical report. arXie preprint arXiv:2303.08774
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training lan- guage models follow instructions with human feedback. Advances in Newral Information Processing Systems.
Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. 2022. BBQ: A hand-built bias bench- mark for question answering. In Findings of the Association for Computational Linguistics: ACL 2022.
Jonas Pfeiffer, Ivan Vulié, Iryna Gurevych, and Sebastian Ruder. 2021. UNKs everywhere: Adapting multilingual language models to new scripts. In Proceedings of the 2021 Con- ference on Empiriaal Methods in Natural Language Processing.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Nyy Sulskever. 2019. Language models are unsupervised multitask learners.
Jenalea Rajab. 2022. Efflect of tokenisation strategies for low-resourced Southern African languages. In 3rd Workshop on African Natural Language Processing.
Krithika Ramesh, Sunayana Sitaram, and Monojit Choudhury. 2023. Fairness in language models beyond English: Gaps and challenges. Tn Findings of the Association for Compu- totional Linguistics: EACL 2023. Association for Computational Linguistics.
15

---

Surangika Ranathunga, En-Shiun Annie Lee, Marjana Prifti Skenduli, Ravi Shekhar, Mehreen Alam, and Rishemjit Kaur. 2023. Neural machine translation for low-resource languages: A survey. ACM Compuling Surveys.
Jessica Rumbelow and Matthew Watkins. 2023a. SolidGoldMaggkarp II: Glitch token archaelogy. Accessed on April 11, 2023.
Jessica Rumbelow and Matthew Watkins. 2023b. SolidGoldMagikarp (plus, prompt gener- ation). Accessed on April 11, 2023.
Charles Russ. 1990. The Dialects of Modern German: A Lènguistic Survey.
Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. 2021. How good is your tokenizer? On the monolingual performance of multilingual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers).
Ali Safaya, Moutasem Abdullatif, and Deniz Yuret. 2020. KUTSAIL at SemFval-2020 task 12: BERT-CNN for offensive speech identification in social media. In Proceedings of the Fourteenth Workshop on Semantic Evaluation.
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. BLOOM: A 176a-parameter open-acoess multilingual language model. arXiv preprint @rXim:2211.05100
Raphael Scheible, Fabian Thomezyk, Patric Tippmann, Victor Jaravine, and Martin Bocker. 2020. GottBERT: A pure German language model. arXiv preprint arXiv:2012.022
Mike Schuster and Kaisuke Nakajima. 2012. Japanese and Korcan voice search. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Anreual Meeting of the Association for Compulational Linguislies ( (Volume 1: Long Pupers).
Picter A. M. Seuren. 1995. Notes on the history and the syntax of Mauritian Creole. Lin- guistics.
Yan Shao, Christian Hardmeier, and Joakim Nivre. 2018. Universal word segmentation: Implementation and interpretation. Transactions of the Association for Computational Linguisties.
Peter Sieber and Horst Sitta. 1987. Deutsch in der Schweiz. Zeilschrift für Germanislik.
Felix Stahlberg, 2020. Neural machine (ranslation: A review. Journal of Arlificial Intelli- gence Research.
Steve Stecklow. 2018. Hatebook. Reulers. Accessed on April 11, 2023.
Hui Su, Weiwei Shi, Xiaoyu Shen, Zhou Xiao, Tuo Ji, Jiarui Fang, and Jie Zhou. 2022. RoCbert: Robust Chinese BERT with multimodal contrastive pretraining. In Annual Meeting of the Association for Computational Linguistics.
Lichao Sun, Kazuma Hashimoto, Wenpeng Yin, Akari Asai, Jia Li, Philip Yu, and Caim- ing Xiong. 2020. Adv Ady-BERT: BERT is not robust on misspellings! Generating nature adversarial samples on BERT. Xr preprint Nw:2003304055
Shinnosuke Takamichi and Hiroshi Saruwatari. 2018. CPJD corpus: Crowdsourced parallel speech corpus of Japanese dialects. In Proceedings of the Fleventh International Confer- ence Language Resources and Evoluation (LREC 2018).
16

---

Yuging Tang, Chan Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chandhary, Jiatao Gu, and Angela Fan. 2020. Multilingual translation with extensible multilingual pretraining and finetuning. arXiv preprint ar Xrv:20008. 00401.
Timm Teubner, Christoph M Flath, Christof Weinhardt, Wil van der Aalst, and Oliver Hinz. 2023. Welcome to the era of ChatGPT et al: The prospects of large language models. Business & Information Systems Engineering.
The Unicode Consortium. 2022. The Unicode standard, Version 15.0.0.
Jörg Tiedemann. 2012. Parallel data, tools and interfaces in OPUS. In Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC 12).
Tohoku NLP Group. 2019. BERT models for Japanese NLP.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, hée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. LLaMA: Open and efficient foundation language models. arXiv preprint arXic:2302.1530
Emma Trentman and Sonia Shiri. 2020. The mutual intelligibility of Arabic dialects: Impli- cations for the language classroom. Orilical Multilinguualism Studies.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advonces in Neural Information Processing Systems.
Antti Virtanen, Jenna Kanerva, Rami No, Jouni Luoma, Juhani Luotolahti, Tapio Salakoski, Filip Ginter, and Sampo Pyysalo. 2019. Multilingual is not enough: BERT for Finnish. arxio preprint arxiv:1912 07076.
Jonathan J. Webster and Chunyu Kit. 1992. Tokenization as the initial phase in NLP. In The International Conference Computational Linguistics.
Jiacheng Xu, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. Discourse-aware neural extractive lext summarization. In Proceedings of the 58th Annual Meeling of the Association for Computational Linguistics. Association for Computational Linguistics.
Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2022. ByT5: Towards a token-free future with pre- trained byte-to-byte models. Transactions of the Association for Computational Linguis- lics.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2020. m A massively multilingual pre-trained text- to-text transformer. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
Joseph K. Yamagiwa. 1967. On dialect intelligibility in Japan. Anthrropologcall Linguistics.
Shiyue Zhang, Vishrav Chaudhary, Naman Goyal, James Cross, Guillaume Wenzek, Mo- hit Bansal, and Francisco Guzman. 2022. How robust is neural machine translation to language imbalance in multilingual tokenizer training? In Proceedings of the 15th Bien- nial Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track).
Judit Ács. 2019. Exploring BERT's vocabulary. Accessed April 11, 2023.
Slavomir Čéplö, Ján Bátora, Adam Benkato, Jifí Milička, Christophe Pereira, and Petr Zemánek. 2016. Mutual intelligibility of spoken Maltese, Libyan Arabic, and Tunisian Arabic functionally tested: A pilot study. Folia Linguistica.
17

---

A Background on Tokenization
To enable automatic processing of language, it must first be represented in a suitable form. The current practice is to use tokenization which is the process of turning natural language into sequences of tokens coming from a finite and pro-dctermined set called vocabulary (Webster and Kit, 1992). Each Loken is lypically associated with an integer value. Language models process such sequences of integers, rather than sequences of characters or words. In this section, we offer brief overview of the contemporary tokenization methods. For further details, we recommend the comprehensive survey by Mielke et al. (2021).
Word tokenization. The simplest Lokenization method is splitting al while spaces, where each word is assigned its own token (Bengio et al., 2000). This approach, however, requires that all possible words are in the vocabulary which is not possible in practice. Therefore word tokenization often fails to handle cases like "won't" "von't", words spelled with accented characters like "naive" or "açaf", speling mistakes and named entities like "Cootonnshopeburnoor"" (Sun ct al., 2020). This makes unsuitable for representing open vocabularies, where the words encountered are not limited to a predetermined set. Furthermore, languages that do not use spaces to separate words, such as Chinese, Japanese and Burmese, pose additional challenges for this approach (Shao et al., 2018).
Subword tokenization. Hence, most current models use subword tokenization, where complex words are broken down into multiple tokens. Subword tokenization can efficiently handle complex terms by breaking them down into parts, e.g,, "Cotlonshhopeburuooo"" "sh""""""""""""""""""""""""
This approach can represent novel words, including misspelled ones, in open vocabulary setling.
Subword vocabularies are usually based approaches which use large corpora to learn which subword sequences occur frequently in practice. Schuster and Nakajima (2012) in- troduced one of the first subword tokenizers, WordPiece, as a way handle Japanese and Korcan. Scnnrich et al. (2016) proposed using Byte-Pair Encoding (BPE) (Gage, 1994) for learning subwords by merging the most froquently occurring pairs. BPE has since been widely used for most of the popular tokenizers. Kudo (2018) proposed an alternative ap- proach via gradually pruning a large vocabulary. It removes lokens that are less likely to improve the performance of simple unigram language model. Both methods rely on pre- tokenization (splitting on whitespaces, when available), which not an invertible process. SentencePiece (Kudo and Richardson, 2018) addresses this de-tokenization ambiguity by treating whitespace as a special symbol, including in the vocabulary, and supports both methods. SentencePiece with BPE is by far the most popular tokenization method for the models considered in this paper.
Unicode support. Even subword tokenization ensures that individual characters are in the vocabulary, this still leaves the question of which characters are to be included. Simple solution is to take the ASCII characters. However, this mcans that words in other scripts or accented letters will fall out of it. A common workaround is to represent strings outside the vocabulary as a special UNK token. However, if there are too many UNK tokens in an input, the performance of the model tends to deteriorate (Pfeiffer et al., 2021). Therefore, it is desirable that the number of UNK tokens in the input is kept. as low as possible. A simple and commonly used solution is to base the vocabulary building on Unicode.
Unicode is a computing industry standard for representing text characters (The Unicode Consortium, 2022). Unicode supports virtually all languages (including many ancient ones, emojis and special characters) by assigning every grapheme, modifier, punctuation mark, control character or formatting character one of 1,114,112 integer codepoints. The codepoints can be represented in binary as the variable-width encoding UTF-8, which encodes every codepoint with one to four bytes, or the fixed-width UTF-32 which encodes all codepoints with four bytes (see Figure 4).
UTF-8 can therefore represent any string in any language as string of bytes. As each byte can take only one out of 256 values, 256 tokens can be sufficient to encode all texts. In practice this is usually combined with the BPE tokenizer. At first, the corpus is en-
18

---

A 00000041
Ω
III
UTF-32
03000348
00C05ASE
00010364
A 41
III BE
UTF-8
CE
AB
E8
M
SE
FD 90
84
Figure 4: Comparison of variable width Unicode encoding (UTF-8) and fixed width encoding (UTF-32), Image adapted from (The Unicode Consortium, 2022).
coded UTF-8 bytes and then BPE is ran on lop of it. As most characters oceur fre- quently, BPE would assign them a dedicated token. If the model encounters a character that didn't exist in the training corpus (e.g., the medium skin tone waving hand D, can still represent it byte-by-byte 9F-+9F- for the waving hand and F0+9F +F for the skin tone modifier). This allows the vocabulary efficiently represent frequently occur- ring words and rare characters, For example, the sentence "1 love açaf" could be tokenized as "I "+"love "==""""3++7+"a""""
Byte-level and character-level tokenization. If we can represent any input with just 256 characters, then why bother with subword tokens? A key consideration is sequence length. This is since transformers (Vaswani et al., 2017), the currently predominant deep learning architecture for language models, have attention layers with quadratic complexity in the input length. Hence, as the number of characters is much longer than the sub-word tokenization, working on the character level has been traditionally considered computation- ally inefficient. However, Chung et al. (2016), Lee et al. (2017), Gao et al. (2020), Clark et al. (2022) and Xue al. (2022) proposed various architectures working around this issue and operating directly on characters or UTF-8 bytes.
B Parity for Linguistic Varieties
A language can vary according to factors such geography, history, social class and culture. As a result, different dialects, pidgin and creole language variations emerge, each with its own distinct set of grammar, vocabulary and pronunciation rules. 9 Unequal treatment of certain dialects or languages can lead to social and economic disadvantages for those who speak them. Therefore, it is important to also study the tokenization differences between the "standard" language and its varieties." 10 Unfortunately, parallel corpora for dialects, pidgin and creole language variations are far and few in between. In this section, however, we show results on regional Swiss German varieties, Arabic and Japanese dialects, as well as Haitian and Mauritian creoles.
Swiss German dialects. Swiss German is a dialect continuum which significantly differs from the formal High German German-speaking Switzerland is diglossic: High German is used alongside regional dialects (Hogg et al., 1984). In contrast. to other dialects, the use of Swiss dialects is increasing (Sieber and Sitta, 1987) especially online (Lüdi, 2007). Swiss German dialects are often considered unintelligbbe to High German speakers and sometimes even speakers of different dialects may find difficulty understanding each other (Russ, 1990). Therefore, ensuring that German-targeting NLP applications can process Swiss German dialccts is important.
To this end, we compare the tokenization parity relative to High German of GottBERT (Scheible et al., 2020) on the regional dialects of Aargau, Bern, Basel, Graubünden, Luzern,
"While no standard definitions exist, dialects usually considered be regional variations of a language, whereas pidgin and creole languages are contact languages that emerge from the interaction of speakers of different languages (Muysken and Smith, 1994).
We refer to the language that the datasets label as "standard", "official" "dominant" without necessarily endorsing this designation.
11 "Diglossia is the situation of two dialects or languages being used by a single language commu- nity (Kaye, 2001).
19

---

Table 6: GottBERT tokenizer premiums on the SwissDial dataset for Swiss German dialects.
The premium is computed with respect to High German.
Region
GottBERT parity
High German
1.00
Zürich
1.38
St. Gallen
1.40
Basel
1.41
Granbünden
1.44
Luzern
1.52
Aargau
1.53
Wallis
1.58
Bern
1.59
Table ArabicBERT tokenizer premiums on the MADAR dataset for Arabic dialects.
The premium is computed relative to Standard Arabic.
City
ArabicBERT
City
ArabicBERT
Jeddah
0.91
Sanaa
1.01
Doha
0.92
Beirut
1.02
Riyadh
0.92
Benghazi
1.02
Muscat
0.94
Cairo
1.03
Basra
0.95
Sfax
1.03
Salt
0.95
Tripoli
1.05
Baghdad
0.96
Aswan
1.06
Damascus
0.97
Alexandria
1.06
Aleppo
0.97
Tunis
1.06
Jerusalem
0.97
Algiers
1.07
Khartoum
0.98
Mosul
1.10
Amman
0.99
1.11
Std. Arabic
1.00
Rabat
1.17
St. Gallen, Wallis and Zürich. We use SwissDial, parallel multidialectal corpus, the Schönberger et al., 2021). It is worth noting, that the dialect
basis of comparison of each city and its corresponding region may differ significantly. Therefore there might be large variations within regions as well.
The results in Table 6 show disparity between the tokenization lengths for High Ger- man and the Swiss dialects with premium ranging from 1.38 for the Zürich dialect, or Zürätüütsch, to 1.59 for the Bernese Bürndütsch. In fact, English has lower premium than any Swiss dialect (1.35 on FLORES-200, Table 2) and the premium for Bernese German close to the linguistically further Swedish (1.64) and Norwegian Bokmål (1.65). The follow- ing example from SwissDial shows how the sentence "Like he's 's waiting for something" has almost twice long tokenization in Bernese German compared to High German:
963 Als
15628
63
145
warte
auf
etwas
1134 8808
226
751
2912
13621
288
361
67
11769
Aus
wür
der
uf
P is
war
tä
The fact that the GottBERT tokenizer results in better parity for English, Swedish and Norwegian Bokmål than for Swiss German dialects highlights that it does not likely pick out stable linguistic constructs.
Arabic dialects. Similarly to Swiss German, Arabic is usually spoken in diglossic speech communities, where Modern Standard Arabic is spoken alongside least one prestigious vernacular particular to the country or region (Bassiounyy, 2009). As both Standard Arabic
20

---

Table 8: BERT Japanese tokenizer premiums on the CPJD dataset for Japanese dialects. The premium is computed with respect to Standard Japanese. The CPJD dataset consists of two parallel corpora with the dialects split across the two. Hence, we have also indicated the corpus for each dialect. Nara-ben has two entries as the dataset has transcriptions for two separate speakers. The suffix "-ben" (ft) means "speech" or "dialect".
Dialect
Corpus Parity
Dialect
Corpus Parity
Akita-ben
1.09
Miyazaki-ben
1.05
Awa-ben
2
1.09
Morokata-ben
1.15
Fukui--en
2
1.04
Nara-ben
2
1.09
Fukuoka-ben
1.03
Nara-ben
1.03
Hiroshima-ber
1.02
Okayana-
1.15
Hokkaido-ben
2
1.06
Oosaka-ben
1.03
Iwaki-ben
2
1.08
Saitama-ben
1.01
Iyo-ben
1.05
Tosa-ber ben
1.03
Izumo-ben
1.10
Toshu- ben
1.06
Kanazawa-ben
1.11
Tsugaru-ben
1.09
Kyokotoba
2
1.07
and its dialects are commonly used in written communication, is vital that tokenizers handle them equally well.
To assess the performance of Arabic tokenizers, we compare the tokenization lengths of ArabicBERT (Safaya et al., 2020) across 25 Arabic dialects. To this end, we use the MADAR parallel corpus of Arabic dialects (Bouamor et al., 2018).
Table shows the premiums relative to Standard Modern Arabic. The premium varies from 0.91 for the Jeddah dialect to 1.17 for the Rabat dialect. This is significantly lower than the premium for English (1.83 on FLORES-20 200 Table 2). The range is also much smaller than for the Swiss German dialects and approximately half of the considered dialects have a lower premium than Standard Modern Arabic. Therefore, one could say that the tokenizer of ArabicBERT achieves tokenization parity for these 25 Arabic vernaculars. This is likely because the corpus and vocabulary set on which ArabicBERT was trained contained dialectical Arabic. It is also possible that Arabic dialects are closer to Modern Standard Arabic and more mutually intelligible than Swiss German dialects are to High German (Céplö et al., 2016; Trentman and Shiri, 2020). Still, this difference between the parity for Swiss and Arabic dialects indicates that including broader set of vernaculars and dialects in the corpus results in improved tokenization parity.
Japanese dialects. Japanese also has number of regional dialects (Hattori, 1973). We compare the woknizatiion parily of BERT Japanese (Tohoku NLP Group, 2019) across them. We employ the CPJD dataset by Takamichi and Saruwalari (2018) which contains transcrip- tions of the voice recordings of 250 sentences across 20 dialects.
The results in Table 8 show that the premium compared to Standard Japanese (Tokyo dialect) ranges from 1.01 (for Saitama prefecture, neighbouring Tokyo) to 1.15 (for Morokata- ben and Okayama-ben). These all are significantly lower than the premium for English (1.49, as shown in Table 2). Therefore, similarly to ArabicBERT, this is an example of the tokenizer being relatively well-aligned with the dialects. This is likely because Japanose dialects are more closely related (and intelligible (Yamagiwa, 1967) to Standard Japanese speakers) than the Swiss dialects are to High German speakers.
Mauritian and Haitian Creoles. While creoles often have some similarities with a high- resource language (usually English or French), the differences are significant to necessitate special attention to their support (Lent et al., 2021, 2022). This is especially critical for emergency services and disaster management (Munro, 2010).
Mauritian Creole is based on French well as the languages of slaves imported from Mada- gascar and East Africa. As the British gained control of Mauritius, they brought indentured labourers from India who further had effect on the formation of the modern Mauritian
21

---

Creole (Seuren, 1995). Similarly, Haitian Creole (Kreyol) emerged from the interaction of French and the various Niger-Congo languages spoken by the Africans brought slaves (DeGraff, 2007).
Considering that both languages have their basis in French, one would expect that tok- enizers targeting French would have low tokenization parities for Mauritian and Haitian Creoles. However, taking the tokenizer of CamemBERRI (Martin al., 2020), the premium for Mauritian Creole is 1.20 using the MorisienM'T parallel corpus (Dabre and Sukhoo, 2022). The premium for Haitian Creole 1.64 when using the QEDv2 corpus (Tiedemann, 2012; Abdelali et al., 2014). Haitian Creole is also represented in the FLORES-200 dataset where the premium relative to French is 1.58. This is significantly larger than linguistically further languages such as English (1.20), Pangasinan (1.49) and Nigerian Fulfulde (1.54). Therefore, CamemBERT not well-placed to tokenize French-related creoles despite the model being trained for French.
C Extended Tables of Tokenization Premiums
In addition to the models presented in the main text, these extended tables also include LLAMA (Touvron et al., 2023), MBart50 (Liu et al., 2020; Tang et al., 2020), SeamlessM4T (Barrault et al., 2023) and Qwen-VL (Bai et al., 2023).
22

---

XLM--RHHERIS
cLLOOK basse
CamemBER
Language
ILAMA
r50k buse
p50k baso
p50k cilic
ROBERTA
GORBERE
PhoRERT
RoCBert
M2M100
GPT-2
Acchnese (Arabic script)
4.00
4.78
4.78
4.78
4.78
3.78
4.78
4.95
1.94
1.89
Acehmese (Latin script)
1.89
2.16
2.16
2.16
2.16
1.98
2.16
1.56
1.55
1.37
1.10
1.57
Mesopotamian Arabic
3.34
4.27
4.27
4.27
4.27
2.99
4.27
5.10
1.16
1.27
Taniiii- Ademi Arabic
3.38
4.34
4.34
4.34
4.34
3.01
4.34
5.16
1.17
1.28
Tunisian Arabic
3.31
4.20
4,20
4.20
5.03
Afrikcans
1.55
1.94
1.94
1.94
1.94
1.69
1.94
1.25
1.38
1.26
1.06
1.20
1.22
South Levantine Arabic
3.20
4.02
4.02
4.02
4.02
2.84
4.02
4.84
1.12
1.22
Akan
2.20
80
2.80
2.80
2.68
2.
1.90
1.61
1.45
1.98
1.83
Tosk Albanian
2.26
2.65
2.65
2.65
2.65
2.25
2.65
1.77
1.82
1.69
1.12
1.32
1.36
Ambaric
7.32
7.79
7.79
7.79
7.79
7.68
7.79
5.19
1.34
1.42
North Levantine Arabic
3.19
4.04
4.04
4.04
4.01
2.83
4.04
4.83
1.15
1.24
Standard Arabic
3.42
4.40
4.40
4.41
4.40
3.04
4.40
5.21
1.18
1.29
Standard Arabic (Romanized)
2.31
2.51
2.51
2.51
2.51
2.45
2.51
1.76
1.72
1.55
1.19
1.94
1.83
Najdi Arabic
3.43
4.41
4.41
4.41
4.41
3.04
4.41
5.22
1.18
1.30
Moroccan Arabic
3.35
4.21
1.21
4.21
1.21
2.96
4.21
5.08
1.25
1.33
Egyptian Arabic
3.36
4.23
4.23
4.23
4.23
2.96
4.
5.10
1.17
1.27
Assamese
6.14
9.79
9.79
9.78
9.78
6.20
9.79
8.32
1.90
2.24
Asturian
1.89
1.89
1.89
1.89
1.31
1.04
1.27
Awadhi
4.53
7.19
7.19
7.19
7.19
4.78
7.19
8.19
1.37
1.47
Central Aymara
2.03
2.32
2.32
2.33
2.32
2.17
2.332
1.62
1.62
1.47
1.09
1.70
1.64
South Aserbaijani
3.7G
5.16
5.16
5.16
3.34
5.32
1.50
North Azerboijani
2.61
3.47
3.47
3.47
3.47
2.64
3.47
2.31
1.90
1.15
1.26
Bashkir
2.91
6.01
6.01
6.01
6.01
4.28
6.01
3.97
2.06
1.23
Bambara
1.99
2.66
2.66
2.66
2.66
2.57
2.66
1.84
1.54
1.40
1.82
1.72
Balinese
1.77
1.97
1.97
1.97
1.97
1.80
1.97
1.39
1.43
1.28
1.14
1.32
1.29
Belarusian
2.38
6.56
6.56
1.46
Bemba
2.15
2.46
2.46
2.46
2.46
2.23
2.46
1.69
1.68
1.53
1.26
1.76
1.67
Bengali
5.38
9.65
9.65
9.65
9.65
5.84
9.65
8.54
1.38
1.55
Bhojpuri
1.52
7.18
7.18
7.18
7.18
$.08
1.47
1.51
Banjer (Arabic script)
4.22
5.03
5.03
5.03
5.03
3,80
5.03
5.53
1.92
1.93
Banjar (Latin script)
1.75
1.98
1.98
1.98
1.98
1.98
1.38
1.35
1.21
1.08
1.21
1.16
Standard Tibetan
6.67
14.93
14.913
14.93
14.91
11.27
14.33
10.87
Bosnian
1.69
2.19
2.19
2.19
2.19
1.87
2.19
1.17
1.16
1.35
1.02
1.12
1.17
Huginese
2.20
2.20
1.45
1.51
Bulgarian
1.78
5.51
5.51
5.51
5.51
2.64
5.51
3,51
2.57
1.16
1.23
Catalan
1.51
1.92
1.92
1.992
1.92
1.71
1.92
1.40
1.33
1.31
1.10
1.26
1.26
Cebuano
1.96
2.24
2,24
2.24
2.24
1.57
1.59
1.20
1.52
1.38
Ciech
1.69
2.62
2.62
2.62
2.62
2.11
2.62
1.73
1.48
0.99
1.17
1.23
Chokwe
1.91
2.16
2.16
2.16
2.16
1.98
2.16
1.51
1.49
1.32
1.10
1.55
1.47
Central Kurdish
4.43
6.49
6.49
6.49
6.49
4.80
6.49
5.82
2.30
2.48
Crimean Tatar
2.13
2. 19
2.49
2.49
2.49
2.12
2.19
1.G7
1.68
1.54
1.38
1.37
Websh
2.09
2.34
2.34
2.34
2.34
1.68
1.53
1.06
1.43
1.44
Danish
1.54
1.90
1.90
1.90
1.90
1.62
1,90
1.26
1.39
1.29
1.01
1.09
1.12
German
1.41
2.14
2.14
2.14
2.14
1.58
2.14
0.74
1.55
1.40
1.20
1.17
1.24
Southwestern Dinka
1.88
18
2.48
2.48
2.25
1.60
1.43
1.32
0.75
1.68
1.55
Dyula
1.88
2.
2.20
2.20
2. 2.20
2.05
2,20
1.54
1.43
1.30
0.98
1.65
1.53
Dzongkha
7.42
16.36
16.36
16.36
16.36
12.33
16.36
11.95
Greek
4.99
6.54
6.54
6.54
6.54
5.15
6.54
4.99
3.11
1.15
1.45
1.58
English
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
Esperanto
2.03
2.03
2.03
2.03
1.35
Estonian
1.76
2.11
2.11
2.11
2.11
1.87
2.11
1.39
1.42
1.33
1.03
1.12
1.20
Basque
1.79
2.10
2.10
2.10
2,10
1.88
2.10
1.39
1.44
1.33
1.11
1.16
1.23
Ewe
2.28
2.90
2.90
2.90
2.75
2,00
1.97
1.69
1.46
2.01
1.86
Faroese
1.92
2.38
2.38
2.38
2.38
2.07
2.38
1.66
1.64
1.46
1.44
1.41
Fijian
2.02
2.30
2.30
2.30
2.30
2.15
1.52
1.39
1.13
1.72
Finuish
1.91
2.28
2.28
2.28
2.28
1.99
2.28
1.46
1.56
1.47
1.13
1.14
1.23
Fon
2.83
1.08
1.08
1.08
1.08
3.67
1.08
2.75
2.51
2.31
Freuch
1.47
2.00
2.00
2.00
2.00
1.47
0.84
1.20
1.30
Friulisn
1.70
2.07
2.07
2.07
2.07
1.85
2.07
1.47
1.38
1.33
1.07
1.56
1.17
Nigerian Fulfulde
1.72
1.99
1.99
1.99
1.99
1.85
1.99
1.37
1.29
1.16
0.86
1.46
1.27
West Central Oromo
2.22
2.53
2.53
2.53
2.53
1.72
1.73
1.61
1.21
1.78
1.49
Scottish Gaelic
2.33
2.70
2.70
2.70
2.70
2.42
2.70
1.86
1.80
1.61
1.24
1.75
1.61
Irish
2.17
2.56
2.56
2.56
2.56
2.33
2.56
1.76
1.75
1.55
1.15
1.50
1.50
Galician
1.48
1.91
1.91
1.91
1.91
1.56
1.91
1.39
1.36
1.30
1.11
1.13
1.14
Guarani
1.99
2.46
2.46
2.46
2.46
2.17
2.46
1.68
1.55
1.45
1.05
1.72
1.63
Gujarati
9.98
12.27
12.27
12.27
12.27
7.69
12.27
8.17
1.12
1.58
Haitian Croole
1.58
1.90
1.90
1.90
1.90
1.74
1.90
1.35
1.32
1.15
0.89
1.39
1.16
Hausa
1.89
2.15
2.15
2.15
2.15
2.00
2.15
1.49
1.47
1.26
1.02
1.40
1.29
23

---

BERT Japanss
ArabicBERR
ScanlessM4r
Language
MBari50
CANINE
ml55
PboT5
Dy1s
BLOOM
MORIL
UTF-32
NLLB
Qwen
Acehnese (Arabic script)
1.94
1.79
1.51
0.85
2.65
0.85
1.89
1.89
2.66
Aoehnese (Latin script)
1.57
1.44
2.55
1.09
1.07
1.74
1.44
2.02
1.07
1.41
1.24
1.24
1.95
Mesopolamian Arabic
1.16
1.28
1.56
0.86
1.15
0.55
1.93
0.86
1.37
1.37
1.63
Ta 'izzi-Adeni Arabic
1.17
1.32
1.58
0.87
1.15
0.55
0.87
1.39
1.39
1,63
Tumisian Arabic
1.20
1.29
1.54
0.85
1.19
0.57
1.90
0.85
1.39
1.39
1.66
Afrikaans
1.20
1.20
2.15
1.07
1.06
1.69
1.33
1.84
1.06
1.27
1.22
1.22
1.67
South Levantiue Arabic
1.12
1.24
0.83
1.12
0.55
1.82
0.83
1.31
1.55
Akan
1.98
1.82
2. 2.96
1.10
1.00
2.05
1.00
1.45
1.40
1.40
2.28
Took Albanian
1.32
1.48
3.09
1.20
1.12
2.17
1.46
2.52
1.12
1.35
1.35
2.23
Ambaric
1.34
1.73
1.72
0.67
5.07
0.67
1.32
1.32
4.16
North Levantine Arabic
1.15
1.23
1.48
0.82
1.13
0.55
1.83
0.82
1.33
1.33
1.58
Standard Arabie
1.18
1.35
1.00
0.88
1.14
0.55
1.97
0.88
1.40
1.40
1.63
Standard Arabie (Romanized)
1.94
1.73
2.94
1.17
1.17
2.15
1.60
2.28
1.17
1.64
1.86
1.86
2,42
Najdi Arabic
1.18
1.35
1.60
0.88
1.15
0.55
1.97
0.88
1.40
1.40
1.63
Moroccan Arabic
1.25
1.29
1.56
0.86
1.26
0.63
1.91
0.86
1.39
1.39
1.70
Egyptian Arabie
1.17
1.28
1.56
0.86
1.16
0.57
1.89
0.86
1.36
1.36
1.64
Assumese
1.90
1.94
2.54
0.96
1.41
1.24
0.96
1.39
1.39
5.46
Asturian
1.27
1.28
2.07
1.07
1.03
1.31
1.24
1.81
1.03
1.26
1.17
1.17
1.56
Awadhi
1.37
1.62
2.50
0.98
1.13
0.98
1.22
1.22
4.36
Central Aymara
1.70
1.57
2.71
1.07
1.05
1.94
1.44
1.98
1.05
1.45
1.32
1.32
2.15
South Azerbaijani
1.43
1.42
1.63
0.89
1.81
1.11
1.72
0.89
1.37
1.37
2.62
North Azerbaijani
1.15
1.35
1.09
2.30
1.74
1.09
1.33
1.33
2.49
Bashkir
2.06
1.60
1.85
1.01
3.57
1.01
1.22
1.22
3.14
Bambara
1.82
1.65
2.
1.04
0.96
1.89
0.96
1.34
1.27
1.27
2.14
Balinese
1.32
1.29
2.37
1.11
1.11
1.46
1.40
1.83
1.11
1.33
1.08
1.08
1.79
Belarusian
1.46
1.59
2.06
1.13
3.24
2.60
1.13
1.72
1.72
3.00
Bemba
1.76
1.57
3.01
1.23
1.23
1.92
1.65
2.17
1.23
1.64
1.39
1.39
2,20
Bengali
1.38
1.58
2.61
0.98
1.17
1.01
0.98
1.28
1.28
5.09
Bhojpuri
1.47
1.63
2.47
0.97
1.5/
1.39
0.97
1.28
1.28
4.33
Banjar (Arabic scripl]
1.92
1.76
1.69
0.93
2.47
1.04
0.93
1.88
1.88
2.63
Baujar (Latin script)
1.21
1.16
2,20
1.05
1.05
1.30
1.32
1.71
1.05
1.29
1.08
1.08
1.70
Standard Tibetan
3.68
3..1
1.13
6.66
1.13
1.44
1.44
7.33
Bosnian
1.12
1.33
2.48
1.03
1.01
1.84
1.39
1.01
1.30
1.19
1.19
1.86
Buginese
1.51
1444
2.51
1.09
1.06
1.71
1.45
1.96
1.06
1.39
1.30
1.30
1.96
Bulgarian
1.16
1.28
1.89
1.04
2.49
2.35
1.04
1.31
1.31
2.20
Catalan
1.26
1.36
2.14
1.12
1.10
1.18
1.29
1.90
1.10
1.30
1.25
1.25
1.69
Cobuano
1.52
1.42
2.86
1.20
1.20
1.78
1.51
2.10
1.20
1.5/3
1.29
1.29
1.91
Czech
1.17
1.27
2.72
1.08
0.97
2.03
1.31
0.97
1.26
1.26
2.07
Chokwe
1.55
1.41
2.66
1.07
1.07
1.72
1.47
1.94
1.07
1.42
1.34
1.31
1.94
Contral Kurdish
2.30
1.75
1.78
0.97
3.21
1.65
0.97
1.30
1.30
3.46
Crimean Tatar
1.38
1.32
2.80
1.13
1.03
2.07
1.45
1.03
1.25
1.25
1.95
Welsh
1.43
1.70
3.12
1.07
1.07
2.09
1.55
2.32
1.07
1.47
1.38
1.38
2.09
Danish
1.09
1.14
2,26
1.05
1.03
1.67
1.28
1.83
1,03
1.11
1.11
1.61
German
1.17
1.19
1.37
1.18
1.17
1.68
1.44
2.02
1.17
1.37
1.29
1.29
1.55
Southwestern Diuka
1.68
1.58
0.96
0.86
1.82
0.86
1.25
1.25
2.01
Dyula
1.65
1.55
2.68
1.07
1.01
1.50
1.30
2.06
1.01
1.39
1.41
1.44
1.96
Dzongkha
4.24
3.64
1.25
7.36
1.25
1.48
1.48
8.19
Greek
1.45
1.65
2.17
1.20
3.81
2.70
1.20
1.65
1.65
4.95
English
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1,00
Esperanto
1.20
1.19
2.19
1.02
1.00
1.65
1.24
1.00
1.23
1.23
1.80
Estonian
1.12
1.12
2.43
1.01
0.98
1.77
1.25
1.71
0.98
1.16
1.16
1.85
Basque
1.16
1.22
2.33
1.07
1.06
1.14
1.41
1.90
1.06
1.35
1.27
1.27
1.87
Ewe
2.01
1.82
2.85
1.07
0.97
2.11
0.97
1.27
1.27
2.36
Faroese
1.44
1.40
2.73
1.09
1.02
1.95
1.41
1.02
1.31
1.31
2.04
Fijian
1.72
1.59
3.02
1.17
1.17
1.99
1.65
2.01
1.17
1.53
1.32
1.32
2.13
Finnish
1.14
1.16
2.61
1.11
1.07
1.89
1.42
2.05
1.07
1.45
1.21
1.21
1.97
Fou
2.51
2.36
1.26
1.02
2.21
1.02
1.59
1.59
2.87
French
1.30
1440
1.60
1.19
1.20
1.33
1.96
1.19
1.36
1.35
1.35
1.57
Friulian
1.56
1.52
2.
2.
1.13
1.10
1.70
1.28
1.94
1.10
1.29
1.37
1.37
1.83
Nigerian Fulfulde
1.46
1.32
2.14
0.96
0.93
1.66
1.16
1.54
0.93
1.21
1.24
1.24
1.75
West Central Oromo
1.78
1.69
3.16
1.20
1.19
2.19
1.63
2.17
1.19
1.63
1.42
1.42
2.29
Scottish Gaelic
1.75
1.85
3.24
1.28
1.24
2.25
1.57
2.27
1.24
1.49
1.56
1.56
2.38
Irish
1.50
1.67
3.14
1.23
1.16
2.15
1.45
2.16
1.16
1.51
1.42
1.42
2.28
Galician
1.13
1.31
2.18
1.13
1.11
1.27
1.30
1.91
1.11
1.32
1.16
1.16
1.54
Guarani
1.72
1.62
2.57
1.09
1.01
1.87
1.40
1.99
1.01
1.34
1.34
2.09
Gujarati
1.42
1.73
2.50
0.96
1.35
1.19
0.96
1.35
1.35
6.78
Haitian Creole
1.39
1.22
2.32
0.95
0.92
1.56
1.18
1.68
0.92
1.19
1.11
1.11
1.72
Hausa
1.40
1.37
2.61
1.08
1.07
1.78
1.34
1.78
1.07
1.35
1.18
1.18
1.95
24

---

XLM-R.BERI.
CamemBERR
Language
LLAMA
r50k base
p50k beso
p50k. edit
c1100k b
RODERTA
GOMUBERT
PhoBERT
RoCBen
M2M100
GPT-2
Hebrew
3.29
1.39
4.39
4.39
4.39
3.66
1.39
4.52
1.12
1.22
Hindi
4.60
7.46
7.46
7.46
7.46
4.79
7.46
8.34
1.25
1.36
Chhattisgarhi
1.11
7.21
7.21
7.21
7.21
1.69
7.21
8.05
1.41
1.51
Cruatian
2.15
2.15
2.15
1.85
2.15
1.46
1.43
1.33
1.10
Hungarian
1.79
2.66
2.66
2.66
2.66
2.15
2.66
1.79
1.78
1.57
1.08
1.18
1.28
Armenian
10.01
10.01
10.01
10.01
9.98
10.01
6.67
1.38
Igbo
2.32
3.42
3.42
3.42
3.42
2.44
3.42
2.33
1.77
1.48
0.99
2.12
1.47
Hlccmo
2.01
2.26
2.26
2.26
2.26
2.05
2.26
1.50
1.61
1.41
1.21
1.61
1.33
Indonesion
1.76
1.98
1.98
1.98
1.55
1.98
1.37
1.40
1.25
0.91
0.98
Icelandle
1.98
2.43
2.43
2.43
2.43
2.15
2.43
1.72
1.50
1.23
1.29
Italian
1.46
2.01
2.01
2.01
2.01
1.64
2.01
1.43
1.36
1.33
1.19
1.19
1.25
Javanese
1.72
1.93
1.93
1.93
1.93
1.73
1.93
1.36
1.39
1.21
1.06
1.15
1.10
Japanese
2.24
3.00
3.00
3.00
3.00
2.30
3.00
3.23
0.52
1.11
1.20
Kabyle
2.00
2.50
2.50
2.50
2.50
2.47
2.50
1.74
1.59
1.43
0.90
1.84
1.71
Jagpho
2.27
2.65
2.65
2.65
2.65
2.35
2.65
1.89
1.78
1.54
1.20
1.94
1.78
Kamba
2.32
2.17
2.32
1.30
1.62
Kannada
10.83
13.69
13.69
13.68
13.68
8.90
13.69
9.27
1.36
1.53
Kashmiri (Arabic script)
1.13
6.19
6.19
6.19
6.19
1.62
6.19
5.63
1.93
1.93
Kashmiri (Devanagari script]
4.44
7.03
7.03
7.03
7.03
4.69
7.03
7.76
1.82
1.86
Georgian
1.87
13.85
13.85
13.85
13.85
9.85
13.85
9.22
1.34
1.56
Kazakh
2.51
5.92
5.92
5.92
3.79
5.92
2.66
1.15
Kabiye
3.48
4.87
4.87
4.87
4.87
4.74
4.87
3.28
2.98
2.71
Kabuverdlanu
1.58
1.93
1.93
1.93
1.93
1.72
1.93
1.32
1.30
1.21
0.98
1.35
1.30
Halli Mongolian
2.76
6.42
6.42
6.42
6.42
3.77
6.42
4.24
2.72
1.21
1.34
Khiner
10.26
15.33
15.33
15.33
15.33
8.88
15.33
10.22
1.62
1.87
Kikuyu
2.52
3.44
3.44
3.44
3.44
3.29
3.44
2.36
1.66
1.18
2.31
2.17
Kinyarwanda
2.04
2.37
2.37
2.37
2.37
2.14
2.37
1.61
1.59
1.47
1.15
1.72
1.63
Kyrgyz
5.74
5.74
5.74
3.51
5.74
2.67
1.16
Kimbundu
2.02
2.33
2.33
2.33
2.33
2.13
2.33
1.61
1.58
1.413
1.12
1.61
1.51
Northern Kurdish
2.05
2.45
2.45
2.45
2.45
2,20
2.45
1.66
1.65
1.40
0.99
1.38
1.66
Central Kanuri (Arabic script]
3.82
4.74
4.74
4.74
3.63
4.74
5.20
2.60
Central Kamuri (Latin script)
2.15
2.57
2.57
2.57
2.57
2.37
2.57
1.78
1.60
1.44
1.74
1.65
Kikougo
2.17
2.17
2.17
1.37
1.58
Korran
3.18
5.07
5.07
5.07
5.07
2.38
5.07
3.86
0.99
1.16
1.21
Lao
11.47
13.19
13.19
13.19
13.19
9.62
13.19
8.79
1.39
1.61
Ligurian
1.84
2.29
2.29
2.29
2.29
1.98
2.29
1.57
1.50
1.43
1.09
1.65
1.59
Limburgish
1.64
2.05
2.05
2.05
2.05
1.80
2.05
1.34
1.39
1.332
1.04
1.45
1.38
Lingala
1.79
2.03
2,03
2,03
1.86
2.03
1.47
1.37
1.26
1.52
Lithuanian
1 1.89
2.45
2.45
2.45
2.45
2.21
2.45
1.63
1.53
1.42
1.04
1.17
1.25
Lombard
1.85
2.37
2.37
2.37
2.37
2.04
2.37
1.58
1.52
1-41
1.11
1.71
1.56
Latgalian
1.99
2.39
2.39
2.39
2.39
2.20
2.39
1.67
1.62
1.48
1.02
1.57
1.51
Laxembourgish
1.80
2.25
2.25
2.25
2.25
1.99
2.25
1.30
1.32
1.43
1.15
1.64
1.32
Laba-Kasai
2.13
2.13
2.13
1.50
1.41
1.31
1.51
1.43
Ganda
1.90
2.17
2.17
2.17
2.17
1.96
2.17
1.48
1.47
1.36
1.07
1.55
1.38
2.04
2.04
2.04
2.04
1.39
1.27
1.52
Mizo
1,86
2.09
2.09
2.09
2.09
1.96
2.09
1.53
1.52
1.29
1.06
1.65
1.54
Standard Latvian
2,10
2.54
2.54
2.54
2.54
2.35
2.54
1.76
1.68
1.56
1.05
1.23
1.29
Magahi
4.49
7.22
7.22
7.22
7.22
4.70
7.22
8.07
1.41
1.50
Maithuli
1.63
7.13
7.13
7.43
7.13
1.90
7.43
8.27
1.58
1.61
Malayalam
15.24
15.24
15.24
10.16
1.38
Marathi
4.58
7.87
7.87
7.87
7.87
5.07
7.87
8.76
1.22
1.38
Missugkabau (Arabie script)
4.32
5.25
5.25
5.25
5.25
3.97
5.25
5.71
2.02
1.99
Minaugkaban (Latin scripl)
1.77
1.97
1.97
1.97
1.97
1.77
1.97
1.40
1.39
1.25
1.09
1.31
1.25
Macrdemian
1.84
5.46
5.46
5.46
5.46
2.77
5.46
3.48
2.58
1.17
1.24
Mallese
2.69
2.69
2.41
2.69
1.57
1.96
Meitei (Bengali script)
5.84
10.22
10.22
10.22
10.22
6.71
10.22
9.06
2.56
2.59
Mossi
2.54
2.54
2.54
2.32
2.54
1.51
1.38
1.78
Maori
2.18
2.45
2.15
2.45
2.15
2.35
2.45
1.77
1.69
1.:17
1.05
1.86
1.74
Burmese
8.37
16.89
16.89
16.89
16.89
11.70
16.89
11.26
1.72
2.21
Dutch
1.46
1.97
1.97
1.97
1.59
1.97
1.28
1.40
1.32
1.14
Norwegian Nynorsk
1.54
1.93
1.93
1.93
1.93
1.64
1.93
1.25
1.40
1.29
1.02
1.17
1.17
Norwegian Boknil
1.86
1.86
1.56
1.86
1.27
1.07
Nepali
1.19
7.59
7.59
7.59
7.59
4.79
7.59
8.37
1.13
1.28
Northeru Sotho
2.02
2.32
2.32
2.32
2.32
2.18
2.32
1.63
1.58
1.48
1.12
1.75
1.52
Nuer
2.83
4.23
4.23
4.23
4.23
4.00
4.23
2.79
2.62
2.44
Nyanja
2.02
2.26
2.26
2.26
2.26
2.08
2.26
1.57
1.55
1-42
1.17
1.59
1.55
Ocritan
2.07
2.07
2.07
2.07
1.47
1.38
1.50
Odia
11.59
13.38
13.38
13.38
13.38
12.48
13.38
8.94
1.45
1.56
25

---

BERT Japanese
ArabieBERT
SeamkssM/T
Language
MBart50
CANINE
m1
FlanTS
ByT5
BLOOK
MuRIL.
UTP-32
NLLB
Qween
Hehrew
1.12
1.22
1.39
0.78
2.92
1.72
0.78
1.24
1.24
1.48
Hindi
1.25
1.59
2.55
1.00
1.28
1.16
1.00
1.22
1.22
4.47
Chhattisgarhi
1.41
1.60
2.46
0.97
1.44
1.34
0.97
1.26
1.26
4.26
Croatian
1.10
1.30
2. 13
1.01
0.98
1.80
1.30
0.98
1.27
1.17
1.17
1.83
Uungarian
1.18
1.26
1.16
1.05
2.07
1.40
2.31
1:27
2.12
Armenian
1.38
1.38
2.04
1.11
4.31
1.11
1.51
1.51
5.31
Igbo
2.12
1.70
3.17
1.21
1.02
1.72
1.50
1.00
1.32
1.32
2.37
Пloaano
1.61
1.61
2.82
1.21
1.21
1.90
1.55
2.01
1.21
1.55
1.33
2.03
Indonesian
0.94
1.08
2.24
1.08
1.08
0.96
1.35
1.74
1.08
1.33
0.93
0.93
1.54
Icelandie
1.23
1.32
2.81
1.09
0.99
1.99
1.34
0.99
1.29
1.29
2.11
Italian
1.19
1.34
2.18
1.19
1.18
1.62
1.41
1.922
1.18
1.37
1.25
1.25
1.62
Javanese
1.15
1.21
2.21
1.04
1.04
1.40
1.36
1.74
1.04
1.29
1.03
1.03
1.72
Japanese
1.11
0.90
1.27
1.01
1.01
Kabyle
1.84
1.82
2.83
1.0G
0.99
2.02
1.29
0.99
1.56
1.56
2.14
Jingpho
1.94
1.79
3.11
1.27
1.28
2. 2.14
1.71
2.32
1.28
1.65
1.17
1.47
2.32
Kamba
1.62
1.52
2.69
1.01
0.98
1.33
0.98
1.28
Kannada
1.36
1.44
2.83
1.05
1.31
1.06
1.05
1.37
1.37
6.98
Kashmiri (Arabic script)
1.93
2.00
1.72
0.96
2.32
1.26
1.75
0.96
1.81
1.81
3.48
Kashmiri (Devanagari script)
1.82
1.79
2.40
0.96
1.85
1.75
0.96
1.69
4.41
Georgian
1.34
1.35
2.95
1.10
4.98
1.10
1.61
1.61
5.25
Kazakh
1.15
1.20
1.89
1.03
3.23
1.18
3,02
Kabiye
2.98
2.83
1.37
1.09
3.34
1.09
1.56
1.56
3.35
Kabuverdianu
1.35
1.28
2.21
1.02
0.99
1.51
1.25
1.81
0.99
1.29
1.28
1.28
1.70
Halh Mongulian
1.21
1.48
1.91
1.36
3.10
Khmer
1.62
1.43
3.33
1.18
6.40
1.18
1.80
1.50
6.61
Kikuyu
2.31
2.18
1.30
1.17
2.48
1.56
1.17
1.52
1.52
2.66
Kinyarwands
1.72
1.51
1.13
1.11
1.51
2.15
1.11
1.50
1.30
1.30
2.12
Kyrgys
1.16
1.32
1.88
1.02
3.02
1.02
1.25
1.25
2.74
Kimbundu
1.64
1.48
2.91
1.11
1.11
1.81
1.55
1.99
1.11
1.52
1.35
1.35
2.10
Northern Kurdish
1.38
1.42
2.74
1.10
1.00
2.00
1.29
1.00
1.44
1.44
2.16
Central Kanuri (Arabic script)
2.60
2.43
1.60
0.88
2.10
2.37
0.88
2.54
2.54
3.15
Central Kanrri (Latin script)
1.74
1.58
1.11
1.55
2.16
Kikongo
1.58
1.46
3.01
1.14
1.14
1.75
1.59
1.97
1.14
1.54
1,21
1.21
1.98
Korran
1.16
1.27
1.20
0.51
2.79
1.30
0.51
1.03
1.03
1.64
Lao
1.39
1.27
2.73
1.47
5.79
Ligurian
1.65
1.69
2.54
1.17
1.10
1.81
1.38
2.05
1.10
1.60
1.60
1.95
Limburgish:
1.45
1.38
1.07
1.04
1.75
1.32
1.92
1.04
1.28
1.44
1.44
1.78
Lingala
1.52
1.38
2.73
1.08
1.08
1.65
1.47
1.90
1.08
1.41
1.12
1.12
1.85
Lithuanian
1.17
1.23
2.58
1.06
1.00
1.94
1.33
1.00
1.18
1.18
2,06
Lombard
1.71
1.70
1.16
1.07
1.84
1.29
1.96
1.07
1.61
2,00
Lacgalian
1.57
1.46
2.70
1.05
0.99
1.99
1.36
0.99
1.42
1.42
2.14
Luxembourgiih
1.64
1.46
2.24
1.15
1.12
1 1.89
1.40
2.17
1.12
1.31
1.44
1.44
1.96
Luba-Kasai
1.54
1.37
2.
1.08
1.08
1.68
1.44
1.89
1.08
1.41
1.21
1.21
1.92
Ganda
1.55
1.40
2.65
1.0K3
1.02
1.67
1.46
1.94
1.02
1.41
1.26
1.26
1.94
Luo
1.52
1.41
2.55
1.05
1.05
1.68
1.35
1.87
1.05
1.35
1.24
1.24
1.81
Mizo
1.65
1.57
2.76
1.10
1.10
1.83
1.43
1.92
1.10
1.37
1.31
1.31
1.94
Standard Latvian
1.23
1.30
2.78
1.11
1.02
2.08
1.35
1.02
1.20
1.20
2.29
Magahi
1.41
2.46
1.34
1.28
Maithili
1.58
1.74
2.53
0.98
1.56
1.50
0.98
1.24
1.24
4.42
Malayalam
1.38
1.35
3.10
1.13
1.38
1.18
1.13
1.49
1.49
7.31
Marathi
1.22
1.32
2.67
1.01
1.06
1.01
1.26
1.26
4.65
Minangkabeu (Arabic script)
2.02
1.84
1.74
0.96
2.58
1.13
0.96
1.97
1.97
2.79
Minsugkabau (Latin script)
1.31
1.25
2.35
1.07
1.07
1.44
1.36
1.77
1.07
1.32
1.15
1.15
1.75
Macedonian
1.17
1.29
1.89
1.04
2.50
1.04
1.24
1.24
2.26
Maltese
1.96
1.69
2.94
1.16
1.11
2. 2.25
1.44
1.11
1.16
1.46
2.24
Meitei (Bengali script]
2.56
2.21
2.77
1.03
2.35
2.34
1.00
1.73
Mussi
1.78
1.80
2.90
1.03
0.96
1.99
1.19
0.96
1.36
1.36
2.06
Maori
1.86
1.69
3.22
1.16
1.11
2.12
1.49
2.12
1.11
1.45
1.38
1.38
2.33
Burnese
1.72
1.56
3.51
1.24
10.05
1.59
8.99
Dutch
1.14
1.17
2.19
1.11
1.11
1.71
1.38
1.91
1.11
1.33
1.19
1.19
1.58
Norwegian Nynorsk
1.17
1.18
2.29
1.04
1.01
1.65
1.28
1.82
1.01
1.22
1.16
1.16
1.63
Norwegian Bokmål
1.07
1.12
2.21
1.03
1.01
1.62
1.26
1.79
1.18
1.10
1.55
Nepali
1.13
1.47
2.56
0.96
1.17
1.01
0.96
1.18
1.18
4.45
Northern Sotho
1.75
1.57
2.81
1.17
1.15
1.91
1.48
2.18
1.15
1.48
1.35
1.35
2.17
Nuer
2.62
2.42
1.32
1.08
2.79
1.08
1.89
1.50
3.39
Nyauja
1.59
1.35
2.71
1.12
1.12
1.78
1.52
2.02
1.12
1.44
1.15
1.15
2.06
Ocritan
1.50
1.48
1.17
1.14
1.93
1.40
Odia
1.45
3.11
2.73
1.03
1.36
1.21
1.03
1.38
1.38
9.79
26

---

XLM-RoBERI.
CamemBERT
c1100k base
Language
LLAMA
150k: brsse
p50k Touse
pōlk edlt
ROBERTA
CotBEETT
PhoBERT
RoCBert
M2M100
GPT-2
Pangasinan
1.50
1.66
1.66
1.66
1.66
1.57
1.66
1.27
1.25
1.11
1.00
1.29
1.23
Eastern Panjabi
9.44
7.90
7.90
7.90
7.90
7.87
7.90
8.47
1.57
1.68
Papiamento
1.65
1.98
1.98
1.98
1.98
1.75
1.98
1.33
1:37
1.25
1.03
1.37
Southern Pashto
4.27
5.39
5.39
5.39
5.39
3.83
5.39
5.37
1.38
1.40
Western Persian
3.98
5.32
5.32
5.32
5.32
3.28
5.32
5.47
1.10
1.17
Plateau Malagasy
2.12
2.58
2.58
2.58
2.58
2.26
2.58
1.74
1.69
1.49
1.26
1.57
Polish
1.70
2.69
2.69
2.69
2.69
1.91
2.69
1.79
1.71
1.58
1.00
1.19
1.26
Portuguese
1.42
1.94
1.94
1.94
1.94
1.48
1.94
1.38
1.36
1.30
1.09
1.11
1.14
Dari
3.88
5.11
5.11
5.11
5.11
3.16
5.11
5.31
1.09
Ayneucho Quechua
1.96
2.20
2.20
2.20
2.20
2.08
2.20
1.61
1.54
1.40
1.14
1.59
1.54
Romanian
1.70
2,48
2.18
2.18
2.48
1,88
2.48
1 1.69
1.54
1.16
1.13
1.21
1.29
Rundi
2.05
2.333
2.333
2.33
2.333
2.13
2.333
1.63
1.47
1.15
Russian
1.64
5.74
5.74
5.74
5.74
2.49
5.74
3.67
2.71
1.03
1.17
1.22
Sango
1.95
2.23
2.23
2.23
2.23
2.08
2.23
1.51
1.50
1.32
1.02
1.66
1.53
Sanskrit
4.59
7.94
7.94
7.94
7.94
5.00
7.94
8.60
1.43
Santali
11.92
12.86
12.86
12.86
12.86
122800
12286
8.56
Sieilian
1.81
2.27
2.27
2.27
2.27
2.011
2.27
1.57
1.43
1.37
1.06
1.58
1.53
Shan
11. 11.85
18. 76.76
18. 76
18. 76
188.
15.05
18.76
12.51
4.43
Sinhala
7.86
12.86
12.86
12.86
12.86
8.83
12.86
8,59
1.35
1.53
Slovak
1.82
2.52
2.55
2.52
2..2
2.14
2.52
1.65
1.60
1.46
1.02
1.18
1.24
Slovenian
1.67
2.11
1.01
Samoan
2.14
2.57
2.57
2.57
2.57
2.29
2.57
1.69
1.63
1.50
1.09
1.92
1.80
Shona
2.01
2.29
29
2. 29
2. 2,29
2.13
2.29
1.58
1.58
1.44
1.18
1.63
1.58
Sindhi
4,20
5.00
5.00
5.00
5.00
4.00
5.00
5.22
1.228
Somali
2.14
2.36
2.36
2.36
2.36
2.18
2.36
1.G6
1.69
1.48
1.16
1.39
1.37
Southern Sotho
2.07
2.34
2.331
2.3-1
2.34
2.21
2.31
1.64
1.63
1.48
1.18
1.78
1.60
Spanish
1.45
1.99
1.99
1.99
1.35
1.99
1.45
1.36
1.19
1.20
Sardinisn
1.82
2.26
2.26
2,26
2,26
1.99
2.26
1.53
1,48
1,40
1.16
1.61
1.51
Serbian
1.78
5.33
5.34
5.34
5.34
2,.92
5-34
33.4
2.45
1.18
1.26
Swati
2.03
2.31
2.31
2.31
2.16
1.59
1.G0
1.45
1.21
1.61
1.44
Sundsnese
1.76
2.02
2.02
2.02
2.02
1.82
2.02
1.39
1.39
1.24
1.07
1.22
1.10
Swedish
1.44
1.95
1.95
1.95
1.95
1.58
1.95
1.22
1.41
1.33
1.02
1.07
1.10
Swahili
1.86
2.13
2.13
2.13
1.06
1.16
Silesisn
1.95
2.00
2.60
2,60
2,60
2.18
2.60
1.71
1.70
1.59
0.99
1.65
1.59
Tamil
5.87
15.58
15.58
15.58
15.58
7.65
15.58
10.38
1.35
1.55
Tamasheq (Latin script)
1.93
2.39
2.39
2.39
2.39
2.22
2.39
1.62
1.50
1.29
1.71
1.57
Tamasheq (Tifinggh script)
8.42
10.43
10.43
10.43
10.43
10.13
10.44
6.95
Tatar
2.53
5.82
5.82
5.82
5.82
3.75
5.82
3.84
1.81
1.54
Telugu
10.71
13.09
13.09
13.09
8.34
L3.09
8.73
1.33
Tajik
2.70
6.09
6.09
6.09
6.09
3.64
6.09
4,00
2.82
2.14
2.06
Tagaing
2.00
2.28
2.28
2.28
2.28
2.06
2.28
1.63
1.67
1.45
1.27
1.43
1.43
Thai
4.35
9.05
9.05
9.05
9.05
6.59
2.83
1.08
Tigrinya
7.47
7.88
7.88
7.88
7.88
7.80
7.88
5.25
1.97
1.91
Tok Pisin
1.95
2.21
2.21
2.21
2.21
2.04
2.21
1.55
1.G6
1.45
1.25
1.73
Tswatt
2.12
2.39
2.39
2.39
2.39
2.28
2.39
1.68
1.55
1.21
1.85
1.68
Tsonga
2.16
2.45
2.445
2.44
2.44
2,226
22.45
1.70
1.70
1.46
1.19
1.79
1.69
Turkmen
2.23
2.82
2.82
2.82
2.82
2.40
2.82
1.76
1.78
1.62
1.11
1.78
1.71
Tumbuka
2.16
2.78
2.78
2.78
2.78
2.37
2.78
1.93
1.85
1.67
1.34
1.92
1.88
Turkish
2.09
2.43
22.33
22.33
22.33
1.991
22.43
1.61
1.65
1.51
1.04
1.15
Twi
2.01
2.62
2.62
2.62
2.62
2.51
2.62
1.80
1.57
1.38
1.88
1.74
Central Atlas Tamazight
8.86
10.339
10.339
10.39
T0.39
10.04
10.399
6.92
Uyghur
4,89
7.16
7.16
7.16
7.16
5.
7.16
6.44
1.41
3.00
Ukrainian
1.72
5.75
5.75
5.75
5.75
3.00
5.75
3.69
2.58
1.21
1.28
Umbundu
1.89
2.24
2.24
2.24
2.24
2.01
2.24
1.53
1.48
1.36
1.05
1.57
1.49
Urdu
4.37
6.30
6.30
6.30
6.30
4.39
6.30
5.74
1.23
1.30
Northeru Uzbek
2,03
2.30
2.30
2.30
2.30
2.17
2.30
1.63
1.48
1.19
1.33
1.37
Venetian
1.56
2.00
2.00
2.00
2.00
1.70
2.00
1.38
1.34
1.23
1.36
1.31
Vietnamese
2.92
4.54
4..5A
4.5-1
4.54
2.15
4.54
3.06
0.83
0.98
1.18
1.15
Waray
2.02
2.38
22.38
2.38
22.88
1.95
2.338
1.42
1.25
1.55
Wolof
1.80
2.14
2.14
2.14
2.14
1.92
2.14
1.49
1.43
1.28
0.93
1.60
1.40
Xhosa
1.97
2.26
2.26
2.26
2.26
2.0G
2.26
1.57
1.57
1.40
1.13
1.50
1.37
Eastern Yiddish
4.57
6.63
6.63
6.63
6.63
5.37
6.63
6.34
1.58
Yoruba
2.70
3.89
3.89
3.89
3.89
2.96
3.89
2.63
1.66
0.88
2.27
1.74
Yue Chinese
2.11
3.09
3.09
3.09
3.09
2.12
3.09
2.78
0.36
0.93
1.03
Chinese (Simplified)
2.00
3.21
3.21
3.21
3.21
1.91
3.21
2.93
0.39
0.97
Chinese (Traditional)
2.16
3.16
3.16
3.16
3.16
2.18
3.16
2.83
0.36
0.96
1.06
Standard Malay
1.83
2.05
2.05
2.05
2.05
1.62
2.05
1.42
1.45
1.28
1.15
0.95
1.00
Zulu
2.09
2.20
1.65
1.20
1.55
27

---

BERT Japaneso
AmbicBERT
SoamlessM4r
Language
MBart50
CANINE
B3
BL.OOM
M.RIL
NLLB
1T33
Pangasinan
1.29
1.22
2.18
1.00
1.00
1.45
1.24
1.54
1.00
1.21
1.11
1.11
1.56
Eastern Panjabi
1.57
2.11
2.59
1.01
1.43
1.35
1.01
1.50
1.50
7.30
Papiamento
1.37
1.36
2:28
1.08
1.05
1.54
1.25
1.80
1.05
1.30
1.27
1.27
1.73
Southern Pashto
1.64
0.95
Western Persian
1.10
1.34
1.70
0.91
1.78
1.11
1.62
0.94
1.13
1.13
2.60
Platean Malagasy
1.57
1.59
3.00
1.26
1.22
2.07
1.64
2.33
1.22
1.59
1.39
1.39
2.23
Polish
1.19
1.31
2.82
1.13
1.06
2.14
1.52
1.06
1.37
1.37
1.76
Portugusse
1.11
1.29
2.21
1.12
1.09
1.12
1.30
1.88
1.09
1.24
1.17
1.17
1.45
Dari
1.09
1.31
1.63
0.92
1.64
1.09
1.58
0.92
1.11
1.11
2,50
Ayacucho Quechua
1.59
1.42
2.59
1.08
1.07
1.47
1.95
1.07
1.42
1.28
2.06
Romanian
1.24
1.37
1.50
1.19
1.13
1.91
1.33
1.13
1.35
1.35
1.86
Rundi
1.71
1.52
2.78
1.12
1.12
1.64
1.54
2.13
1.12
1.50
1.33
1.33
2.11
Russian
1.17
1.27
1.98
2.48
2.50
1.09
1.34
1.34
1.75
Sango
1,66
1.63
3.14
1.12
1.45
2.05
1.09
1.49
1.39
1.39
2.04
Sanskrit
1.43
1.65
2.63
0.98
1.63
1.21
0.98
1.40
1.40
4.58
Santali
2.79
1.06
12.71
1.06
2.49
2.49
8.99
Sicilian
1.58
1.53
2.16
1.11
1.41
184
1.05
1.95
Shan
4.43
3.28
3.94
1.42
12.06
1.42
1.94
1.94
10.51
Sinhala
1.35
1.66
2.64
1.00
8.21
1.00
1.68
1.68
7.02
Slovak
1.18
1.30
2.74
L.09
2.01
1.35
1.00
1.21
2,08
Slovenian
1.13
1.20
2.42
1.02
1.00
1.81
1.37
1.00
1.30
1.17
1.17
1.87
Samoan
1.92
1.92
3.09
1.22
1.16
2.13
1.57
2.22
1.16
1.55
1.60
1.60
2.26
Shona
1.63
1.35
2.79
1.12
1.55
2.06
1.12
1.48
1.23
1.23
2.11
Sindhi
1.28
1.74
1.60
0.91
2.51
1.22
0.91
1.33
1.33
2.87
Somali
1.39
1.48
33.6
1.14
1.14
2.03
1.52
2.05
1.14
1.52
1.39
1.39
2.16
Southern Sotho
1.78
1.59
2.92
2.16
1.20
1.54
Spanish
1.20
1.31
2.23
1.21
1.19
1.21
1.38
1.98
1.19
1.41
1.24
1.24
1.52
Sardiniun
1.61
1.57
2.46
1.19
1.16
1.73
1.38
1.98
1.16
1.36
1.44
1.44
1.97
Serbian
1.30
2.34
Swati
1.61
1.41
2,80
1.12
1.13
1.83
1.55
2.09
1.13
1.52
1.28
1.28
2.14
Sundanese
1.22
1.22
2.32
1.05
1.04
1.48
1.33
1.80
1.04
1.31
1.04
1.04
1.80
Swedish
1.07
1.11
2.22
1.04
1.01
1.21
1.90
1.01
1.20
1.13
1.57
Swahili
1.16
1.25
2.66
1.05
1.05
1.24
1.45
1.86
1.05
1.43
1.13
1.13
1.93
Silesian
1.65
1.57
2.87
1.10
1.04
2.16
1.52
1.04
1.52
1.52
2.09
Tamil
1.35
1.26
3.17
1.17
1.06
1.17
1.42
1.42
6.15
Tamasheq (Latin script)
1.71
1.64
2.55
1.01
0.95
0.95
1.52
1.52
1.99
Tamasheq (Tifinagh script)
3.59
2.29
0.94
7.74
0.94
2.43
2.43
5.37
Tatar
1.81
1.41
1.55
3.15
1.01
1.21
1.21
2.88
Teingu
1.33
1.42
1.01
1,21
1.01
1.34
1.34
7,06
Tajik
2.14
1.62
2.01
1.11
3.29
2.39
1.11
1.57
1.57
2.90
Tagalog
1.43
1.46
2.85
1.26
1.26
1.56
2.08
1.26
1.00
1.34
1.34
2.04
Thai
1,08
0.99
2.75
0.96
1.52
1.52
2.59
Tigrinya
1.97
2.03
1.75
0.69
5.16
0.69
1.44
1.44
4.24
Pisin
1.73
1.65
2.76
1.28
1.28
1.92
1.61
2.10
1.28
1.57
1.39
1.39
2.02
Tswana
1.85
1.68
3.01
1.25
1.25
1.62
2.25
1.25
1.57
1.45
1.45
2.26
Tsonga
1.79
1.61
3.13
1.20
1.20
2.01
1.65
2.19
1.20
1.64
1.30
1.30
2.23
Turkmen
1.78
1.68
2.87
1.17
1.06
2.19
1.44
1.06
1.36
1.36
2.20
Tumbuka
1.92
1.61
3.29
1.32
1.30
1.79
1.30
1.43
2.51
Turkish
1.04
1.12
2.67
1.12
1.03
1.96
1.45
1.03
1.14
1.14
1.61
Twi
1.88
1.71
2.85
1.05
0.98
1.81
0.98
1.40
1.25
1.25
2.15
Central Atlas Tamazight
3.48
2.28
2.06
5.09
Uygbur
1.41
2.57
1.97
1.07
3.67
1.07
1.40
1.40
3.74
Ukrainian
1.21
1.33
1.86
1.02
2.75
2.85
1.02
1.28
1.28
2.51
Umbundu
1.57
1.47
2.72
1.94
Urdu
1.23
1.52
1.76
0.99
1.36
1.45
1.26
0.99
1.30
1.30
3.19
Northeru Uzbek
1.33
1.38
2,80
1.13
1.13
1.98
1.58
2.12
1.13
1.53
1.32
1.32
2.15
Venetian
1.36
1.36
2.21
1.06
1.24
1.84
1.01
1.23
1.29
1.68
Vietnamese
1.18
1.95
1.39
1.05
1.27
1.38
1.05
1.18
1.18
1.41
Waray
1.55
1.45
2.66
1.25
1.25
1.80
1.60
2.15
1.25
1.52
1.36
1.36
1.93
Wolof
1.60
1.44
2.62
L.00
1.28
1.93
0.96
1.26
1.31
1.31
1.89
Xhosa
1.50
1.35
2.73
1.06
1.06
1.52
2.05
1.06
1.45
1.21
1.21
2.04
Eastern Yiddish
1.58
1.66
1.94
1.08
4.42
2.41
1.08
1.69
1.69
2.77
Yoruba
2.27
2.06
1.28
0.97
1.64
1.24
0.97
1.52
1.52
2.69
Yue Chinese
0.93
0.95
0.87
0.31
0.31
0.55
1.05
1.05
1.17
Chinese (Simplifed)
0.97
0.92
0.93
0.34
0.95
0.34
01.55
1.11
1.11
1.07
Chinese (Traditional)
0.96
0.98
0.59
0.32
0.97
0.32
0.57
1.08
1.08
1.21
Standard Malay
0.95
1.11
2.32
1.12
1.11
1.39
1.80
1.11
1.36
1.61
Zulu
1.55
1.40
2.84
1.12
1.12
1.76
1.62
2.15
1.12
1.54
1.24
1.24
2.18
28


Reviews:
[{"review_id": "1CebDZbdlz", "reviewer": "Reviewer_Fdv1", "summary": "The paper is very interesting in highlighting the disparities in tokenization across different languages, leading to cost, latency, and long-distance modeling inequalities in LLMs. The authors show that this is not limited to one type of tokenizer or a single family of LLMs. They show that different types of tokenizers for LLMs in practice, including subword, multilingual, or byte-level tokenization, all show disparities for languages in FLORES-200.\n\nBased on these results, the paper convincingly argues that we need multilingually fair tokenizers for future LLMs, so that some languages are not in disadvantages in terms of the cost to access LLMs, the latency of the service, and the amount of data that can be processed.", "questions": "I think the paper does a very good job of explaining why there are language disparities among different tokenizers. However, I feel these reasons are known already, what are the challenges in overcoming them? ", "limitations": "The paper presents a large-scale study of disparities in tokenization across different languages. It discusses how LLMs are putting some languages to their disadvantage due to this. The paper itself does not propose a new tokenization scheme that needs to address its limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well written and the first one (to my knowledge) to study tokenization disparities in LLMs at this scale. \n\nThe presented results will be very interesting to LLM researchers and practitioners in understanding the disparities in tokenization across different languages in different LLMs. This has major consequences for some languages, putting them at a disadvantage in terms of the cost to access LLMs, the latency of the service, and the amount of data that can be processed. \n\n", "weaknesses": "Section 6 argues that future tokenizers should be based on subword tokenization and support all Unicode codepoints. However, one of the main weaknesses of the paper is that it does not develop such multilingual fairer tokenizers, and compares them with existing tokenizers in practice. Attempting to do that would have emphasized challenges in developing such tokenizers and how one can overcome them.\n\nI found the argument around disparities in long-distance modeling a bit thin. The second paragraph of Section 5.3 discusses this briefly, but additional discussion or experiments are needed to strengthen the argument. For example, it might be helpful to include analysis with multilingual document summarization (e.g., XLSum). ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> I found the argument around disparities in long-distance modeling a bit thin. The second paragraph of Section 5.3 discusses this briefly, but additional discussion or experiments are needed to strengthen the argument. For example, it might be helpful to include analysis with multilingual document summarization (e.g., XLSum).\n\nThe critical issue is that assuming a fixed context size, the more tokens required for the same content, the smaller the maximum size of the content that can be processed. For example, if the context of GPT-4 is just enough to fit a blog post in English, it can only fit 1/15 of the same blog post when translated into Shan. Experiments comparing the ratio content that can be represented would not bring additional value as they would be precisely the reciprocal of the premium values. Furthermore, Liu et al. (2023) show that even if the content can fit in the context size, the performance of language models decreases as the input sizes grow. We will extend Section 5.3 in terms of clarity of the writing and depth of argumentation to explain this better.\n\n\n> I think the paper does a very good job of explaining why there are language disparities among different tokenizers. However, I feel these reasons are known already, what are the challenges in overcoming them?\n\nPlease check our \"On the development of a multilingually fair tokenizer\" comment, where we explain what are the challenges in overcoming these language disparities and what is the approach we are working on towards addressing them.\n\n--\n\nLiu, Nelson F., et al. \"Lost in the middle: How language models use long contexts.\" arXiv preprint arXiv:2307.03172. (2023)\n"}, {"review_id": "vY2UDgpUec", "reviewer": "Reviewer_EmCj", "summary": "The paper proposes the concept of tokenizer parity as a way to measure the fairness of tokenization across different languages in natural language processing. The authors argue that achieving tokenizer parity is necessary to improve the performance of multilingual models and address potential unfairness in the cost of accessing commercial language services, processing time and latency, and the amount of content that can be provided as context to the models. The paper suggests that training language models from scratch with a multilingually fair subword tokenizer is the only approach that can effectively address tokenization unfairness and achieve tokenizer parity. The authors provide several examples to demonstrate the effectiveness of the proposed method for measuring tokenizer parity and suggest that a variation of subword tokenization is necessary to achieve parity. The paper contributes to the field of natural language processing by proposing a new metric for measuring fairness in tokenization and providing guidance on how to achieve tokenizer parity in multilingual models.", "questions": "1. Can you provide more detailed recommendations for how to integrate tokenizer parity into existing natural language processing pipelines?\n2. What is the potential impact of tokenizer disparity on downstream task accuracies?", "limitations": "The lack of clear guidance on how to integrate tokenizer parity into existing natural language processing pipelines may limit the impact of the paper's findings. Additionally, the absence of discussion on the impact of tokenizer disparity on downstream task accuracies is a limitation that could be addressed in future research. Without this information, it may be difficult for researchers and practitioners to fully understand the potential benefits of using a multilingually fair subword tokenizer.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "In terms of originality, the paper introduces the concept of \"tokenizer parity\" as a systematic way to assess the fairness of tokenization across different languages. This is a novel idea that has not been explored extensively in the literature.\n\nIn terms of quality, the paper provides a detailed analysis of the tokenization lengths of different languages using various tokenizers. The authors also propose a method to measure tokenizer parity and demonstrate its effectiveness using several examples.\n\nIn terms of clarity, the paper is well-written and easy to follow. The authors provide clear definitions of the key concepts and use examples to illustrate their points.\n\nIn terms of significance, the paper addresses an important issue in natural language processing, namely the fairness of tokenization across different languages. The concept of tokenizer parity has the potential to improve the performance of multilingual models and make natural language processing more equitable across different languages. Therefore, the paper could have a positive impact on the development of multilingual models.", "weaknesses": "One major weakness is that the paper does not provide a clear roadmap for how the concept of tokenizer parity could be integrated into existing natural language processing pipelines. While the authors suggest that training language models with a multilingually fair subword tokenizer is the only approach that can effectively address tokenization unfairness, they do not provide guidance on how this could be achieved in practice. More detailed recommendations for how to integrate tokenizer parity into existing natural language processing pipelines would be helpful for researchers and practitioners in the field.\n\nAnother potential weakness is the lack of discussion on the impact of tokenizer disparity on downstream task accuracies. This aspect is particularly interesting and may attract more attention.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> One major weakness is that the paper does not provide a clear roadmap for how the concept of tokenizer parity could be integrated into existing natural language processing pipelines. While the authors suggest that training language models with a multilingually fair subword tokenizer is the only approach that can effectively address tokenization unfairness, they do not provide guidance on how this could be achieved in practice.\n\nWe wrote an extensive explanation about the challenges of developing a multilingually fair tokenizer in our \"On the development of a multilingually fair tokenizer\" comment. We hope that addresses your concerns.\n\n> 1.Can you provide more detailed recommendations for how to integrate tokenizer parity into existing natural language processing pipelines? \n\nIn most language processing pipelines, tokenization happens before any other training or modelling efforts. Therefore, tokenization can be independently addressed without having to adjust other elements of the pipeline. Learning a tokenizer is typically posed as an optimization problem: finding the vocabulary that minimizes the number of tokens necessary to encode a given corpus. Obtaining a multilingually fair tokenizer can be thought of as the addition of a constraint that content in different languages should have approximately similar tokenized lengths. Please see our \"On the development of a multilingually fair tokenizer\" comment, where we outline one way towards implementing that.\n\n> 2.What is the potential impact of tokenizer disparity on downstream task accuracies?\n\nWe intentionally focus on tokenizers to show that disparities between languages exist even before the model is trained, and even without considering the differences in the model performance. Still, tokenization does also affect downstream performance. In the Related Works section, we cite the work by Zhang et al. (2022), which shows that a balanced tokenizer corpus results in better translation performance. The works of Hofmann et al. (2021, 2022) also show that larger tokens, as well as tokenization informed by morphological structures, result in better downstream performance. We will add them to our Related Works section and will discuss further the downstream implications of the tokenization disparities.\n\n> Additionally, the absence of discussion on the impact of tokenizer disparity on downstream task accuracies is a limitation that could be addressed in future research. Without this information, it may be difficult for researchers and practitioners to fully understand the potential benefits of using a multilingually fair subword tokenizer. \n\nWe respectfully disagree: even if there weren’t downstream improvements, the fundamental benefit of using multilingually fair tokenizers is ensuring that users of different languages pay the same for the same service, can process similar amounts of content, and can enjoy similar generation speeds. Nevertheless, as mentioned above, we will add a discussion highlighting the prior works showing that better tokenization does improve downstream task performance.\n\n--\n\nZhang, Shiyue, et al. “How robust is neural machine translation to language imbalance in multilingual tokenizer training?” Biennial Conference of the Association for Machine Translation in the Americas (2022)\n\nHofmann, Valentin, et al. “Superbizarre Is Not Superb: Derivational Morphology Improves BERT’s Interpretation of Complex Words”. Annual Meeting of the Association for Computational Linguistics (2021)\n\nHofmann, Valentin, et al. “An Embarrassingly Simple Method to Mitigate Undesirable Properties of Pretrained Language Model Tokenizers”. Annual Meeting of the Association for Computational Linguistics (2022)\n"}, {"review_id": "WawiChplxz", "reviewer": "Reviewer_Qimt", "summary": "This paper investigates the disparities between languages caused by tokenization policies used in large language models (LLMs). The authors compare the numbers of tokens needed to represent the translations of the same sentence in different languages and observe that the number of tokens needed in one language can be an order of magnitude larger than that in the target language (mostly English). They argue that these disparities have significant real-world implications such as the increased cost and performance degradations in using LLMs.", "questions": "- Would it be difficult to conduct some (preliminary) experiments regarding the proposals discussed in Section 6?\n", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- I think investigating the disparities of the cost and performance in using LLMs between different languages is an important topic.\n- The approach taken by the authors (i.e., the use of the FLORES-200 parallel corpus) seems sound.\n- The observations made in their experiments are informative.\n", "weaknesses": "- There is little technical novelty in the work.\n- The authors point out the problems caused by the disparities but do not present a concrete solution. They argue that LLMs should be trained from scratch with a multilingually fair subword tokenizer but do not provide any experimental results towards that solution. I would be interested to see how much the performance and efficiency of an LLM in English needs to be sacrificed to achieve the parity.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The authors point out the problems caused by the disparities but do not present a concrete solution. They argue that LLMs should be trained from scratch with a multilingually fair subword tokenizer but do not provide any experimental results towards that solution.\n\nPlease refer to our \"On the development of a multilingually fair tokenizer\" comment discussing a strategy we are currently working on. We also explain why it is non-trivial to implement it in practice and why we believe it is a separate work in its own right.\n\n> I would be interested to see how much the performance and efficiency of an LLM in English needs to be sacrificed to achieve the parity.\n\nThe additional cost for English would be much smaller than the total benefit for the rest of the languages. That is because increasing the vocabulary size has diminishing returns: the additional tokens correspond to increasingly rare (parts of) words. Therefore, by removing rarely used English (sub)words and replacing them with frequently used (sub)words in other languages, we would likely see an overall net befit."}, {"review_id": "o4z9bQEcbS", "reviewer": "Reviewer_qshy", "summary": "Tokenization is a crucial, yet underappreciated component of language models. This paper presents a welcome investigation into the effects of tokenization choices across languages. The paper introduces the notion of premium for language A relative to B which measures the ratio of the average number of tokens for translations of the same sentence in the two languages. The authors show that the premium relative to English can be over 10 for certain low resource languages.\n\nThe paper then highlights the negative effects of having a high premium, which include: cost, latency, and the ability to fit less content in the fixed context window of an LM. Based on these findings, the authors make a case for developing multilingually fair tokenizers where the premiums are close to one across languages.", "questions": "Could you elaborate on the following sentence “training a model from scratch with this fair tokenizer is necessary, despite potential suboptimal performance in individual languages due to vocabulary limitations”? What are the potential suboptimalities you’re referring to? Are you arguing for using a fair tokenizer regardless of any potential negative side effects?", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. Tokenization is, in my view, an underappreciated problem and this work provides an insightful analysis on the effects of the tokenizer selection.\n2. The discrepancies presented in the paper across languages are surprisingly large. As such, this paper may stimulate new research in the field to address the discrepancies.\n3. The paper is very well written and a pleasure to read.\n", "weaknesses": "1. The paper doesn’t discuss how to achieve the goal of training multilingually fair tokenizers, apart from providing an overview of previous approaches to developing multilingual tokenizers. An obvious solution is to reserve more tokens for the languages with high premiums. However, this would mean that the number of tokens for other languages would need to be reduced to keep the vocabulary size fixed. This in turn will likely hurt the performance on high-resource languages such as English and it may even end up hurting the performance for the rest of the languages if there is less cross-lingual transfer from high-resource languages (NB: The paper states that “multilingual models struggle to deliver on the promises of deep transfer learning” but I’m not aware of works suggesting that there’s zero transfer happening across languages.)\n2. The paper lacks discussion on the potential negative effects of using a fair tokenizer (such as the one presented above) and discussion on the potential means of developing fair tokenizers. Yet, it argues strongly why the switch to fair tokenizers is necessary. This makes the paper seem a bit like a position paper (which doesn’t reduce the value of the paper in my view but makes me wonder a bit whether NeurIPS is an ideal venue for it).\n3. An obvious explanation for the tokenization length differences is the widely varying amounts of available training datasets across languages. However, another possible explanation is inherent differences between languages in terms of characteristics such as morphological richness of a language. Discussion of the different explanations seems to be missing from the paper.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">The paper doesn’t discuss how to achieve the goal of training multilingually fair tokenizers, apart from providing an overview of previous approaches to developing multilingual tokenizers. An obvious solution is to reserve more tokens for the languages with high premiums.\n\nPlease see our \"On the development of a multilingually fair tokenizer\" comment, where we discuss the development of a multilingually fair tokenizer. We also discuss why giving quotas to different languages will not work: languages sharing the same script also partially share tokens.\n\n> This in turn will likely hurt the performance on high-resource languages such as English and it may even end up hurting the performance for the rest of the languages if there is less cross-lingual transfer from high-resource languages.\n\nThe additional cost for English would be much smaller than the total benefit for the rest of the languages. That is because increasing the vocabulary size has diminishing returns: the additional tokens correspond to increasingly rare (parts of) words. Therefore, removing rarely used English (sub)words and replacing them with frequently used (sub)words in other languages would likely be of a net befit overall. Furthermore, cross-lingual transfer can be bi-directional: there can be information present in the data in the less-resourced language that is not present in the high-resourced language (for example, cultural, legal or historical knowledge). Therefore, more fair tokenization can also improve cross-lingual transfer in the opposite direction.\n\n> The paper lacks discussion on the potential negative effects of using a fair tokenizer (such as the one presented above) and discussion on the potential means of developing fair tokenizers. Yet, it argues strongly why the switch to fair tokenizers is necessary. This makes the paper seem a bit like a position paper (which doesn’t reduce the value of the paper in my view but makes me wonder a bit whether NeurIPS is an ideal venue for it).\n\nWe are not aware of other negative effects of using a fair tokenizer, apart from the above-mentioned issue of slightly increasing the tokenization lengths of the most tokenisation-efficient language. Moreover, in the fairness literature, reducing the preferential treatment of a dominant group to balance the treatment of all groups is typically considered a positive effect rather than a limitation. \n\nWe do not consider our work as a position paper, especially because it is based on a comprehensive empirical evaluation. We do say that if one doesn't want to charge users of different languages more, reduce their effective context size and increase their processing time, then they would need to switch to a more fair tokenizer. However, this is backed by our results and analysis of the underlying technical factors, rather than simply our views or opinions, as customary for position papers.\n\n> An obvious explanation for the tokenization length differences is the widely varying amounts of available training datasets across languages. However, another possible explanation is inherent differences between languages in terms of characteristics such as morphological richness of a language. Discussion of the different explanations seems to be missing from the paper.\n\nThe widely varying amounts of available training datasets across languages are certainly the main culprit. However, as mentioned above, even if we use a perfectly balanced dataset, parity will not be achieved because some languages share tokens while others do not. Assessing morphological differences between languages is a challenging task as more morphological richness does not necessarily imply that more tokens would be necessary. As the information content of all languages is the same (Coupé et al., 2019), we argue that similar tokenization lengths should be achievable. We looked into the absolute number of tokens needed to encode the FLORES-200 dataset across several languages and tokenizers targeting them. The numbers cannot be directly compared, though, as the tokenizers might have different vocabulary sizes and might be trained differently (e.g. BPE vs SentencePiece). Still, the differences are much smaller than the fairness premiums we observe. That indicates that morphological differences cannot explain the drastic variations reported in the paper. Thank you for highlighting this question; we will incorporate it in the manuscript!\n\n|Language|Tokenizer|Number of tokens for FLORES-200|\n|----|----|----|\n|Standard Arabic|ArabicBERT|52834|\n|German|GottBERT|58508|\n|English|RoBERTa|52567|\n|French|CamemBERT |67031|\n|Hindi|MuRIL|62712|\n|Japanese|BERT Japanese|69209|\n|Chinese (Simplified) | RoCBert |83317|\n|Vietnamese|PhoBERT|69628|\n\n> Could you elaborate on the following sentence “training a model from scratch with this fair tokenizer is necessary, despite potential suboptimal performance in individual languages due to vocabulary limitations”? What are the potential suboptimalities you’re referring to?\n\nWe mean that one cannot get away from having to retrain the model using the fair tokenizer. In other words, it is not possible to post factum make the tokenizer of a pre-trained model fair. The suboptimal performance can stem from representing a language in the tokenization, but then training the model with little data from this language. In such a case the tokenizer would be fair but the model would not perform well in the underrepresented language. However, this problem is resolvable by one ensuring sufficient training data for all languages with respect to which the model should be fair. We agree that the sentence could be better worded and will edit it.\n\n--\n\nChristophe Coupé et al. “Different languages, similar encoding efficiency: Comparable information rates across the human communicative niche”. Science Advances (2019)\n"}, {"review_id": "tp9ZZURUeO", "reviewer": "Reviewer_okDv", "summary": "The paper studies the discrepancy in the tokenization length in different languages. It shows the unfairness of utilizing tokenization in different languages due to the cost, latency, and long-term language dependencies. The paper evaluates the unfairness between different tokenization strategies and model architectures. The paper's motivation is clear, and it provides an extensive analysis of multiple languages, showing the significance of the study and why it is essential in research and practical perspectives.", "questions": "**Questions**\n\n- Does the tokenization lengths impact the performance on languages (e.g., Shan)?\n\n**Typo**\nLine 177: XML-R => XLM-R", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Originality**\n- The paper introduces an important issue on utilizing LLM in different languages and investigates the aspects of cost, latency, and long-dependency. The problem formulation is important for multilingual researchers and practitioners in deciding what tokenization strategies are cost-effective.\n\n**Quality / Significance**\n- The paper shows a comprehensive analysis of how the different tokenization on different languages was done in different tokenization strategies (byte-based, Unicode-based, and subword-based) on the wide range of pre-trained LLM (on encoder-only, encoder-decoder, and decoder-only) and provides the quantitative evidence on the processing time of each language compared to English.\n\n**Clarity**\n- The paper is well-written, and the examples (in Japanese and Shan) give a high-level idea of why the problem introduced in the paper is important. ", "weaknesses": "- Besides the aspects mentioned by the paper, it is unclear if the choice of the tokenization strategies would impact the performance of LLM in particular languages. \n- The suggestion of \"building a multilingually fair parallel corpus\" is vague. It would be great if the authors could give a more straightforward and practical solution.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Besides the aspects mentioned by the paper, it is unclear if the choice of the tokenization strategies would impact the performance of LLM in particular languages.\n\nTokenization does also affect downstream performance. In the Related Works section, we mention the work by Zhang et al. (2022), which shows that a balanced tokenizer corpus results in better translation performance. The works of Hofmann et al. (2021, 2022) also show that larger tokens, as well as tokenization informed by morphological structures, results in better downstream performance. We will add them to our Related Works section and will discuss further the downstream implications of the tokenization disparities.\n\n> The suggestion of \"building a multilingually fair parallel corpus\" is vague. It would be great if the authors could give a more straightforward and practical solution.\n\nIn order to build a multilingually fair tokenizer one first needs a parallel corpus to use in order to ensure that the conent against which tokenization lengths are measured is indeed comparable. FLORES-200 is big enough for evaluation of tokenizers, but is too small for training ones. However, building a large parallel corpus for so many languages is extremely difficult. In our \"On the development of a multilingually fair tokenizer\" comment we offer a way to approximate such a multulingual parallel corpus via many bilingual parallel corpora.\n\n> Does the tokenization lengths impact the performance on languages (e.g., Shan)?\n\nDecoupling the effects of the tokenizations and the amount of data for a specific language is not possible. However, a recent work by Liu at el. (2023) showed that the performance of language models decreases as the input sizes grow. Hence, it is very likely that the longer tokenizations for some languages (e.g. Shan) do hurt the performance of that language, keeping everything else the same.\n\n--\n\nLiu, Nelson F., et al. \"Lost in the middle: How language models use long contexts.\" arXiv preprint arXiv:2307.03172. (2023)\n\nZhang, Shiyue, et al. “How robust is neural machine translation to language imbalance in multilingual tokenizer training?” Biennial Conference of the Association for Machine Translation in the Americas (2022)\n\nHofmann, Valentin, et al. “Superbizarre Is Not Superb: Derivational Morphology Improves BERT’s Interpretation of Complex Words”. Annual Meeting of the Association for Computational Linguistics (2021)\n\nHofmann, Valentin, et al. “An Embarrassingly Simple Method to Mitigate Undesirable Properties of Pretrained Language Model Tokenizers”. Annual Meeting of the Association for Computational Linguistics (2022)\n"}]

Author rebuttal:
# On the development of a multilingually fair tokenizer

Reviewers Fdv1, EmCj and Qimt mentioned as a shortcoming of our work that we did not develop a new multilingually fair tokenizer. We would like to highlight several reasons why we found this to be more challenging than it may seem. These are all problems that we are actively working on and we hope to soon have a follow up work addressing them.

**How to account for token sharing across languages:** 
As discussed in the paper, the byte-level, character-level and word-level tokenizers cannot achieve tokenization parity and subword tokenization is needed. However, simply training a subword tokenizer on a balanced dataset is also not sufficient as languages can share tokens. For example, “hotel” is written the same way in English, Spanish, Italian, Portuguese, Dutch, Danish, Hungarian, Polish and more. Hence, languages from more numerous language families will also witness shorter tokenization lengths while more isolated languages, e.g. Korean, would see larger language premiums. (The spelling of “hotel” in Korean is “호텔” and no other language has the same spelling as no other language uses the Korean script).

Instead, we suggest a two-stage process: first, training individual monolingual tokenizers for all target languages, and then merging them while maintaining parity. The merging can be done by starting with the 256 tokens corresponding to each value a byte can take and then repeatedly adding the most frequently used token for the language with the highest premium. This approach can account for the shared tokens across languages. For example, if at some stage Polish has the highest premium, adding the token for “hotel” will simultaneously reduce the premiums for all the other languages using the same spelling. 

However, training 200 individual tokenizers is computationally expensive. We looked into leveraging already trained tokenizers but they are often incomparable. For example, it is not trivial to combine BPE-based tokens and SentencePiece tokens. One reason is that SentencePiece pre-processes the input by splitting it at spaces and has tokens which cannot be at the beginning of a word, something incompatible with BPE. Therefore, the only methodologically clean way we see to proceed is via training individual tokenizers from scratch using the same tokenization procedure.

**The lack of large multilingual parallel corpora and how to get around it:**
A separate challenge is the need for a large parallel corpus. FLORES-200 is rather small, just 2000 sentences. It is good enough for the evaluation of parity but not sufficient for building a fair tokeniser. That is because there might be characters and words not present in FLORES-200 which should nevertheless be present in the vocabulary. Furthermore, to do proper evaluation, a split into training and test datasets is necessary. This would reduce the training data even further. Unfortunately, we are not aware of a larger parallel corpus spanning so many languages. Constructing one is extremely difficult and expensive as well. 

That is why we are looking into approximating a multilingual parallel corpus by many bilingual parallel corpuses. In such a case, the parity between, for example, Greek and Shan would not be evaluated only using Greek—Shan translations (which may not be available) but also Greek—English and English—Shan translations, Greek—German and German—Shan translations, etc. However, this requires further analysis into the conditions under which leveraging bilingual data in such a way would constitute a valid approximation.

Because of these reasons, we believe that properly building a multilingually fair tokenizer is a substantial piece of work which requires its own special treatment. We are currently working towards it but cannot possibly give it justice in the limited space of the present paper. Nevertheless, we will extend the discussion in Section 6 to better highlight these challenges.

Meta-review: This paper investigates the disparities between languages caused by tokenization methods used in LLMs and highlights a potential source of unfairness in language models.  

In general the reviewers liked the paper. 
In terms of originality the reviewers found the paper to be the first to study tokenization disparities in LLMs.  
Reviewers also liked the detailed analysis and found the work to have significant value: Addresses language fairness (EmCj, Qimt), could stimulate new research (qshy), and has practical consequences like cost and latency (Fdv1, okDv).
All reviewers found the paper to be well written.

On the other hand, reviewers commonly point out the paper's lack of practical solutions to the problem studied, as well as gaps in argumentation and exploration. Some of the more major concerns were resolved during the discussion period.
Decision: Accept (poster)