Title: Language Model Tokenizers Introduce Unfairness Between Languages
Venue: NeurIPS 2023
Accepted: yes
Keywords: LLM

language model

tokenizer

multilingual

language

fairness
Abstract: Recent language models have shown impressive multilingual performance, even when not explicitly trained for it.
Despite this, there are concerns about the quality of their outputs across different languages.
In this paper, we show how disparity in the treatment of different languages arises at the tokenization stage, well before a model is even invoked.
The same text translated into different languages can have drastically different tokenization lengths, with differences up to 15 times in some cases.
These disparities persist even for tokenizers that are intentionally trained for multilingual support.
Character-level and byte-level models also exhibit over 4 times the difference in the encoding length for some language pairs.
This induces unfair treatment for some language communities in regard to the cost of accessing commercial language services, the processing time and latency, as well as the amount of content that can be provided as context to the models.
Therefore, we make the case that we should train future language models using multilingually fair subword tokenizers.

Language Model Tokenizers Introduce Unfairness Between Languages

Abstract

Recent language models have shown impressive multilingual performance, even when not explicitly trained for it. Despite this, there are concerns about the quality of their outputs across different languages. In this paper, we show how disparity in the treatment of different languages arises at the tokenization stage, well before a model is even invoked. The same text translated into different languages can have drastically different tokenization lengths, with differences up to 15 times in some cases. These disparities persist even for tokenizers that are intentionally trained for multilingual support. Character-level and byte-level models also exhibit over 4 times the difference in the encoding length for some language pairs. This induces unfair treatment for some language communities in regard to the cost of accessing commercial language services, the processing time and latency, as well as the amount of content that can be provided as context to the models. Therefore, we make the case that we should train future language models using multilingually fair subword tokenizers.

1 Introduction

Language models are increasingly important in natural language processing tasks, as they can understand and generate human-like language. They have been deployed in applications such as virtual assistants ( Chen et al. , 2021 ; Ouyang et al. , 2022 ) , chatbots ( Kuhail et al. , 2023 ; Lee et al. , 2023 ) , machine translation ( Stahlberg , 2020 ; Ranathunga et al. , 2023 ) , and text summarization ( Kryściński et al. , 2019 ; Xu et al. , 2020 ) . As general-purpose technologies, it is also projected that Large Language Models ( LLMs ) will have a significant impact on the economy and the labour market ( Teubner et al. , 2023 ; Eloundou et al. , 2023 ) .

Such LLMs are often trained using large swaths of internet content regardless of language. Hence, these models often end up being multilingual, even if not by design. ChatGPT ( OpenAI , 2022 ) is a prominent recent example ( Bang et al. , 2023 ; Jiao et al. , 2023 ; Johnson , 2023 ) . Given the economic benefits of LLMs and LLM-derived technology, it ' s beneficial that they support multiple languages. Equal access is crucial, and multilingual support is a key component of this.

However, this multilingualism is currently treated as a curious emergent phenomenon rather than a carefully designed, controlled and managed process. The performance of LLMs has been shown to be generally lower in non-target languages, a problem especially pronounced for low-resource languages ( Virtanen et al. , 2019 ; Ahuja et al. , 2023 ) . Providing access to the same technology in different languages but moderation and safety tools only for some has resulted in dire societal consequences before ( Stecklow , 2018 ; Facebook , 2021 ; Leung , 2022 ) . Differing cost of access could also reinforce inequality in opportunities for economic mobility and social participation ( Lythreatis et al. , 2022 ) . Therefore, as LLM multilingualism emerges,

we should pay attention to ensuring comparable performance and accessibility across the supported languages, regardless of whether by design or by chance.

This work demonstrates how the unequal treatment of languages arises at the tokenization stage, 1 well before the language model sees any data at all. For instance, the tokenizer employed by ChatGPT ( OpenAI , 2022 ) and GPT-4 ( OpenAI , 2023 ) uses about 1.6 times more tokens to encode the same text in Italian as it does in English, 2.6 times for Bulgarian and 3 times for Arabic. For Shan -the native language of people from the Shan State in Myanmar- that difference can be as high as 15 times. Unicode character and byte-level tokenization also result in drastically different encoding lengths across languages: byte-level representation of the same text is over 4 times longer for Burmese or Tibetan than Chinese.

We discuss three fairness implications of these differences in tokenization:

Cost: Commercial services charge users per token or Unicode character. In either case, these discrepancies lead to users of some languages paying at least 2.5 times more for the same task as users of English.

Latency: The number of tokens has a direct effect on the processing time for a task. Some languages can require twice the time to process the same content as English. This may be critical for real-time applications like emergency services.

Long context processing: Many models have a fixed-size context. Users of languages that are more token-efficient can use these systems to process or generate texts that may be more than an order of magnitude longer than users of other languages. This may lead to significant discrepancies in the quality of service.

Therefore, we make the case for multilingual tokenization parity : tokenizers should produce similar encoded lengths for the same content across languages. Hence, we advocate for multilingually fair tokenizers for the next generation of language models.

2 Intriguing Properties of Tokenization Across Languages

Subword tokenization is currently the preferred approach for state of the art language models ( Kudo and Richardson , 2018 ) . In this section, we show how artefacts from data collection might result in technical terms or rare words having dedicated tokens, while more commonly used words and non-Latin characters end up requiring multiple tokens.

Using large corpora scraped from the internet results in peculiar choices for tokens. For instance, GPT-2 contains glitch tokens which can be usernames or concepts from games ( Rumbelow and Watkins , 2023b ; Miles and Riley , 2023 ) . As an example, BuyableInstoreAndOnline , likely coming from an online store backend, has a dedicated token. Another such token is rawdownloadcloneembedreportprint .

While such obscure terms get their own tokens, the frequently used Arabic word ' لماذا ' ( meaning ' why ') is broken into letters with each letter having its own token. The same word in Bulgarian (' защо ') is not only broken down to letters, but some of the letters require two tokens to be represented, resulting in 6 tokens for this 4 letter word.

One may argue that this is because Arabic and Bulgarian are not target languages for GPT-2. However, glitch tokens also exist for Japanese: there are dedicated tokens for 'ゼウス', the name of the ancient Greek god Zeus and 'サ -ティワン', the name of an ice cream chain ( Rumbelow and Watkins , 2023a ) . At the same time, GPT-2 requires 3 tokens to represent the much more commonly used kanji character for ' to say ':

In fact, more than half of the Japanese kanji characters require three tokens.

The existence of glitch tokens like 'ゼウス' and 'サ -ティワン' despite the lack of a dedicated token for '言' shows that tokenizers are heavily influenced by the biases of the corpus source. If one uses non-natural inputs, log files, or specialist forums, the tokenizer vocabulary would reflect this. While cl100k_base , the tokenizer used for the newer ChatGPT and GPT-4, may not have glitch tokens it still requires two tokens to represent some Cyrillic letters and three tokens for more than 65% of kanji characters. Therefore, to place all languages on an equal footing, it is important to have the tokens balanced across languages.

3 Measuring Tokenizer Parity

To demonstrate that the above examples are not anecdotal evidence, we introduce the notion of tokenizer parity to systematically assess how fairly tokenizers treat equivalent sentences in different languages. Parity occurs when a tokenizer exhibits similar tokenized lengths for the same sentence in different languages. Take a sentence s A in language A and its translation s B to language B . Then, a tokenizer t achieves parity for A with respect to B at s A and s B if | t ( s A ) | / | t ( s B ) | ≈ 1 , where t ( s A ) is the tokenization of the sentence s A and | t ( s A ) | represents its length. The ratio | t ( s A ) | / | t ( s B ) | is the premium for A relative to B . 2

4 Tokenization Length Differences Across Languages

Languages vary significantly in the number of tokens required to encode the same content, as demonstrated in the examples in Section 2. Hence, following Section 3, we measure the tokenization premium of different tokenizers. To this end, we use the FLORES-200 parallel corpus, comprising of the same 2000 sentences taken from Wikipedia and human-translated to 200 different languages ( Guzmán et al. , 2019 ; Goyal et al. , 2021 ; Costa-jussà et al. , 2022 ) . We look at subword tokenization models which target English, languages other than English, language varieties, multi-lingual tokenizers, as well as tokenizer-free ( byte-level ) modelling.

4.1 Parity for English-centric Models

As most models target English, we report in Table 1 the tokenization parity for a subset of languages in FLORES-200. The parities for all 200 languages are in Appendix C. 3 GPT2 ( Radford et al. , 2019 ) , RoBERTa ( Liu et al. , 2019 ) , and the r50k_base , p50k_base and p50k_edit tokenizers ( OpenAI , 2022 ) have close 4 tokenization lengths so we report them together. ChatGPT and GPT-4 share the same cl100k_base tokenizer and are also reported together. Some models, such as FlanT5 ( Chung et al. , 2022 ) , use a special UNK token to model unknown symbols not encountered during training. Hence, to ensure a fair comparison, we report only languages where no more than 10% of the input characters are mapped to UNK tokens ( marked with -).

Table 1 shows large variations in the tokenizer parity for all tokenizers. For GPT-2 and RoBERTa, Pangasinan, the language with shortest tokenization, is already 66% more expensive to process than English. ChatGPT and GPT-4 are slightly closer to parity, likely

Table 3: Tokenizer premiums on the FLORES-200 dataset for the MuRIL model focusing on 16 Indian languages and English . The premium is computed with respect to English.

MuRIL

English

Nepali

Bengali

Marathi

Hindi

Malayalam

Gujarati

Sanskrit

Telugu

Odia

Sindhi

Assamese

Urdu

Eastern Panjabi

Kashmiri

(

Arabic

Kashmiri

(

)

Devanagari

)

1.00

1.01

1.01

1.06

1.06

1.06

1.16

1.18

1.19

1.21

1.21

1.21

1.22

1.24

1.26

1.35

1.75

1.75

due to their larger vocabulary size. However, the cheapest languages, Portuguese, Pangasinan and German, still see a premium of 50% when compared to English. Shan has the worst tokenizer parity for all four models. Take as an example ' မ ႂ်း ', one of the Shan words for ' you '. It is tokenized by ChatGPT and GPT-4 as:

25870 247

157 224 224

25870 118

25870 116

မ

◌ ႂ

◌း

This word is constructed from one consonant and three diacritics. As the diacritics are encoded separately, there are four Unicode codepoints for this Shan character, resulting in 9 tokens. The English ' you ' has three characters but a single token.

FlanT5 has more than 10% UNK tokens for 42% of languages ( -in Table 1 ). It has a higher premium than the other tokenizers for all other languages except German and Romanian.

Summary. All four English-centric tokenizers we consider are far from tokenization parity. Portuguese is closest to parity with English for the ChatGPT and GPT-4 tokenizer but still requires about 50% more tokens for the same content. Shan is furthest from parity for this tokenizer with 15 times longer encodings compared to English. FlanT5 is closer to parity with its premium range 1.37-3.41 but it encodes only 54% of the languages, so we cannot say that it is more multilingually fair than the other tokenizers.

4.2 Parity for Models with Other Target Languages

There are models targeting languages other than English as well. Table 2 shows six such models based on the BERT architecture ( Devlin et al. , 2019 ) : ArabicBERT ( Safaya et al. , 2020 ) , RoCBert for Chinese ( Su et al. , 2022 ) , CamemBERT for French ( Martin et al. , 2020 ) , GottBERT for German ( Scheible et al. , 2020 ) , BERT Japanese ( Tohoku NLP Group , 2019 ) and PhoBERT for Vietnamese ( Nguyen and Nguyen , 2020 ) .

Bulgarian

Burmese

Chinese

(

Simplified

)

Chinese

(

Traditional

Dzongkha

English

Italian

Japanese

Shan

Standard Arabic

Standard Tibetan

Tumbuka

Yue Chinese

CANINE

UTF-32 bytes

1.04

1.24

0.34

0.32

1.25

1.00

1.18

0.44

1.42

0.88

1.13

1.28

1.30

0.31

ByT5

UTF-8 bytes

1.89

3.51

0.93

0.89

3.64

1.00

1.19

1.27

3.94

1.60

3.31

1.28

1.32

0.87

The English premium for GottBERT ( 1.35 ) is lower than those for Dutch ( 1.73 ) and Luxembourgish ( 1.75 ), which are more linguistically similar to German. CamemBERT is similar: English has the lowest premium ( 1.20 ), while Catalan ( 1.59 ) and Friulian ( 1.66 ) have higher premiums. PhoBERT also has English with the lowest tokenizer premium ( 1.20 ). Thus, even models targeting other languages exhibit a preference for English tokenization.

RoCBert and BERT Japanese differ by having the other target language as the one closest to parity, possibly due to the partially shared script. ArabicBERT demonstrates a similar behaviour, with Central Kanuri ( 1.27 ) and Acehnese ( 1.73 ), both written in Arabic script, and with English at 1.82. Sharing writing systems seems to improve tokenization parity.

Across all tokenizers, the premium for English relative to the respective target language is significantly lower than the premium of RoBERTa for that target language. This asymmetry between English and all other languages likely stems from the extensive incorporation of English in documents written in other languages ( Zhang et al. , 2022 ) .

We also consider MuRIL, a BERT-based model trained on 16 Indian languages and English ( Khanuja et al. , 2021 ) . Despite the model ' s focus on Indian languages, it remains most token-efficient for English ( see Table 3 ).

Unequal treatment of dialects or linguistic varieties can lead to social and economic disadvantages making it important to also study the tokenization differences between the ' standard ' language and its varieties. For Swiss German and the Mauritian and Haitian Creoles, there are large differences in tokenization lengths compared respectively to High German ( on GottBERT ) and French ( on CamemBERT ). English is much closer to parity for both models than these language varieties. Therefore subword tokenizers might not be able to generalize to language varieties, such as dialects and creoles. The tokenizers of ArabicBERT and BERT Japanese, however, are close to parity across various dialects of both languages and have lower premiums for the dialects than for English. This is likely due to the good representation of the dialects in the dataset as well as the dialects being linguistically closer to the respective standard languages. The detailed analysis is deferred to Appendix B.

Summary. We observed that the tokenizers targeting French, German and Vietnamese have English as the language closest to parity, rather than more linguistically close languages. On the other hand, tokenizers for Arabic, Chinese and Japanese have lower premiums for languages they share a script with. Notably, despite targeting Indian languages, MuRIL still has the shortest tokenizations for English. Finally, across all tokenizers, the premium for English is lower than the premium for the same language for the English-centric RoBERTa. Hence, we conclude that tokenizers for other languages give English preferential treatment.

)

1

5

10

15

Tokenization premium

Yue Chinese

Yoruba

Turkish

Thai

Telugu

Swedish

Std. Malay

Std. Arabic

Shan

Pangasinan

Odia

Nuer

Norwegian Nynorsk

Norwegian Bokmål

Meitei (Bengali sc.)

Malayalam

Lao

Khmer

Javanese

Japanese

Italian

Indonesian

Haitian Creole

Georgian

Galician

English

Danish

Chinese (Traditional)

Chinese (Simplified)

Central Kurdish

Catalan

Burmese

Bulgarian

Asturian

RoBERTa

XLM-RoBERTa

4.3 Parity for Multilingual Models

There has been a growing interest in multilingual language models, particularly for translation ( Dabre et al. , 2020 ) . As these models are intended to support a variety of languages, one would expect them to be close to tokenizer parity. We compare several such multilingual models: XML-R ( Conneau et al. , 2020 ) , NLLB ( Costa-jussà et al. , 2022 ) , M2M100 ( Fan et al. , 2021 ) and mT5 ( Xue et al. , 2020 ) . All of these models use the SentencePiece tokenizer with upsampling for rare languages. The final model, BLOOM ( Scao et al. , 2022 ) , uses byte-level BPE instead of SentencePiece and is designed to maintain similar ratios of tokens per word for each language as reference monolingual tokenizers.

BLOOM and NLLB encode all languages with less than 10% UNK tokens, respectively thanks to byte-level BPE tokenization and being trained on the same 200 languages as FLORES200 ( see Table 4 ). The other three models fail to encode at least one language. All five models have languages with premiums of more than 2.5. Still, all models are better than the English-centric models in Table 1. Figure 1 shows how XLM-R is much closer to parity than RoBERTa ( on which it is based ), over all languages it can encode. However, none of the models uniformly reaches parity across all languages. Therefore even models which are intentionally designed to be multilingual suffer from a lack of tokenization parity.

Summary: Multilingual models can improve the tokenization parity for different languages but challenges remain in achieving tokenization parity across all languages.

4.4 Parity for Byte-level Tokenization Models

Byte-level representation is crucial for multilingual support, as it encodes any Unicode codepoint, even if unseen during training. One can also bypass vocabulary construction and directly employ the 256 byte values, enabling end-to-end training ( byte-level tokenization ). CANINE ( Clark et al. , 2022 ) is a large model that operates at the Unicode codepoint level rather than the byte level. The CANINE tokenizer is thus equivalent to the UTF-32 encoding, resulting in an implicit tokenizer with a vocabulary of 1,114,112. ByT5 ( Xue et al. , 2022 ) , on the other hand, uses the UTF-8 encoding: an implicit vocabulary of 256 tokens. 5

0.0

0.2

0.4

0.6

0.8

1.0

Tokenisation length for the FLORES-200 parallel corpus

1e6

14

16

18

20

22

24

RoBERTa execution time [s]

English

German

French

Shan

Bhojpuri

Sango

Malay

Hebrew

Zulu

Bulgarian

Burmese

Dzongkha

Italian

Japanese

Odia

Portuguese

Spanish

Standard

Arabic

Vietnamese

Chinese (Simp.)

Chinese (Trad.)

Yue Chinese

Korean

Script family:

Arabic

Armenian

Northern Brahmi

Southern Brahmi

CJK

Greek

Ge ez

Georgian

Hebrew

Berber

These byte-level models can represent any Unicode codepoint without an explicit tokenization step but there are still significant tokenization disparities. For CANINE, Shan has a premium of 4.58 relative to Yue Chinese. This can be attributed to the fact that CANINE provides a single token for each Unicode codepoint, which results in Chinese being more token-efficient ( with a premium range 0.31-0.34 relative to English for the three Chinese languages ) as each character is treated as a single token. This encoding also puts Shan at a disadvantage, as its encoding relies on diacritics represented as separate Unicode codepoints. Other languages, such as Tok Pisin and Tumbuka, which use the Latin script but require more characters than English for the same text, also face similar challenges.

Tokenization disparity is also present in the ByT5 model. The tokenization premium for ByT5 ranges from 0.87 ( for Yue Chinese ) to 3.94 ( for Shan ). The introduction of the variablewidth UTF-8 encoding of Unicode characters in ByT5 creates another issue of unequal treatment. ASCII characters, which are sufficient for English, require only one byte. Other Latin script characters, as well as Greek, Cyrillic, Coptic, Armenian, Hebrew, Arabic and Syriac, require two bytes, while Chinese, Japanese and Korean characters require three bytes. Therefore, the tokenization of Chinese and Japanese is about three times as long for ByT5 as it is for CANINE ( Table 5 ). Shan ' s premium of 3.94 is due to the fact that all its consonants and diacritics require three bytes. For example, the word ' မ ႂ်း ' is encoded by ByT5 as 12 tokens, whereas the corresponding ' you ' requires 3 tokens. The situation is similar for other languages like Dzongkha, Tibetan and Burmese.

Summary. Byte-level models also fail to achieve parity among the languages from FLORES-200 exhibiting a premium of over 4 times for some language pairs. There are two sources of multilingual tokenizer disparities. First, there are natural differences in the number of characters used in different languages to communicate the same content. Second, the UTF-8 standard uses different number of bytes to encode codepoints of different scripts.

5 Fairness Implications of Tokenization Length Differences

We showed that no matter whether one uses subword, multilingual, or byte-level tokenization, none of the tokenizers gets close to parity for all languages in FLORES-200. This lack of tokenization parity is not merely a curiosity: it leads to unfairness in the cost to access language models, the latency of the service and the amount of data that can be processed.

5.1 Cost

It is increasingly common to access LLMs as paid API services. One pricing approach, employed by OpenAI at the time of writing, 6 is to charge per token. Therefore, the tokenization premiums discussed in Section 4 directly map to cost premiums. For ChatGPT and GPT-4, the cost to process a text in German or Italian is about 50% higher than to process the same text in English ( Table 1 ). Using them in Dzongkha, Odia, Santali or Shan, the most expensive languages for these services, costs more than 12 times more than in English.

Another pricing strategy is per Unicode character: the approach currently taken by the Google Cloud Natural Language service. 7 However, as we showed in Section 4.4, the same content can have very different lengths when measured in Unicode characters. Burmese, Dzongkha, Shan, Tok Pisin or Tumbuka require more than 4 times more characters than Yue Chinese for the same text, resulting in a proportional cost difference. Therefore, both the per-token and the per-character approaches result in large disparities in the cost for users of different languages to use the exact same service.

5.2 Latency

High latency of real-time interactions for users of certain languages can result in a suboptimal experience and communication breakdowns. For customer support or emergency services, delays in response time can lead to miscommunication or delayed assistance.

As some languages have significantly longer tokenized inputs, they would also experience longer processing times. The transformer attention mechanism has a quadratic complexity in the number of input tokens ( Keles et al. , 2023 ) . However, the full model architecture contains other submodules and therefore the overall complexity might be different.

To assess the effect of the tokenization length on the latency, in Figure 2 we plot the computation time of RoBERTa against the tokenization lengths. It appears that the processing time is linear in the tokenization length rather than quadratic, showing a strong correlation between sequence length and execution time. Therefore, tokenization disparities across languages also affect the latency and processing time for text in these languages.

As expected, English is on the left lower corner, having the shortest tokenization and one of the fastest processing times. Shan is on the other extreme with the longest tokenization length and execution time ( almost twice that of English ). We can also observe clear trends dependent on the script used. Latin script and other Greek-derived scripts show the shortest tokenization lengths and processing times followed by the Chinese-Japanese-Korean ( CJK ) and Arabic languages. Other predominantly Asian and African scripts have longer tokenization lengths and processing times.

The latency implications of tokenization disparity are not limited to text models. Speech recognition models often produce a series of tokens as their output sequentially. Similarly, speech synthesis takes as an input tokenized text ( Latif et al. , 2023 ) . Therefore, differences in tokenization affect speech models too.

5.3 Long context processing

Transformers models have difficulty processing long inputs ( Liu et al. , 2023 ) . Given that the size of the input is contingent upon the tokenization process, inputs of greater length may impose a challenge for language models to adequately reason over. Such a predicament may result in reduced abilities or limited applicability for languages with high tokenization premiums. For example, RoBERTa has a fixed block size of 512, GPT-2 has 768, 1024, 1280, or 1600 Radford et al. ( 2019 ) , GPT-4 comes in 8,000 and 16,000 context variants. 8 These models cannot process inputs longer than that. Therefore, one can process less than a tenth of the content in languages like Burmese and Dzongkha than they can in English.

Alongside inconveniencing the users of these languages, this can also result in diminished performance on automated systems, such as content moderation. Reliable content moderation is crucial for tackling hate speech and diminished performance has already been shown to fail to prevent its spread ( Stecklow , 2018 ; Facebook , 2021 ) . Therefore, reduced long context capabilities for some languages could have severe real-world impacts.

6 Towards Multilingual Tokenization Fairness

Section 5 showed that high values of tokenization parity for a language lead to increased cost and latency and decreased capacity for long context processing. In this section, we argue that training language models from scratch with a multilingually fair subword tokenizer is the only approach that can effectively address all these aspects of tokenization unfairness.

Subword tokenization is necessary to achieve parity. In Section 4.4, we showed that neither character-level nor byte-level input representation can achieve tokenization parity. Therefore, a variation of subword tokenization is necessary. For example, Chinese characters could be individual tokens, Latin characters might be represented as tokens with an average length of about 3 characters while pairs of Burmese characters and their diacritics being assigned single tokens. Such an approach would account for Chinese requiring one-third the characters English does ( as shown in Table 5 ).

A separate tokenizer for determining the processing cost is not sufficient. An easy patch for existing models is to use a separate tokenizer for calculating how much a user should be charged. Using one tokenizer for computing the cost and another to process

0

10000 20000 30000 40000 50000 60000 70000 80000 90000100000

Vocabulary size

52000

54000

56000

58000

60000

62000

64000

66000

68000

70000

72000

74000

76000

78000

80000

Tokens necessary to encode FLORES-200

With one-third of the vocabulary,

English sequences will become

just 10% longer for ChatGPT/GPT-4

A 10-fold reduction in the vocabulary

would result in only

30% longer sequences for English.

the input can easily be applied to existing systems without the need to retrain the LLM itself. However, as the tokenizer for the language model is unchanged, this approach would still suffer from latency and inability to process long contexts. Therefore, to ensure similar processing times and long context capabilities across languages, the language model has to be trained with a multilingually fair tokenizer.

The tokenization needs to support all Unicode codepoints. Amongst all tokenizers we examine in this paper, the ones which encode all FLORES-200 languages all have one thing in common: they build their tokenization on top of Unicode representation, allowing them them to represent all characters. Therefore, a multilingually fair tokenizer should also start from a Unicode ( or equivalent ) encoding. Considering that subword tokenization is necessary, building the vocabulary from UTF-8 would likely result in a smaller dictionary than building it on top of UTF-32. Hence, UTF-8 is likely the more appropriate choice.

Building a multilingually fair parallel corpus. Building and evaluating multilingually fair tokenizers requires attention to the parallel corpus used. One must ensure a balanced representation of topics, otherwise, the resulting tokenizer might end up being multilingually fair only for a subset of topics. The presence of named entities must also be balanced. For example, in FLORES-200, there are many English-centric names and institutions, which might skew the results in favour of English. Additionally, the same sentence can have different translations with varying tokenization lengths. To account for this, a diversity of translations could ensure tokenization fairness across languages. These limitations also hold for the results in this paper. Hence, developing a well-curated and diverse parallel corpus is crucial for the development and evaluation of a multilingually fair tokenizer.

Building a multilingually fair tokenizer from monolinugal tokenizers. As discussed in Section 4, byte-level, character-level and word-level tokenizers cannot achieve tokenization parity and subword tokenization is needed. However, simply training a subword tokenizer on a balanced dataset is also not sufficient as languages can share tokens. For example, ' hotel ' is written the same way in English, Spanish, Italian, Portuguese, Dutch, Danish, Hungarian, Polish, etc. Hence, languages from more numerous language families will also witness shorter tokenization lengths while more isolated languages and scripts, e.g. Korean, would see larger language premiums: ' hotel ' in Korean is '호텔' and no other language has the same spelling as no other language uses the Korean script.

To address this issue, we suggest a two-stage process towards building a multilingually fair tokenizer. First, train individual monolingual tokenizers for all target languages. Then, merge them while maintaining parity. The merging can be done by starting with the 256 tokens corresponding to each value a byte can take and then repeatedly adding the most frequently used token for the language with the highest premium.

While a multilingually fair tokenizer would lead to more tokens being needed for the dominant language, this additional cost would likely be much smaller than the benefit for the rest of the languages. The vocabulary size has diminishing returns: the additional tokens correspond to increasingly rare ( parts of ) words. For example, with only a third of the vocab-

ulary, English sequences will become just 10% longer for ChatGPT/GPT-4 ( see Figure 3 ). Therefore, by removing rarely used tokens of the dominant language and replacing them with frequently used tokens in other languages, we would likely see an overall net benefit.

7 Related Works

Fairness and bias in language models. The rapid increase in the size of language models has raised concerns regarding their biases and unfairness ( Bender et al. , 2021 ) . For example, Bolukbasi et al. ( 2016 ) , May et al. ( 2019 ) and Nadeem et al. ( 2021 ) showed that stereotypes and biases exist in language models, while Magee et al. ( 2021 ) identified the presence of intersectional biases which may be resistant to debiasing techniques. Language models were also shown to rely on social biases in question answering ( Parrish et al. , 2022 ) . Another challenge is the generation of toxic content which can occur even without prompting ( Gehman et al. , 2020 ) . Interestingly, Gururangan et al. ( 2022 ) point out that datasets consider one type of English as a higher quality depending on the location of the writer rather than on factuality or literary acclaim. Moreover, Ramesh et al. ( 2023 ) and Levy et al. ( 2023 ) highlighted the need to consider fairness issues of languages other than English, as they may have distinct sources of bias and solutions for English may not be applicable.

Multilingual performance. One approach towards similar multilingual performance is to frame languages as entities as recently proposed by Choudhury and Deshpande ( 2021 ) . Another method is to separately train vocabularies for different language clusters to balance cross-lingual and language-specific tokens ( Chung et al. , 2020 ) . Still, multilingual models struggle to deliver on the promises of deep transfer learning for lower-resourced languages ( Virtanen et al. , 2019 ) and perform differently depending on the script and resource level of the language ( Bang et al. , 2023 ) . Ahuja et al. ( 2023 ) found that generative models perform better on higher-resource languages and languages that use the Latin script, possibly due to the context length restrictions for some languages. Zhang et al. ( 2022 ) show that a balanced tokenizer corpus results in better translation performance. Separately, Hofmann et al. ( 2021 , 2022 ) show that the BPE results in suboptimal token choices even for English and demonstrate that addressing this issue boosts performance. Similarly, Rajab ( 2022 ) and Oladipo et al. ( 2022 ) discuss how tokenization affects performance for African languages.

Measuring tokenization lengths. Zhang et al. ( 2022 ) suggested using the ratio of the average sentence length in tokens to the length in characters as a measure of closeness to the character level. However, this method may not be suitable for comparing languages due to differences in sentence length across languages. On the other hand, Ács ( 2019 ) and Scao et al. ( 2022 ) measure the number of tokens created per word, but this method may not be effective for comparing languages due to differences in semantic content per word and the lack of word delineation in some languages. Rust et al. ( 2021 ) show that mBERT ( Devlin et al. , 2019 ) breaks down English words the least, in line with our findings of English receiving special treatment. However, to the best of our knowledge, we are the first to leverage a parallel corpus to compare tokenization lengths across languages.

8 Conclusion

This paper highlights the significant disparities in tokenization across different languages which can lead to unequal treatment and disadvantages for certain language communities. The findings reveal that even tokenizers explicitly trained for multilingual support exhibit tokenization lengths that vary by up to a factor of 13. Furthermore, character-level and byte-level models also demonstrate encoding length discrepancies that are more than 4 times longer. These disparities have important real-world implications including increased costs for accessing commercial language services, longer processing times and limitations on the amount of contextual information provided to language models. To address these issues, we propose the development of multilingually fair tokenizers for future language models emphasizing the importance of ensuring comparable performance and accessibility across supported languages. By achieving tokenization parity, we can mitigate inequalities and promote fair access to language technologies across diverse linguistic communities.

References

Ahmed Abdelali, Francisco Guzman, Hassan Sajjad, and Stephan Vogel. 2014. The AMARA corpus: Building parallel language resources for the educational domain. In Proceedings of the Ninth International Conference on Language Resources and Evaluation ( LREC ' 14 ) . European Language Resources Association ( ELRA ).

Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David R. Mortensen, Noah A. Smith, and Yulia Tsvetkov. 2023. Do all languages cost the same? Tokenization in the era of commercial language models. arXiv preprint arXiv:2305.13707 .

Kabir Ahuja, Rishav Hada, Millicent Ochieng, Prachi Jain, Harshita Diddee, Samuel Maina, Tanuja Ganu, Sameer Segal, Maxamed Axmed, Kalika Bali, and Sunayana Sitaram. 2023. MEGA: Multilingual evaluation of generative AI. arXiv preprint arXiv:2303.12528 .

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-VL: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966 .

Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. 2023. A multitask, multilingual, multimodal evaluation of ChatGPT on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.04023 .

Loïc Barrault, Yu-An Chung, Mariano Cora Meglioli, David Dale, Ning Dong, PaulAmbroise Duquenne, Hady Elsahar, Hongyu Gong, Kevin Heffernan, John Hoffman, et al. 2023. SeamlessM4T - massively multilingual & multimodal machine translation. arXiv preprint arXiv:2308.11596 .

Reem Bassiouney. 2009. Arabic Sociolinguistics . Edinburgh University Press.

Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency .

Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. 2000. A neural probabilistic language model. Advances in Neural Information Processing Systems .

Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. In Advances in Neural Information Processing Systems .

Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdulrahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, et al. 2018. The MADAR Arabic dialect corpus and lexicon. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation ( LREC2018 ) .

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 .

Monojit Choudhury and Amit Deshpande. 2021. How linguistically fair are multilingual pre-trained language models? In Proceedings of the AAAI Conference on Artificial Intelligence .

Hyung Won Chung, Dan Garrette, Kiat Chuan Tan, and Jason Riesa. 2020. Improving multilingual models with language-clustered vocabularies. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing ( EMNLP ) .

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instructionfinetuned language models. arXiv preprint arXiv:2210.11416 .

Junyoung Chung, Kyunghyun Cho, and Yoshua Bengio. 2016. A character-level decoder without explicit segmentation for neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics ( Volume 1: Long Papers ) .

Jonathan H. Clark, Dan Garrette, Iulia Turc, and John Wieting. 2022. Canine: Pre-training an Efficient Tokenization-Free Encoder for Language Representation. Transactions of the Association for Computational Linguistics .

Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Édouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Annual Meeting of the Association for Computational Linguistics .

Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672 .

Raj Dabre, Chenhui Chu, and Anoop Kunchukuttan. 2020. A survey of multilingual neural machine translation. ACM Computing Surveys .

Raj Dabre and Aneerav Sukhoo. 2022. MorisienMT: A dataset for Mauritian Creole machine translation. arXiv preprint arXiv:2206.02421 .

Michel DeGraff. 2007. Kreyòl Ayisyen, or Haitian Creole ( Creole French ).

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pretraining of deep bidirectional transformers for language understanding. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies .

Pelin Dogan-Schönberger, Julian Mäder, and Thomas Hofmann. 2021. SwissDial: Parallel multidialectal corpus of spoken Swiss German. arXiv preprint arXiv:2103.11401 .

Tyna Eloundou, Sam Manning, Pamela Mishkin, and Daniel Rock. 2023. GPTs are GPTs: An early look at the labor market impact potential of large language models. arXiv preprint arXiv:2303.10130 .

Facebook. 2021. Sri Lanka human rights impact assessment. Accessed on April 11, 2023.

Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. 2021. Beyond English-centric multilingual machine translation. The Journal of Machine Learning Research .

Philip Gage. 1994. A new algorithm for data compression. C Users Journal .

Yingqiang Gao, Nikola I. Nikolov, Yuhuang Hu, and Richard H.R. Hahnloser. 2020. Character-level translation with self-attention. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics .

Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. RealToxicityPrompts: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP . Association for Computational Linguistics.

Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc ' Aurelio Ranzato, Francisco Guzmán, and Angela Fan. 2021. The FLORES-101 evaluation benchmark for low-resource and multilingual machine translation. Transactions of the Association for Computational Linguistics .

Suchin Gururangan, Dallas Card, Sarah K. Dreier, Emily K. Gade, Leroy Z. Wang, Zeyu Wang, Luke Zettlemoyer, and Noah A. Smith. 2022. Whose language counts as high quality? Measuring language ideologies in text data selection. arXiv preprint arXiv:2201.10474 .

Francisco Guzmán, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, and Marc ' Aurelio Ranzato. 2019. Two new evaluation datasets for low-resource machine translation: Nepali-English and Sinhala-English. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing ( EMNLP-IJCNLP ) .

Shiro Hattori. 1973. Japanese dialects. In Diachronic, areal, and typological linguistics .

Valentin Hofmann, Janet Pierrehumbert, and Hinrich Schütze. 2021. Superbizarre is not superb: Derivational morphology improves BERT ' s interpretation of complex words. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing ( Volume 1: Long Papers ) .

Valentin Hofmann, Hinrich Schuetze, and Janet Pierrehumbert. 2022. An embarrassingly simple method to mitigate undesirable properties of pretrained language model tokenizers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( Volume 2: Short Papers ) .

Michael A. Hogg, Nicholas Joyce, and Dominic Abrams. 1984. Diglossia in Switzerland? A social identity analysis of speaker evaluations. Journal of Language and Social Psychology .

Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, and Zhaopeng Tu. 2023. Is ChatGPT a good translator? Yes with GPT-4 as the engine. arXiv preprint arXiv:2301.08745 .

Johnson. 2023. ChatGPT is a marvel of multilingualism. The Economist .

Alan S. Kaye. 2001. Diglossia: The state of the art. International Journal of the Sociology of Language .

Feyza Duman Keles, Pruthuvi Mahesakya Wijewardena, and Chinmay Hegde. 2023. On the computational complexity of self-attention. In International Conference on Algorithmic Learning Theory .

Simran Khanuja, Diksha Bansal, Sarvesh Mehtani, Savya Khosla, Atreyee Dey, Balaji Gopalan, Dilip Kumar Margam, Pooja Aggarwal, Rajiv Teja Nagipogu, Shachi Dave, Shruti Gupta, Subhash Chandra Bose Gali, Vish Subramanian, and Partha Talukdar. 2021. MuRIL: Multilingual representations for Indian languages. arXiv preprint arXiv:2103.10730 .

Wojciech Kryściński, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Neural text summarization: A critical evaluation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing ( EMNLP-IJCNLP ) , Hong Kong, China.

Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics ( Volume 1: Long Papers ) .

Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations .

Mohammad Amin Kuhail, Nazik Alturki, Salwa Alramlawi, and Kholood Alhejori. 2023. Interacting with educational chatbots: A systematic review. Education and Information Technologies .

Siddique Latif, Aun Zaidi, Heriberto Cuayahuitl, Fahad Shamshad, Moazzam Shoukat, and Junaid Qadir. 2023. Transformers in speech processing: A survey. arXiv preprint arXiv:2303.11607 .

Jason Lee, Kyunghyun Cho, and Thomas Hofmann. 2017. Fully character-level neural machine translation without explicit segmentation. Transactions of the Association for Computational Linguistics .

Peter Lee, Sebastien Bubeck, and Joseph Petro. 2023. Benefits, limits, and risks of GPT-4 as an AI chatbot for medicine. New England Journal of Medicine .

Heather Lent, Emanuele Bugliarello, Miryam de Lhoneux, Chen Qiu, and Anders Søgaard. 2021. On language models for creoles. In Proceedings of the 25th Conference on Computational Natural Language Learning . Association for Computational Linguistics.

Heather Lent, Kelechi Ogueji, Miryam de Lhoneux, Orevaoghene Ahia, and Anders Søgaard. 2022. What a creole wants, what a creole needs. In Proceedings of the Thirteenth Language Resources and Evaluation Conference .

Janny Leung. 2022. Shortcuts and shortfalls in Meta ' s content moderation practices: A glimpse from its oversight board ' s first year of operation. Comparative Law and Language .

Sharon Levy, Neha Anna John, Ling Liu, Yogarshi Vyas, Jie Ma, Yoshinari Fujinuma, Miguel Ballesteros, Vittorio Castelli, and Dan Roth. 2023. Comparing biases and the impact of multilingual training across multiple languages. arXiv preprint arXiv:2305.11242 .

Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023. Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2307.03172 .

Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics .

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 .

Georges Lüdi. 2007. The Swiss model of plurilingual communication. Receptive multilingualism: Linguistic analyses, language policies and didactic concepts .

Sophie Lythreatis, Sanjay Kumar Singh, and Abdul-Nasser El-Kassar. 2022. The digital divide: A review and future research agenda. Technological Forecasting and Social Change .

Liam Magee, Lida Ghahremanlou, Karen Soldatic, and Shanthi Robertson. 2021. Intersectional bias in causal language models. arXiv preprint arXiv:2107.07691 .

Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric Villemonte de La Clergerie, Djamé Seddah, and Benoît Sagot. 2020. CamemBERT: A tasty French language model. In Annual Meeting of the Association for Computational Linguistics .

Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, and Rachel Rudinger. 2019. On measuring social biases in sentence encoders. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 ( Long and Short Papers ) .

Sabrina J Mielke, Zaid Alyafeai, Elizabeth Salesky, Colin Raffel, Manan Dey, Matthias Gallé, Arun Raja, Chenglei Si, Wilson Y Lee, Benoît Sagot, et al. 2021. Between words and characters: A brief history of open-vocabulary modeling and tokenization in NLP. arXiv preprint arXiv:2112.10508 .

Rob Miles and Sean Riley. 2023. Glitch tokens - Computerphile. Accessed on April 11, 2023.

Robert Munro. 2010. Crowdsourced translation for emergency response in Haiti: the global collaboration of local knowledge. In Proceedings of the Workshop on Collaborative Translation: technology, crowdsourcing, and the translator perspective . Association for Machine Translation in the Americas.

Pieter Muysken and Norval Smith. 1994. The study of pidgin and creole languages. In Pidgins and creoles: An introduction .

Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. StereoSet: Measuring stereotypical bias in pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing ( Volume 1: Long Papers ) .

Dat Quoc Nguyen and Anh-Tuan Nguyen. 2020. PhoBERT: Pre-trained language models for Vietnamese. In Findings of the Association for Computational Linguistics: EMNLP .

Akintunde Oladipo, Odunayo Ogundepo, Kelechi Ogueji, and Jimmy Lin. 2022. An exploration of vocabulary size and transfer effects in multilingual language models for African languages. In 3rd Workshop on African Natural Language Processing .

OpenAI. 2022. Introducing ChatGPT. Accessed on April 11, 2023.

OpenAI. 2022. tiktoken. Git commit: 82facf9 .

OpenAI. 2023. GPT-4 technical report. arXiv preprint arXiv:2303.08774 .

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems .

Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. 2022. BBQ: A hand-built bias benchmark for question answering. In Findings of the Association for Computational Linguistics: ACL 2022 .

Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. 2021. UNKs everywhere: Adapting multilingual language models to new scripts. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing .

Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.

Jenalea Rajab. 2022. Effect of tokenisation strategies for low-resourced Southern African languages. In 3rd Workshop on African Natural Language Processing .

Krithika Ramesh, Sunayana Sitaram, and Monojit Choudhury. 2023. Fairness in language models beyond English: Gaps and challenges. In Findings of the Association for Computational Linguistics: EACL 2023 . Association for Computational Linguistics.

Surangika Ranathunga, En-Shiun Annie Lee, Marjana Prifti Skenduli, Ravi Shekhar, Mehreen Alam, and Rishemjit Kaur. 2023. Neural machine translation for low-resource languages: A survey. ACM Computing Surveys .

Jessica Rumbelow and Matthew Watkins. 2023a. SolidGoldMagikarp III: Glitch token archaelogy. Accessed on April 11, 2023.

Jessica Rumbelow and Matthew Watkins. 2023b. SolidGoldMagikarp ( plus, prompt generation ) . Accessed on April 11, 2023.

Charles Russ. 1990. The Dialects of Modern German: A Linguistic Survey .

Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. 2021. How good is your tokenizer? On the monolingual performance of multilingual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing ( Volume 1: Long Papers ) .

Ali Safaya, Moutasem Abdullatif, and Deniz Yuret. 2020. KUISAIL at SemEval-2020 task 12: BERT-CNN for offensive speech identification in social media. In Proceedings of the Fourteenth Workshop on Semantic Evaluation .

Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. BLOOM: A 176B-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100 .

Raphael Scheible, Fabian Thomczyk, Patric Tippmann, Victor Jaravine, and Martin Boeker. 2020. GottBERT: A pure German language model. arXiv preprint arXiv:2012.02110 .

Mike Schuster and Kaisuke Nakajima. 2012. Japanese and Korean voice search. In IEEE International Conference on Acoustics, Speech and Signal Processing ( ICASSP ) .

Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics ( Volume 1: Long Papers ) .

Pieter A. M. Seuren. 1995. Notes on the history and the syntax of Mauritian Creole. Linguistics .

Yan Shao, Christian Hardmeier, and Joakim Nivre. 2018. Universal word segmentation: Implementation and interpretation. Transactions of the Association for Computational Linguistics .

Peter Sieber and Horst Sitta. 1987. Deutsch in der Schweiz. Zeitschrift für Germanistik .

Felix Stahlberg. 2020. Neural machine translation: A review. Journal of Artificial Intelligence Research .

Steve Stecklow. 2018. Hatebook. Reuters . Accessed on April 11, 2023.

Hui Su, Weiwei Shi, Xiaoyu Shen, Zhou Xiao, Tuo Ji, Jiarui Fang, and Jie Zhou. 2022. RoCbert: Robust Chinese BERT with multimodal contrastive pretraining. In Annual Meeting of the Association for Computational Linguistics .

Lichao Sun, Kazuma Hashimoto, Wenpeng Yin, Akari Asai, Jia Li, Philip Yu, and Caiming Xiong. 2020. Adv-BERT: BERT is not robust on misspellings! Generating nature adversarial samples on BERT. arXiv preprint arXiv:2003.04985 .

Shinnosuke Takamichi and Hiroshi Saruwatari. 2018. CPJD corpus: Crowdsourced parallel speech corpus of Japanese dialects. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation ( LREC 2018 ) .

Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan. 2020. Multilingual translation with extensible multilingual pretraining and finetuning. arXiv preprint arXiv:2008.00401 .

Timm Teubner, Christoph M Flath, Christof Weinhardt, Wil van der Aalst, and Oliver Hinz. 2023. Welcome to the era of ChatGPT et al: The prospects of large language models. Business & Information Systems Engineering .

The Unicode Consortium. 2022. The Unicode standard, Version 15.0.0.

Jörg Tiedemann. 2012. Parallel data, tools and interfaces in OPUS. In Proceedings of the Eight International Conference on Language Resources and Evaluation ( LREC ' 12 ) .

Tohoku NLP Group. 2019. BERT models for Japanese NLP.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 .

Emma Trentman and Sonia Shiri. 2020. The mutual intelligibility of Arabic dialects: Implications for the language classroom. Critical Multilingualism Studies .

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in Neural Information Processing Systems .

Antti Virtanen, Jenna Kanerva, Rami Ilo, Jouni Luoma, Juhani Luotolahti, Tapio Salakoski, Filip Ginter, and Sampo Pyysalo. 2019. Multilingual is not enough: BERT for Finnish. arXiv preprint arXiv:1912.07076 .

Jonathan J. Webster and Chunyu Kit. 1992. Tokenization as the initial phase in NLP. In The International Conference on Computational Linguistics .

Jiacheng Xu, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. Discourse-aware neural extractive text summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics . Association for Computational Linguistics.

Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2022. ByT5: Towards a token-free future with pretrained byte-to-byte models. Transactions of the Association for Computational Linguistics .

Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2020. mT5: A massively multilingual pre-trained textto-text transformer. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies .

Joseph K. Yamagiwa. 1967. On dialect intelligibility in Japan. Anthropological Linguistics .

Shiyue Zhang, Vishrav Chaudhary, Naman Goyal, James Cross, Guillaume Wenzek, Mohit Bansal, and Francisco Guzman. 2022. How robust is neural machine translation to language imbalance in multilingual tokenizer training? In Proceedings of the 15th Biennial Conference of the Association for Machine Translation in the Americas ( Volume 1: Research Track ) .

Judit Ács. 2019. Exploring BERT ' s vocabulary. Accessed on April 11, 2023.

Slavomír Čéplö, Ján Bátora, Adam Benkato, Jiří Milička, Christophe Pereira, and Petr Zemánek. 2016. Mutual intelligibility of spoken Maltese, Libyan Arabic, and Tunisian Arabic functionally tested: A pilot study. Folia Linguistica .

A Background on Tokenization

To enable automatic processing of language, it must first be represented in a suitable form. The current practice is to use tokenization which is the process of turning natural language into sequences of tokens coming from a finite and pre-determined set called vocabulary ( Webster and Kit , 1992 ) . Each token is typically associated with an integer value. Language models process such sequences of integers, rather than sequences of characters or words. In this section, we offer a brief overview of the contemporary tokenization methods. For further details, we recommend the comprehensive survey by Mielke et al. ( 2021 ) .

Word tokenization. The simplest tokenization method is splitting at white spaces, where each word is assigned its own token ( Bengio et al. , 2000 ) . This approach, however, requires that all possible words are in the vocabulary which is not possible in practice. Therefore word tokenization often fails to handle cases like ' won ' t ', words spelled with accented characters like ' naïve ' or ' açaí ', speling mistakes and named entities like ' Cottonshopeburnfoot ' ( Sun et al. , 2020 ). This makes it unsuitable for representing open vocabularies , where the words encountered are not limited to a predetermined set. Furthermore, languages that do not use spaces to separate words, such as Chinese, Japanese and Burmese, pose additional challenges for this approach ( Shao et al. , 2018 ) .

Subword tokenization. Hence, most current models use subword tokenization , where complex words are broken down into multiple tokens. Subword tokenization can efficiently handle complex terms by breaking them down into parts, e.g. , ' Cottonshopeburnfoot ' → ' Cotton ' + ' shop ' + ' e ' + ' burn ' + ' foot '. This approach can represent novel words, including misspelled ones, in an open vocabulary setting.

Subword vocabularies are usually data-based approaches which use large corpora to learn which subword sequences occur frequently in practice. Schuster and Nakajima ( 2012 ) introduced one of the first subword tokenizers, WordPiece, as a way to handle Japanese and Korean. Sennrich et al. ( 2016 ) proposed using Byte-Pair Encoding ( BPE ) ( Gage , 1994 ) for learning subwords by merging the most frequently occurring pairs. BPE has since been widely used for most of the popular tokenizers. Kudo ( 2018 ) proposed an alternative approach via gradually pruning a large vocabulary. It removes tokens that are less likely to improve the performance of a simple unigram language model. Both methods rely on pretokenization ( splitting on whitespaces, when available ), which is not an invertible process. SentencePiece ( Kudo and Richardson , 2018 ) addresses this de-tokenization ambiguity by treating whitespace as a special symbol, including it in the vocabulary, and supports both methods. SentencePiece with BPE is by far the most popular tokenization method for the models considered in this paper.

Unicode support. Even if subword tokenization ensures that individual characters are in the vocabulary, this still leaves the question of which characters are to be included. Simple solution is to take the ASCII characters. However, this means that words in other scripts or accented letters will fall out of it. A common workaround is to represent strings outside the vocabulary as a special UNK token. However, if there are too many UNK tokens in an input, the performance of the model tends to deteriorate ( Pfeiffer et al. , 2021 ) . Therefore, it is desirable that the number of UNK tokens in the input is kept as low as possible. A simple and commonly used solution is to base the vocabulary building on Unicode.

Unicode is a computing industry standard for representing text characters ( The Unicode Consortium , 2022 ) . Unicode supports virtually all languages ( including many ancient ones, emojis and special characters ) by assigning every grapheme, modifier, punctuation mark, control character or formatting character one of 1,114,112 integer codepoints . The codepoints can be represented in binary as the variable-width encoding UTF-8, which encodes every codepoint with one to four bytes, or the fixed-width UTF-32 which encodes all codepoints with four bytes ( see Figure 4 ).

UTF-8 can therefore represent any string in any language as a string of bytes. As each byte can take only one out of 256 values, 256 tokens can be sufficient to encode all texts. In practice this is usually combined with the BPE tokenizer. At first, the corpus is en-

coded as UTF-8 bytes and then BPE is ran on top of it. As most characters occur frequently, BPE would assign them a dedicated token. If the model encounters a character that didn ' t exist in the training corpus ( e.g. , the medium skin tone waving hand ), it can still represent it byte-by-byte ( F0 + 9F + 91 + 8B for the waving hand and F0 + 9F + 8F + BD for the skin tone modifier ). This allows the vocabulary to efficiently represent frequently occurring words and rare characters. For example, the sentence ' I love açaí ' could be tokenized as ' I ' + ' love ' + ' a ' + C3 + A7 + ' a ' + C3 + AD .

Byte-level and character-level tokenization. If we can represent any input with just 256 characters, then why bother with subword tokens? A key consideration is sequence length. This is since transformers ( Vaswani et al. , 2017 ) , the currently predominant deep learning architecture for language models, have attention layers with a quadratic complexity in the input length. Hence, as the number of characters is much longer than the sub-word tokenization, working on the character level has been traditionally considered computationally inefficient. However, Chung et al. ( 2016 ) , Lee et al. ( 2017 ) , Gao et al. ( 2020 ) , Clark et al. ( 2022 ) and Xue et al. ( 2022 ) proposed various architectures working around this issue and operating directly on characters or UTF-8 bytes.

B Parity for Linguistic Varieties

A language can vary according to factors such as geography, history, social class and culture. As a result, different dialects, pidgin and creole language variations emerge, each with its own distinct set of grammar, vocabulary and pronunciation rules. 9 Unequal treatment of certain dialects or languages can lead to social and economic disadvantages for those who speak them. Therefore, it is important to also study the tokenization differences between the ' standard ' language and its varieties. 10 Unfortunately, parallel corpora for dialects, pidgin and creole language variations are far and few in between. In this section, however, we show results on regional Swiss German varieties, Arabic and Japanese dialects, as well as Haitian and Mauritian creoles.

Swiss German dialects. Swiss German is a dialect continuum which significantly differs from the formal High German. German-speaking Switzerland is diglossic: 11 High German is used alongside regional dialects ( Hogg et al. , 1984 ) . In contrast to other dialects, the use of Swiss dialects is increasing ( Sieber and Sitta , 1987 ) especially online ( Lüdi , 2007 ) . Swiss German dialects are often considered unintelligible to High German speakers and sometimes even speakers of different dialects may find difficulty understanding each other ( Russ , 1990 ) . Therefore, ensuring that German-targeting NLP applications can process Swiss German dialects is important.

To this end, we compare the tokenization parity relative to High German of GottBERT ( Scheible et al. , 2020 ) on the regional dialects of Aargau, Bern, Basel, Graubünden, Luzern,

St. Gallen, Wallis and Zürich. We use SwissDial, a parallel multidialectal corpus, as the basis of comparison ( Dogan-Schönberger et al. , 2021 ) . It is worth noting, that the dialect of each city and its corresponding region may differ significantly. Therefore there might be large variations within regions as well.

The results in Table 6 show a disparity between the tokenization lengths for High German and the Swiss dialects with a premium ranging from 1.38 for the Zürich dialect, or Züritüütsch , to 1.59 for the Bernese Bärndütsch . In fact, English has a lower premium than any Swiss dialect ( 1.35 on FLORES-200, Table 2 ) and the premium for Bernese German is close to the linguistically further Swedish ( 1.64 ) and Norwegian Bokmål ( 1.65 ). The following example from SwissDial shows how the sentence ' Like he ' s waiting for something ' has almost twice as long tokenization in Bernese German compared to High German:

963

15628

63

18

145

4

Als

warte

er

auf

etwas

.

1134 8808 226 751 2912 13621 288 361

67

11769 4

Aus

wür der

uf

ö

p

is

war

tä

.

The fact that the GottBERT tokenizer results in better parity for English, Swedish and Norwegian Bokmål than for Swiss German dialects highlights that it does not likely pick out stable linguistic constructs.

Arabic dialects. Similarly to Swiss German, Arabic is usually spoken in diglossic speech communities, where Modern Standard Arabic is spoken alongside at least one prestigious vernacular particular to the country or region ( Bassiouney , 2009 ) . As both Standard Arabic

and its dialects are commonly used in written communication, it is vital that tokenizers handle them equally well.

To assess the performance of Arabic tokenizers, we compare the tokenization lengths of ArabicBERT ( Safaya et al. , 2020 ) across 25 Arabic dialects. To this end, we use the MADAR parallel corpus of Arabic dialects ( Bouamor et al. , 2018 ) .

Table 7 shows the premiums relative to Standard Modern Arabic. The premium varies from 0.91 for the Jeddah dialect to 1.17 for the Rabat dialect. This is significantly lower than the premium for English ( 1.83 on FLORES-200 Table 2 ). The range is also much smaller than for the Swiss German dialects and approximately half of the considered dialects have a lower premium than Standard Modern Arabic. Therefore, one could say that the tokenizer of ArabicBERT achieves tokenization parity for these 25 Arabic vernaculars. This is likely because the corpus and vocabulary set on which ArabicBERT was trained contained dialectical Arabic. It is also possible that Arabic dialects are closer to Modern Standard Arabic and more mutually intelligible than Swiss German dialects are to High German ( Čéplö et al. , 2016 ; Trentman and Shiri , 2020 ) . Still, this difference between the parity for Swiss and Arabic dialects indicates that including a broader set of vernaculars and dialects in the corpus results in improved tokenization parity.

Japanese dialects. Japanese also has a number of regional dialects ( Hattori , 1973 ) . We compare the tokenization parity of BERT Japanese ( Tohoku NLP Group , 2019 ) across them. We employ the CPJD dataset by Takamichi and Saruwatari ( 2018 ) which contains transcriptions of the voice recordings of 250 sentences across 20 dialects.

The results in Table 8 show that the premium compared to Standard Japanese ( Tokyo dialect ) ranges from 1.01 ( for Saitama prefecture, neighbouring Tokyo ) to 1.15 ( for Morokataben and Okayama-ben ). These all are significantly lower than the premium for English ( 1.49, as shown in Table 2 ). Therefore, similarly to ArabicBERT, this is an example of the tokenizer being relatively well-aligned with the dialects. This is likely because Japanese dialects are more closely related ( and intelligible ( Yamagiwa , 1967 ) to Standard Japanese speakers ) than the Swiss dialects are to High German speakers.

Mauritian and Haitian Creoles. While creoles often have some similarities with a highresource language ( usually English or French ), the differences are significant to necessitate special attention to their support ( Lent et al. , 2021 , 2022 ) . This is especially critical for emergency services and disaster management ( Munro , 2010 ) .

Mauritian Creole is based on French as well as the languages of slaves imported from Madagascar and East Africa. As the British gained control of Mauritius, they brought indentured labourers from India who further had an effect on the formation of the modern Mauritian

Creole ( Seuren , 1995 ) . Similarly, Haitian Creole ( Kreyòl ) emerged from the interaction of French and the various Niger-Congo languages spoken by the Africans brought as slaves ( DeGraff , 2007 ) .

Considering that both languages have their basis in French, one would expect that tokenizers targeting French would have low tokenization parities for Mauritian and Haitian Creoles. However, taking the tokenizer of CamemBERT ( Martin et al. , 2020 ) , the premium for Mauritian Creole is 1.20 using the MorisienMT parallel corpus ( Dabre and Sukhoo , 2022 ) . The premium for Haitian Creole is 1.64 when using the QEDv2 corpus ( Tiedemann , 2012 ; Abdelali et al. , 2014 ) . Haitian Creole is also represented in the FLORES-200 dataset where the premium relative to French is 1.58. This is significantly larger than linguistically further languages such as English ( 1.20 ), Pangasinan ( 1.49 ) and Nigerian Fulfulde ( 1.54 ). Therefore, CamemBERT is not well-placed to tokenize French-related creoles despite the model being trained for French.

C Extended Tables of Tokenization Premiums

In addition to the models presented in the main text, these extended tables also include LLAMA ( Touvron et al. , 2023 ) , MBart50 ( Liu et al. , 2020 ; Tang et al. , 2020 ) , SeamlessM4T ( Barrault et al. , 2023 ) and Qwen-VL ( Bai et al. , 2023 ) .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the discrepancy in the tokenization length in different languages. It shows the unfairness of utilizing tokenization in different languages due to the cost, latency, and long-term language dependencies. The paper evaluates the unfairness between different tokenization strategies and model architectures. The paper's motivation is clear, and it provides an extensive analysis of multiple languages, showing the significance of the study and why it is essential in research and practical perspectives.\n\nSTRENGTHS:\n**Originality**\n- The paper introduces an important issue on utilizing LLM in different languages and investigates the aspects of cost, latency, and long-dependency. The problem formulation is important for multilingual researchers and practitioners in deciding what tokenization strategies are cost-effective.\n\n**Quality / Significance**\n- The paper shows a comprehensive analysis of how the different tokenization on different languages was done in different tokenization strategies (byte-based, Unicode-based, and subword-based) on the wide range of pre-trained LLM (on encoder-only, encoder-decoder, and decoder-only) and provides the quantitative evidence on the processing time of each language compared to English.\n\n**Clarity**\n- The paper is well-written, and the examples (in Japanese and Shan) give a high-level idea of why the problem introduced in the paper is important.\n\nWEAKNESSES:\n- Besides the aspects mentioned by the paper, it is unclear if the choice of the tokenization strategies would impact the performance of LLM in particular languages. \n- The suggestion of \"building a multilingually fair parallel corpus\" is vague. It would be great if the authors could give a more straightforward and practical solution.\n\nQUESTIONS:\n**Questions**\n\n- Does the tokenization lengths impact the performance on languages (e.g., Shan)?\n\n**Typo**\nLine 177: XML-R => XLM-R\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTokenization is a crucial, yet underappreciated component of language models. This paper presents a welcome investigation into the effects of tokenization choices across languages. The paper introduces the notion of premium for language A relative to B which measures the ratio of the average number of tokens for translations of the same sentence in the two languages. The authors show that the premium relative to English can be over 10 for certain low resource languages.\n\nThe paper then highlights the negative effects of having a high premium, which include: cost, latency, and the ability to fit less content in the fixed context window of an LM. Based on these findings, the authors make a case for developing multilingually fair tokenizers where the premiums are close to one across languages.\n\nSTRENGTHS:\n1. Tokenization is, in my view, an underappreciated problem and this work provides an insightful analysis on the effects of the tokenizer selection.\n2. The discrepancies presented in the paper across languages are surprisingly large. As such, this paper may stimulate new research in the field to address the discrepancies.\n3. The paper is very well written and a pleasure to read.\n\nWEAKNESSES:\n1. The paper doesn’t discuss how to achieve the goal of training multilingually fair tokenizers, apart from providing an overview of previous approaches to developing multilingual tokenizers. An obvious solution is to reserve more tokens for the languages with high premiums. However, this would mean that the number of tokens for other languages would need to be reduced to keep the vocabulary size fixed. This in turn will likely hurt the performance on high-resource languages such as English and it may even end up hurting the performance for the rest of the languages if there is less cross-lingual transfer from high-resource languages (NB: The paper states that “multilingual models struggle to deliver on the promises of deep transfer learning” but I’m not aware of works suggesting that there’s zero transfer happening across languages.)\n2. The paper lacks discussion on the potential negative effects of using a fair tokenizer (such as the one presented above) and discussion on the potential means of developing fair tokenizers. Yet, it argues strongly why the switch to fair tokenizers is necessary. This makes the paper seem a bit like a position paper (which doesn’t reduce the value of the paper in my view but makes me wonder a bit whether NeurIPS is an ideal venue for it).\n3. An obvious explanation for the tokenization length differences is the widely varying amounts of available training datasets across languages. However, another possible explanation is inherent differences between languages in terms of characteristics such as morphological richness of a language. Discussion of the different explanations seems to be missing from the paper.\n\nQUESTIONS:\nCould you elaborate on the following sentence “training a model from scratch with this fair tokenizer is necessary, despite potential suboptimal performance in individual languages due to vocabulary limitations”? What are the potential suboptimalities you’re referring to? Are you arguing for using a fair tokenizer regardless of any potential negative side effects?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper investigates the disparities between languages caused by tokenization policies used in large language models (LLMs). The authors compare the numbers of tokens needed to represent the translations of the same sentence in different languages and observe that the number of tokens needed in one language can be an order of magnitude larger than that in the target language (mostly English). They argue that these disparities have significant real-world implications such as the increased cost and performance degradations in using LLMs.

STRENGTHS:
- I think investigating the disparities of the cost and performance in using LLMs between different languages is an important topic.
- The approach taken by the authors (i.e., the use of the FLORES-200 parallel corpus) seems sound.
- The observations made in their experiments are informative.

WEAKNESSES:
- There is little technical novelty in the work.
- The authors point out the problems caused by the disparities but do not present a concrete solution. They argue that LLMs should be trained from scratch with a multilingually fair subword tokenizer but do not provide any experimental results towards that solution. I would be interested to see how much the performance and efficiency of an LLM in English needs to be sacrificed to achieve the parity.

QUESTIONS:
- Would it be difficult to conduct some (preliminary) experiments regarding the proposals discussed in Section 6?

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes the concept of tokenizer parity as a way to measure the fairness of tokenization across different languages in natural language processing. The authors argue that achieving tokenizer parity is necessary to improve the performance of multilingual models and address potential unfairness in the cost of accessing commercial language services, processing time and latency, and the amount of content that can be provided as context to the models. The paper suggests that training language models from scratch with a multilingually fair subword tokenizer is the only approach that can effectively address tokenization unfairness and achieve tokenizer parity. The authors provide several examples to demonstrate the effectiveness of the proposed method for measuring tokenizer parity and suggest that a variation of subword tokenization is necessary to achieve parity. The paper contributes to the field of natural language processing by proposing a new metric for measuring fairness in tokenization and providing guidance on how to achieve tokenizer parity in multilingual models.\n\nSTRENGTHS:\nIn terms of originality, the paper introduces the concept of \"tokenizer parity\" as a systematic way to assess the fairness of tokenization across different languages. This is a novel idea that has not been explored extensively in the literature.\n\nIn terms of quality, the paper provides a detailed analysis of the tokenization lengths of different languages using various tokenizers. The authors also propose a method to measure tokenizer parity and demonstrate its effectiveness using several examples.\n\nIn terms of clarity, the paper is well-written and easy to follow. The authors provide clear definitions of the key concepts and use examples to illustrate their points.\n\nIn terms of significance, the paper addresses an important issue in natural language processing, namely the fairness of tokenization across different languages. The concept of tokenizer parity has the potential to improve the performance of multilingual models and make natural language processing more equitable across different languages. Therefore, the paper could have a positive impact on the development of multilingual models.\n\nWEAKNESSES:\nOne major weakness is that the paper does not provide a clear roadmap for how the concept of tokenizer parity could be integrated into existing natural language processing pipelines. While the authors suggest that training language models with a multilingually fair subword tokenizer is the only approach that can effectively address tokenization unfairness, they do not provide guidance on how this could be achieved in practice. More detailed recommendations for how to integrate tokenizer parity into existing natural language processing pipelines would be helpful for researchers and practitioners in the field.\n\nAnother potential weakness is the lack of discussion on the impact of tokenizer disparity on downstream task accuracies. This aspect is particularly interesting and may attract more attention.\n\nQUESTIONS:\n1. Can you provide more detailed recommendations for how to integrate tokenizer parity into existing natural language processing pipelines?\n2. What is the potential impact of tokenizer disparity on downstream task accuracies?\n\nLIMITATIONS:\nThe lack of clear guidance on how to integrate tokenizer parity into existing natural language processing pipelines may limit the impact of the paper's findings. Additionally, the absence of discussion on the impact of tokenizer disparity on downstream task accuracies is a limitation that could be addressed in future research. Without this information, it may be difficult for researchers and practitioners to fully understand the potential benefits of using a multilingually fair subword tokenizer.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper is very interesting in highlighting the disparities in tokenization across different languages, leading to cost, latency, and long-distance modeling inequalities in LLMs. The authors show that this is not limited to one type of tokenizer or a single family of LLMs. They show that different types of tokenizers for LLMs in practice, including subword, multilingual, or byte-level tokenization, all show disparities for languages in FLORES-200.\n\nBased on these results, the paper convincingly argues that we need multilingually fair tokenizers for future LLMs, so that some languages are not in disadvantages in terms of the cost to access LLMs, the latency of the service, and the amount of data that can be processed.\n\nSTRENGTHS:\nThe paper is well written and the first one (to my knowledge) to study tokenization disparities in LLMs at this scale. \n\nThe presented results will be very interesting to LLM researchers and practitioners in understanding the disparities in tokenization across different languages in different LLMs. This has major consequences for some languages, putting them at a disadvantage in terms of the cost to access LLMs, the latency of the service, and the amount of data that can be processed.\n\nWEAKNESSES:\nSection 6 argues that future tokenizers should be based on subword tokenization and support all Unicode codepoints. However, one of the main weaknesses of the paper is that it does not develop such multilingual fairer tokenizers, and compares them with existing tokenizers in practice. Attempting to do that would have emphasized challenges in developing such tokenizers and how one can overcome them.\n\nI found the argument around disparities in long-distance modeling a bit thin. The second paragraph of Section 5.3 discusses this briefly, but additional discussion or experiments are needed to strengthen the argument. For example, it might be helpful to include analysis with multilingual document summarization (e.g., XLSum).\n\nQUESTIONS:\nI think the paper does a very good job of explaining why there are language disparities among different tokenizers. However, I feel these reasons are known already, what are the challenges in overcoming them?\n\nLIMITATIONS:\nThe paper presents a large-scale study of disparities in tokenization across different languages. It discusses how LLMs are putting some languages to their disadvantage due to this. The paper itself does not propose a new tokenization scheme that needs to address its limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-28"}