Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models
Authors: Orevaoghene Ahia
Sachin Kumar
Hila Gonen
Jungo Kasai
David R Mortensen
Noah A. Smith
Yulia Tsvetkov
Venue: EMNLP 2023 Main (2023)
Keywords: Tokenization
LLMs
under-resourced languages
equitable LLMs
TL;DR: We investigate the effects of subword tokenization in LLMs across languages with different writing systems.

Abstract: Language models have graduated from being research prototypes to commercialized products offered as web APIs, and recent works have highlighted the multilingual capabilities of these products. The API vendors charge their users based on usage, more specifically on the number of ``tokens'' processed or generated by the underlying language models. What constitutes a token, however, is training data and model dependent with a large variance in the number of tokens required to convey the same information in different languages. In this work, we analyze the effect of this non-uniformity on the fairness of an API's pricing policy across languages. We conduct a systematic analysis of the cost and utility of OpenAI's language model API on multilingual benchmarks in 22 typologically diverse languages. We show evidence that speakers of a large number of the supported languages are overcharged while obtaining poorer results.  These speakers tend to also come from regions where the APIs are less affordable, to begin with. Through these analyses, we aim to increase transparency around language model APIs' pricing policies and encourage the vendors to make them more equitable.

Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models
Orevaoghene Ahia
Sachin Kumar
Hila Gonen Jungo Kasai
David R. Mortensen Noah A. Smith "0
Yulia Tsvetkor
Paul G. Allen School of Computer Science & Engineering, University of Washington
Language Technologies Institute, Carnegie Mellon University
"Allen Institute for Artificial Intelligence
Saahia,jkasaj nasmi th, yulllllllsseeee jēcs. washing tighiington...m edu, sachinkeallenai.oom hi hilagnn@gmail i. com, dmortensêcs cmu. edu
Abstract
What is the capital Morocco?
Noia tivor mpwrc Maрóиоu;?
Language models have graduated from being research prototypes to commercialized prod- uets offered web APIs, and recent works have highlighted the multilingual capabilities of these products. The API vendors charge their users based usage, more specifically on the number of "tokens" processed or gener- ated by the underlying language models. What constitutes a token, however, is training data and model dependent with large variance in the number of tokens required convey the same information in different languages. In this work, we analyze the effect this non- uniformity the fairness of API's pricing policy across languages, We conduct a system- atic analysis of cost and utility of OpenAl's language model API multilingual bench- marks in 22 typologically diverse languages. We show evidence that speakers of large num- ber of the supported languages are overcharged while obtaining poorer results. These speak- tend to also come from regions where the APIs are less affordable to begin with. Through these analyses, we aim to increase transparency around language model APIs' pricing policies and encourage the vendors to make them more equitable.
OOL
vol
tapittll of
3 1
Moxocoo?
b$
capte of Wiroces
Hesomittrt Manioo Can zden TAURH Pountoi Zark.
We investigate the effects subword tok- enization in LMs across languages with different writ- ing systems. Our findings highlight disparities in the utility LMs, as well as socio-cconomic disparities and increased costs in using commercial APIs for speakers of underrepresented languages."
Figure 1:
from openly releasing them (Abdalla et al., 2023). Instead, a new business model of LM as a Ser- vice (Sun et al., 2022) has emerged where LMs can be accessed for inference using (paid) web APIs. The majority of these models (Ouyang et al., 2022) offer multilingual capabilities, and the API providers charge the users proportionally the number of tokens processed or generated.
1 Introduction
this work, we examine the fairness this pricing model for different languages, based on how a "token" is defined in practice. Most LMs rely on tokenizers that split text strings into chunks (subwords). Subword tokenizers (Sennrich et al., 2016; Kudo, 2018; Song et al., 2020) are typically data-driven and learn to split text based fre- quency patterns of characters or bytes in some corpus. Prior work argued that, in multilingual settings, subword tokenizers lead to disproportion-
Language models (LMs) have come to be known as general-purpose solutions capable of perform- ing many tasks by following natural language instructions (Brown et al., 2020; Ouyang et al., 2022; Chung et al., 2022), and generalizing to new tasks at test time using handful of demonstra- tions (Brown et al., 2020; Su et al., 2023). Moti- vated by their potential for commercial use, many industrial research institutions have moved away
OpenAA's 's tokenizer interface displays byte tnkens absent from their vocabulary as "?".
"Code available at https: htas:///iihhnn com/ com/orevaahia/1 1m, tokenizer_cost

---

ale fragmentation rates for different languages and writing scripts (Zhang et al., 2022a; Rust et al., 2021; Muller et al., 2021). Many commercial LMs are multilingual, and text from languages that suffer from excessive fragmentation will be represented using more tokens. This directly increases cost of API usage for certain language speakers, even if they convey the same information as the others.
'hee total token count is also immediately obvi- ous users except through a tokenizer interface" separate from the chat interface.
Tokenization in LMs Tokenization segmenting text into atomic units-is an active rescarch area. Proposed approaches range from defining tokens as whitespace-delimited words (for languages that use whitespace) which makes the vocabulary extremely large, to defining tokens as characters or bytes, making the tokenized sequences extremely long in terms of number of tokens; see Mielke et al. (2021) for detailed survey. A commonly-used solution now is tokenize text into subword chunks. With Sennrich et al. (2016), one starts with a base vocabulary of only characters adding new vocabulary items by recursively merging existing ones based on their frequency statistics in the data. Other approaches judge subword candidates to be included in the vocabulary using an LM (Kudo, 2018; Song et al., 2021). For multilingual models containing data in variety of scripts, only characters (based on Unicode symbols) can be very large with over 130K types. Radford et al. (2019) instead proposed using a byte-level base vocabulary with only 256 tokens. Termed byte- level byte pair encoding (BBPE), this approach has become de facto standard used in most modern language modeling efforts (Brown et al., 2020; Muennighoff et al., 2022; Scao al., 2022; Black al., 2022: Rae et al., 2022: Zhang et al., 2022b). In this work, we investigate the impact this tokenization strategy has on LM API cost disparity as well as downstream task performance (i.e., utility) across different languages.
We highlight this unfairness through three stages of systematic analyses. First, we show evi- dence that tokenizers of popular LMs indeed over- fragment texts in certain language scripts and quan- tify the API cost disparity that this issuc causes, We discover that the disparity is not caused just by data imbalance, but is rooted in the language prop- erties the ways they represented in Unicode. Second, show that languages with longer token lengths as a result of greater fragmentation derive less model utility with in-context learning (Brown et al., 2020). Finally, we find that languages that cost more and perform worse are often associated with populations of speakers for whom the APIs are less affordable on average, exacerbating the economic divide in the accessibility of NLP tech- nology.
the base vocabulary of
Through these analyses, we argue that commer- cial LM API vendors should revisit their processing and pricing strategies to be more equitable. In ad- dition, we encourage the NLP community to pay better attention to tokenizers, an often neglected part of the LM pipeline.
2 Do All Languages Cost the Same?
2.1 Background
Language Model APIs Autoregressive LMs are trained to predict the next "token" given previ- context. Following the success of such models, many commercial LM web APIs have emerged and allow users to interface with the models using natural language instructions to perform various tasks with little to no exposure to the underlying workings of the models. The API providers of- ten support dozens of languages and charge users' at fixed rate based on the total number of input and generated tokens.* What constitutes "token," however, not a universally accepted definition but design choice that the model developers make.
2.2 Investigating the Impact of Byte-level Subword Segmentation
There are hundreds of distinct writing systems in the world (Hockett, 1997). BBPE, by design, makes vocabulary construction script-agnostic, al- lowing (in principle) new scripts to be supported later on without modifying the vocabulary. How- ever, not only different scripts encoded dif- ferently, their distribution in the training corpora varies widely. To investigate the effects of this vari- ation, we propose the following research questions as the main focus of this work.
'Wiile most services also have free tiers, they limit daily usage small number of tokens. FF.g. see Ap models" cost: ricing.
hhttps://platform..n wwwpp//www.....mo//o.i.iiiaaeee.

---

RQI (number of tokens): do all languages con- vey the same information with the same num- ber of tokens? We analyze the fragmentation of sequences in different languages with different tokenizers. We find that among the supported lan- guages in popular LMs, there is a large variance in the average number of tokens required to convey the same information with some languages requir- ing 5 times many tokens than others. Previous work has shown that tokenization in multilingual models is usually biased towards high-resourced languages in the pretraining data (Acs, 2019; Rust et al., 2021); ;; we observe that this not always the case, but it could also be dependent on linguistic features or properties of language scripts.
3 Experimental Setup
3.1 Models
Throughout this work, we focus on two LMs: Chat- GPT (Ouyang et al., 2022; Brown et al., 2020) (gpt-3.5-turbb) and BLOOMZ (Muennighoff et al., 2022). Both of these models are trained and ad- vertised as general-purposs models capable of fol- lowing instructions and performing wide range of tasks (Qin et al., 2023; Zhu et al., 2023; Ahuja et al., 2023; Huang et al., 2023)-
ChatGPT (Ouyang et al., 2022) is closed model only accessible through an API (with a premium tier) provided by OpenAl. Studies report that it supports many 90 languages (Ahuja et al., 2023). ChatGPT can handle a maximum sequence length of 4096 tokens (including both the prompt and generated tokens).
RQ2 (cost): do non-uniform tokenization rates lead to LM API cost disparity for speakers of dif- ferent languages? LM APIs like ChatGPT are available worldwide and have been widely claimed to have multilingual capabilities (Kasai et al., 2023; Lai al., 2023). 6 We show that disparate fragmen- tation rates across languages lead significantly high usage costs for less represented languages, and we argue for a more equitable API pricing system.
BLOOMZ (Muennighoff et al., 2022) is an open- source multilingual model trained on 46 natural languages and 13 programming languages. While training its tokenizer, sentences from different lan- guages were sampled according to multinomial distribution (Conneau al., 2020), thereby in- creasing the number of tokens associated with low- resource languages. The best best- perforrmf version of this model has 175B parameters and is not fea- sible to be loaded on our academic servers; hence we rely on free API of BLOOMZ hosted by Hug- gingface." Although BLOOMZ was trained with ALiBi positional embeddings (Press et al., 2022) which allows the model to extrapolate any length sequences during inference, the Huggingface API has context limit of 1000 tokens.
RQ3 (model utilityy:: do non-uniform tokeniza- tion rates affect the models' utility? LMs have exhibited in-context learning capabilities, perform- ing new tasks with few demonstrations as input (without parameter finetuning). This is highly de- sirable in any LM API as avoids computational, annotation (and financial) costs. We show that high fragmentation rate of a language negatively affects the in-context learning performance in that language, resulting in reduced model utility.
3.2 Tasks and Datasets
To answer RQ1 whether the same information is conveyed with similar numbers of tokens in differ- ent languages- use a validation set of FLORES- 200 (Goyal et al., 2022), a multilingual parallel cor- pus containing examples in over 200 languages. We tokenize each sentence in the FLORES-200 subset with ChatGPT's tokenizer' 9 and compute the average number of tokens per sentence for each language. Using parallel data controls for the same information across languages. We consider that
RQ4 (socio-economic aspects): what are the socio-economic implications of the API's cross- lingual cost and performance disparity? Our analysis shows evidence that not only are LMs more expensive for certain languages, they also less effective for them. To highlight the implica- tions of these findings, we correlate those measure- ments with the socio-cconomic indicators of lan- guage speakers as a proxy for affordability of the APIs. This analysis indicates that users who likely cannot afford high API costs are charged more for poorer service, hindering uniform accessibility.
'nttps://hugginggace htt./aaaccccccoooooooooooooooooe ui uicktour.
co/ co/docs/api http://appiiiffeeceeeaaaa
"We also experimented with WMT 2021 data (Akhbardeh al., 2021) and found similar results. Note that the WMT data focused European languages.
"https://help. cat.apppppeaii.ccomm 9 h-hhowwdor do- i-use--eiiee ceee--peeaaaaaee aap--appiiiiii mepi-inndiffeen diffeinfeettt-tee lan guages
"ChaiGPT's tokenizer https: //tbthub... com/openati /ti
ktaken

---

language A is more efficiently tokenized than lan- guage if it uses fewer tokens per sentence on average. While previous studies have computed fragmentation rates with fertility (Acs, 2019), we
300
Average number of tokens
200
instead define it as the average number of tokens in sequence for two reasons. First, our goal is compare LLM API costs across languages that charge users based the number of tokens. To control for content, this analysis. Second, many languages we analyze are understudied and do not have word tokenizers available which are required to compute fertility.
100
122
Labin
Japanese
Hangul
Cyrillic
Arabic
Hebrew
Devanagan
Greek
Bengal
Tamil
Teluga
Georgian
Tibetan
parallel corpus for
Language Script
Figure 2: Average number of tokens by script after tokenizing the Flores dataset. The fragmentation rate is lower for Latin script languages and higher for other scripts. Number of languages per language group is indicated the top of each bar.
For RQ2 and RQ3, to clearly highlight the cost and utilityy disparities, NLP tasks that involve long-form texts either input or output. We evaluate the models on di- verse, challenging natural language generation and classification tasks on the following benchmarks:
evaluate the models
For all tasks, we provide the instructions in En- glish following Ahuja al. (2023), who show that on several multilingual benchmarks, English in- structions outperform the in-language prompts (see Table in the Appendix for the prompting format for all tasks). For each task, we randomly sample at most 500 test examples for evaluation.
Classification We evaluate on (1) XNLI (Con- al., 2018): tross-lingus inference benchmark comprising of 11 typologically diverse languages. It involves two sub-tasks, passage selection and minimum answer span (Gold-P). We focus on the latter task in our experiments. (2) XFACT (Gupta and Srikumar, 2021): a multi- lingual fact verification dataset of naturally existing real-world claims covering 25 languages.
4 Results and Analysis
4.1
RQ1 (number of tokens): do all languages convey the same information with the same number of tokens?
Span Prediction We use XQUAD (Artetxe et al., 2019): a crosslingual question-answenee dataset where each example consists of a paragraph, ques- tion, and the answer as a span in the paragraph.
In Figure 2 show that Latin-script languages are represented with substantially fewer tokens com- pared to languages in other scripts. While Cyrillic and Japanese script languages come close the Latin, languages with their own script (c.g., Telugu) require up to 5x more tokens to convey the same information. We hypothesize that this disparity is due to training data imbalance since ChatGPT's tokenizer was primarily trained on Latin-script lan- guages, mainly English. The training details of make
Generation We evaluate on (1) Cross Sum (Hasan al., 2021a): a cross-lingual abstractive summarization dataset comprising 1.7 million article-summary samples in 1500+ language pairs, and, (2) XLSUM (Hasan et al., 2021b); a summarization dataset covering 44 diverse languages. It comprises news articles and summaries in the same language the article.
ChatGPT not available. However, reasonable assumption that its training data has a similar proportion of languages as the publicly available large corpus CC100 (Wenzek al., 2020). If we sort languages shown in Figure 2 based on their data size in CC100 (see Figure 14 in the Ap- pendix), low-resourced languages of Latin script appear to be less fragmented compared to other mid-resourced languages of non-Latin scripts. Figure 15 in the Appendix, we present a sim- ilar analysis for BLOOMZ's tokenizer. We sort
3.3 Prompting Formulation
We evaluate both models in a k-shot in-context learning setup where we also provide task instruc- tions. We experiment with 0 s k = X, where X is the maximum number of in-context examples that can be provided. Note that X not a fixed value, but is determined by the LM API's limit on the number input tokens and the fragmentation rate of the language.

---

the languages based on their size in the pretraining data (ROOTS corpus; Laurençon al., 2023). We observe that languages with fewer resources gen- erally have higher average token length. Arabic is an outlier here as it appears to have more tokens than some other mid-resourced languages.
4.2 RQ2 (cost): how do non-uniform tokenization rates affect LM API costs for different languages?
LM APIs charge users a fixed amount for a given number of input and generated tokens. Since the same information is expressed using different num- ber of tokens in different languages, we aim to investigate the disparity in what users pay to use the API for different languages. From the results of our analysis in $4.1, we compute the estimated cost of API use per language as a function of the average sequence length derived in Figure 2. We report this on subset of languages in Figure 16 in the Appendix and present a granular analysis of languages that share family and script in Figure 4.
What influences the non-uniformity of a to- kenizer across languages? From our analysis above, we identify two influential factors: (1) the proportion of the language in the pretraining data, and (2) inherent properties of the language and its writing script. While we see some correlation be- tween pretraining data size and fragmentation rate in BLOOMZ, with ChatGPT it is quite different as higher-resourced non-Latin script languages still get excessively tokenized.
Languages that are more heavily segmented have predictably higher costs of usage. Overall, we see that the API costs are biased towards (i.e., cheaper for) Indo-European and Latin script lan- guages and against many non-Latin script lan- guages. In most mid-resourced Indic languages with non-Latin scripts, we see close to 5x in- crease in cost compared to English.
To disentangle the effects of factors (1) and (2) we train BBPE tokenizers on a variety of languages with diverse scripts with vocabulary sizes ranging from 5,000 to 50,000, while controlling for content and data size. Specifically, we train the tokenizers on parallel corpora and include one language per script. We then use these tokenizers to tokenize the text they were trained on, and compute the average number of tokens per sentence.
Cost relative English
Myamma
Latin
Kimer
Cyrlic
bug qumber of au
Tisttan
Japarese
Hangul
Aratic
(IE, Lasin)
(ST, Laii)
(AA, Lasin)
(AC, Latin)
(AA, Arabic)
(IE, Gyrilii
(AA, Hebrew)
IE Devonagan) (EE Arabic
(E Greek)
(IE, Helrew)
(E, Bengali
(ST. Bengal)
(DR Temi)
[DR, Telugo)
(KA, Georgan)
(ST. libetan)
100
Language Family and Script
250
200
1000
2000
10000
2500
10:00
Figure 4: Estimated cost per language family/script, rel- ative to English. The language families are abbreviated as follows: IE: Indo-European, ST: Sino-Tibetan AC: Atlantic-Congo, AA: Afro-Asiatic, DR: Dravidian, KA: Kartvelian.
Figure 3: BBPE tokenizer trained parallel text from different language scripts with varying vocabulary sizes. We display a larger version with 21 more scripts in Figure 19 in the Appendix.
Next, we report the costs of running experiments relative to English. We report costs based on our zero-shot experiments across all tasks listed in $3.2. This due excessive tokenization in some lan- guages for which we can only do zero-shot evalua- tions. For XLSUM, we show in Figure 5 that we spend up to 4x more for both prompting and gen- eration in Telugu and Amharic. We observe similar findings in XFACT and CROSSUM, as displayed in Figure 11 in the Appendix.
As shown in Figure 3, even when controlling for the content, there still a disparity in the tokeniza- tion rate at different vocabulary sizes. In particular, most scripts are very sensitive to small vocabulary sizes compared to Latin and Hangul scripts. We do not achieve uniform fragmentation rate across all language scripts even with large vocabulary sizes. We therefore conclude that uniformity of BBPE tokenizers across languages is not just determined by the proportion of text from language in the pre- training data but also by language/script properties.
While the majority of the commercial LMs are perhaps being optimized to perform well in many

---

Easternn Cot rdlgee to Exgio
verse languages as described in $3.2. Figure 7 shows ChatGPT's performance according to stan- dard automatic metrics of all tasks. Note that the focus of this experiment is to illustrate the impact of tokenization in in-context learning settings. There- fore, we are interested not in the absolute value of the metrics comparisons among languages but the relative improvement within the test sets of the same language we increase the number of in- context examples. For all tasks and most languages, we see consistent performance improvements as we increase the number of in-context examples, from zero-shot to & (even for k: 1). For many lan- guages such as Telugu and Thai, due to their high fragmentation rates, we were unable to fit even one complete demonstration and hence, only report zero-shot results. Based trends from other lan- guages, we suspect that these languages could also have benefitted from more demonstrations, Hence, as a result of unfair tokenization, ChatGPT's utility is much lower for speakers of those languages com- pared to better represented languages like English.
12
611
21
WW
60
70
1h
Language
Figure 5: Average cost of prompt + generated tokens for XLSUM evaluations relative to English.
languages, we show that there is less focus on indi- vidual experiences of speakers of languages other than English. While LMs like ChatGPT might per- form tasks in Telugu, for example, user in Andhra Pradesh might pay 5x more than English user in the US for equivalent use of the model.
RQ3 - Model utility: do non-uniform 4.3
tokenization rates affect the models' utility?
56 teat dara the doan's it the contat: logih
100%
Figure reports the results of the same experi- ment for BLOOMZ. Across all tasks we find that adding in-context examples does not help. In fact, in some cases, there is performance drop even with one in-context example. Upon manual inspec- tion of the generated outputs from the one-shot experiments, the model has a tendency to copy spans from the in-context example, presenting that as output and thus not successfully utilize demon- strations. Our hypothesis here is that BLOOMZ is better optimized for zero-shot prompting and is not as suitable for in-context learning.
80%
60%
40%
Argle
20%
Tokga
Suaril
Findl
0%
Spanst
Patagaus
G
8
10
In-context examples
Figure 6: Percentage of test examples per language in XLSUM that do not successfully fit into the context length of ChatGPT. We can fit more few-shot examples in Latin script languages than in other languages.
Due to the limited number of tokens that BLOOMZ's inference API accepts, some examples in some languages cannot fit the 1000 token con- text length when doing zero-shot prompting. We experienced this with the XLSUM dataset we couldn't fully fit news articles for some languages. Understandably. some of these languages are not even present its pretraining data, and hence we do not expect them to be tokenized efficiently. For these examples that do not fit the context length, we feed in truncated news articles into the model. We therefore evaluate the generations for the frac- tion of examples that fit context and ones that do not fit the context separately. Figure 9 shows the performance comparison when we use truncated summaries in the prompt and when we use the full articles. While the performance drop is expected,
LMs typically have an upper bound of the num- ber of tokens they can handle, e.g., ChatGPT can process maximum of 4,096 tokens. Hence, due to non-uniform fragmentation rates across languages, there is a disparity in the amount of information the models can process per language. In Figure we plot the percentage of XLSUM test instances against the maximum number of in-context exam- ples those instances can be accompanied with. For example, Telugu struggles to fit even one in-context example for the majority of the test set. Hence, the model can only do zero-shot prompting in this case. To measure the impact of this issue on task per- formance, we evaluate ChatGPT and BLOOMZ with a k-shot learning setup on the tasks on di-

---

30
RougeL
Fi-Score
Rouge1
20
20-
10
Number in-context examples
Number in-context examples
Number in-context examples
(a) XLSUM
(b) XFACT
(C) CROSS-SUM
Figure 7: Results from ChatGPT few-shot evaluations, In most tasks, increase in performance increase the number of in contett examples.
Score
Rouge- L
Exact Match
Number of in-context examples
Number in-context examples
Number in-context examples
(a) XNLI
(b) XLSUM
(c) XQUAD
Figure 8: Results from BLOOMr few-shot evaluations. The BLOOMz model is clearly better at zero-shot prompting than feotshot.
40
most cases, the lower the HDI index, the higher the fragmentation rate and vice versa. Evidently, the model's vocabulary is biased towards users of more developed countries.
truncated article
full article
30
Rouge-L
20
10
Cos-HDI
HDI-Uhiky
Cost-itilit
Tisk
t3 hi 2F pt Ja.
ko te sw es enam Vi th mr ky uk ur Languages
Spearman
Speamnn
Peanon
Spearman
Peussn
XFACT SUSUM
#o-0.41
69.0.00
+0.31
+40.3
64.035
*4-042
44.443
.-444
403
40.1
44.0.45
40.24
4.0.77
Figure Zero-shot evaluation of BLOOMz on XL- SUM. Since we cannot fit the full article in the context length for some languages, we compare results on eval- uating full articles truncated articles.
Table 1:
Correlation between model utility, cost of API access and Human Development Index (HDI) for each task. We mark correlations with < 0.05 with and also mark correlations with p < 0.0056 (according Bonferroni correction for multiple hypotheses) with **.
our focus here to highlight a consequence of differentiated tokenization in LMs.
This bias further validated by results shown Table 1, where we mostly find negative correlations between pairs of each of the following variables: average financial cost of experiments, model utility (performance), and human development index of the country in which each language spoken. We term this "double unfairness" people from less economically developed countries are overcharged at fixed rate per-token due to excessive tokeniza- tion, but often derive less utility from the model.
4.4 RQ4 Socio-economic aspects: what are the socio-economic implications the API's cross-lingual cost and performance disparity?
In Figure 10, plot the fragmentation rate per language against the Human Development Index in the country with the highest absolute number of speakers that language. We find a strong negative correlation close to -0.5 showing that in

---

9.0
performs in the few-shot setting, in most cases. However, first, most open-source models are dis- tributed under an academic license whereas most developers are interested in integrating these tech- nologies into their products for commercial use, which may incur licensing costs, Second, barring licensing issues, LMs tend to be large and resource- intensive to train and deploy and require dedicated expensive hardware run commercial scale, which again might not be possible for most devel- opers and users, even exceeding the cost of using the APIs. Research on reducing such hardware re- quirements (Dettmers et al., 2022; Park et al., 2023) could increase accessibility, Still, this requires a considerable level of technical expertise from de- velopers and users which might be infeasible.
Abetagg number of tokets
055
Avem Misngian
Kienftrrada
So Salaalylla
200
Tunkin
1.50
Hegts
Vidhar
Greck
Unda
000
Helro
Sand
Q.V
Human Development Index
Figure 10: Fragmentation rate per language against the Human Development Index in the top country where the language is spoken.
5 What is the Way Forward?
Transparency in API limitations While the NLP community is aware of many of the issues we point out in this work, LM APIs are advertised to a general audience. Much like the policy of adding limitations to research papers, LM API providers ought to be more transparent about the flaws and biases in their models, especially when describing their multilingual capabilities. Many users not privy the inner workings of these models and will be unknowingly charged higher prices if they use the model their native languages.
Technological improvements in LMs Several solutions proposed in recent work to improve lan- guage modeling performance can help alleviate the cost and utility issues we highlight. Tokeniza- tion is an active area of research and various so- lutions based on data balancing (Johnson et al., 2017; Conneau and Lample, 2019), optimal trans- port (Xu et al., 2021), fuzzy subwords (Provilkov et al., 2020), and many more (Chung al., 2020; Tay et al., 2022) have been proposed. BLOOMZ. for instance, relies on data balancing to improve fragmentation rates across languages. Some works also focused on increasing the context lengths of language models (Bulatov et al., 2023; Press et al., 2022) which can help alleviate issues with utility by allowing more in-context examples input.
Rethinking the API pricing models Higher API costs for languages underprivileged communi- ties risks excluding many populations from using language technologies. A potential solution is to de- velop pricing policies based on languages/regions while also accounting for model performance on language-specifie benchmarks. An alternative is not charge by tokens at all. PaLM 2 API, for example, charges the users based on characters. 10 Hence, further analysis is needed assess the fair- ness of character-based pricing. Huggingface also offers an inference service for their enterprise cus- tomers" relying on AWS instances and charging them at an hourly rate. Future work may compare this with a per-token rate we study in this work.
Related Work
Analyzing tokenization methods The impact of tokenization on model performance (Ács, 2019; Rust et al., 2021; Zhang et al., 2022a; Klein and Tsarfaty, 2020; Bostrom and Durrett, 2020; Ka- mali et al., 2022), inference speed and memory usage of LMs in practical settings (Sun al., 2023; Hofmann et al., 2022) has been widely stud- ied. Ács (2019) observes that mBERT's vocabu- lary is largely dominated by Indo-European lan- guages. Rust et al. (2021) find that monolingual LMs perform better than mBERT because some languages suffer from over- framefrrgmentttionn Zhang et al. (2022a) find that sentence-level MT models are not sensitive to language imbalance in their to- kenizer training data. contrast to prior work, our focus is on the cost and performance analysis multilingual LM APIs across languages with regard
Open-souree models vs. paid APIs Given issues with paid APIs, the natural next question might be: should the API users move to open- mod- els or train their own? In fact, in experiments, we find BLOOMZ, an open-source model, per- form better in the zero-shot setting than ChatGPT
10 Prior work has shown evidence that the number of characters used to express the same information in different languages

---

over-fragmentation and in-context learning.
speak languages with scripts less represented in the pretraining data. With the recent widespread use of commercial LMs, we believe that our work is crucial to ensuring that language technologies are accessible to diverse users irrespective of the languages they speak.
Socio-economic impacts of language models Prior work show that unfairness in LMs is a consequence of many stages the development pipeline (Cao and Daumé III, 2020; Talat et al., 2021). Efforts have tried to identify social biases in LM generations (Wolfe and Caliskan, 2021; Dev et al., 2022; Sheng et al., 2021; Chen et al., 2021; Hutchinson et al., 2020). Other works have sur- faced the cultural and language disparity beyond and within multilingual LMs (Gururangan al., 2022; Kreutzer et al., 2022; Virtanen al., 2019). Talat et al. (2022) discuss challenges impacting bias evaluation in multilingual LMs. They exam- ine power dynamics and consequences of training LMs emphasizing implications associated with ad- vancement of such technologies. In this work, study economic unfairness of LMs across differ- ent communities. Concurrent work (Petrov al.. 2023) analyses multilingual tokenizers focusing on financial cost, latency and context size. However, apart from cost, our analysis also covers model utility and socio-economic implications. Kasai et al. (2023) report unfair API costs as a result of tokenization differences between English and Japanese. We extend this to 21 more languages highlighting the pervasiveness of this issue.
There are different factors that contribute to non- uniform tokenization across languages. Whilst our analysis touches on the size of pretraining data and language writing systems we suspect that there might be other factors not yet uncovered; we leave that for future work. The lack of access to Ope- nAl's training data prevents us from making solid claims about all the languages that ChatGPT is optimized for; however, their models have been advertised and shown to work well in many lan- guages. More work large multilingual models should include the release of (details of) training data to further enable this kind of research.
Limitations
Translationese
We conduct the analysis to an- swer RQ1 using a parallel corpus, FLORES- 200 (Team et al., 2022), in order to control for the same information. This corpus consists of many examples that have been professionally translated. Prior studies have shown that translated texts in any language (referred to translationese) may differ from original written text in many ways (Laviosa, 2002). These may have caused the information conveyed different languages to not be exactly the same. We do not have a way to measure these differences. However, we expect them not to be so large as to meaningfully affect the trend of frag- mentation rates.
7 Conclusion
By analyzing popular language model APIs on challenging multilingual benchmarks, we find that (a) API tokenizers disproportionately favor Latin scripted languages and wver-fragmen less repre- sented languages and scripts, (b) the API pricing policy of charging based on the number tokens is flawed and extremely unfair towards speakers of the over-fragmented languages, and (c) the API performs poorly on such languages compared to the less-fragmented counterparts. In the current
Language statistics of ChatGPT training data ChatGPT is a closed model developed by OpenAl who have not released the training details of the model including any information of the languages it supports." 12 Hence, we cannot ascertain the ac- tual statistics of all the languages in their training data. We use CC100 (Wenzek et al., 2020), a large multilingual corpus, estimate these statisties.
NLP research landscape, where more and more in- dustrial labs are building their own APIs, this is concerning trend that may reduce the accessibility of these technologies to already marginalized com- munities. Hence, we encourage the vendors to be more transparent about their models' limitations and rethink their pricing policy.
Reproducibility
One limitation with testing closed LMs is lack of reproducubility particularly
12The only official information they provide about Chat- GPT's multilingual support is here: https: //help. apena t.com/ent
Ethics Statement
opena ieddo---se--ie set-uss-thee-pppe the-opena i i-api-in- di wwwfeffeeeaammne languages Prior studies have specu- lated that CharGPT was trained on at least 90 languages (Ahuja et al., 2023).
This work sheds light on the consequences of unfair tokenization to users of commercial LM APIs that

---

because the model weights are typically updated continually. However, this only affects the down- stream evaluations as our cost analysis repro- ducible, since the tokenizers we evaluate are open- source.
Acknowledgements
We thank the members of the Tsvetshop and Noah's 's ARK labs at the University of Washington for the valuable discussions and useful feedback. We thank Lucille Njoo for help with our illustrations. We also thank the reviewers and area chair for their valuable feedback. During the course of this study, S.K. was supported by Google Ph.D. Fellow- ship. We also gratefully acknowledge support from NSF CAREER Grant No. ISS2142739, the Alfred P. Sloan Foundation Fellowship, and NSF grants No. ISS2125201, ISS220309,, and IIS21 13530. Any opinions, findings and conclusions or recom- mendations expressed in this material are those of the authors and do necessarily state flect those of the United States Government or any agency thereof.
References
Mohamed Abdalla, Jan Philip Wahle, Terry Ruas, Au- rélie Névéol, Fanny Ducel, Saif Mohammad, and Karën Fort. 2023. The elephant in the room: Ana- lyzing the presence of big tech in natural language processing research. Proceedings of ACL.
Kabir Abuja, Rishav Hada, Millicent Ochieng, Prachi Jain, Harshita Diddee, Samuel Maina, Tanuja Ganu, Sameer Segal, Maxamed Axmed, Kalika Bali, and Sunayana Sitaram. 2023. Mega: Multilingual evalua- tion of generative ai.
Farhad Akhbardeh, Arkady Arkhangorodsky. Mag- dalena Biesialska, Ondfej Bojar, Rajen Chatter- jee, Vishrav Chaudhary, Marta R. Costa-jussa, Cristina España-Bonee Angela Fan, Christian Fe- dermann, Markus Freitag, Yvette Graham, Ro- man Grundkiewicz, Barry Haddow, Leonie Harter, Kenneth Heafield, Christopher Homan, Matthias Huck, Kwabena Amponsah-Kann Kaakyire, Jungo Kasai, Daniel Khashabi, Kevin Knight, Tom Koemi, Philipp Koehn, Nicholas Lourie, Christof Monz. Makoto Morishita, Masaaki Nagata, Ajay Nagesh, Toshiaki Nakazawa, Matteo Negri, Santanu Pal, Allahsera Au- guste Tapo, Marco Turchi, Valentin Vydrin, and Mar- cos Zampieri. 2021. Findings of the 2021 conference on machine translation (WMT21).. In Proceedings of the Sixth Conference Machine Translation, pages 1-88, Online. Association for Computational Linguis- tics.
Mikel Artetxe, Sehastian Ruder, and Dani Yogatama. 2019. On the cross-lingual transferability of mono- lingual representations. In Annual Meeting of the Association for Computational Linguistics.
Sidney Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang.

---

Michael Pieler. Usvsn Sai Prashanth, Shivanshu Puro- hit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. 2022. GPT-NeoX-20B: An open- source autoregressive language model. In Proceed- ings of BigScience Episode #5 - Workshop on Chal- lenges & Perspectives in Creating Large Language Models, pages 95-136, virtual+Dublin. Association for Computational Linguistics.
Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Ja- cob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. 2022. Scaling instruction-finetueed language models.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross- pooss-liggual representation learning at scale. In Pro- ceedings of the 58th Amnual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451, Online. Association for Computational Lin- guistics.
Kaj Bostrom and Greg Durrett. 2020. Byte pair encod- ing is suboptimal for language model pretraining. In Findings of the Association for Computational Lin- guistics: EMNLP 2020, pages 4617 Online. Association for Computational Linguistics.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan. Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler. Jeffrey Wu. Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems. volume 33, pages 1877-1901. Curran Associates, Inc.
Alexis Conneau and Guillaume Lample. 2019. Cross- lingual language model pretraining. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Alexis Conneau, Ruty Rinott, Guillaume Lample, Ad- ina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. Xnli: Evaluating cross- lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natu- ral Language Processing. Association for Computa- tional Linguisties.
Tim Detimers Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. GPT3.int80): 8-bit matrix mul- tiplication for transformers at scale. In Advances in Neural Information Processing Systems.
Aydar Bulatov, Yuri Kuratov, and Mikhail S. Burtsev. 2023. Scaling transformer to IM tokens and beyond with RMT.
Yang Trista Cao and Hal Daumé III. 2020. Toward gender-inclusive coreference resolution. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4568 4595, On- line. Association for Computational Linguistics.
Sunipa Dev, Emily Sheng. Jieyu Zhao, Aubrie Amstutz, Jiao Sun, Yu Hou, Mattie Sanseverino, Júin Kim, Ak- ihiro Nishi, Nanyun Peng. and Kai-Wei Chang. 2022. On measures of biases and harms in NLP. In Find- ings of the Association for Computational Linguis- tics: AACL-IJCNLP 2022, pages 246-267, Online only. Association for Computational Linguistics.
Yan Chen, Christopher Mahoney, Isabella Grasso, Fssma Wali, Abigail Matthews, Thomas Middleton, Mariama Njie, and Jeanna Matthews. 2021, Gender bias and under-representation in natural language pro- cessing across human languages. In Proceedings of the 2021 AAAI/ACM Conference on AI. Ethics, and Society, AIES '21, page 34.34, New York, NY, USA. Association for Computing Machinery.
Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng- Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Kr- ishnan, Mare' Aurelio Ranzato, Francisco Guzmán, and Angela Fan. 2022. The Flores- evaluation benchmark for low-resource and multilingual ma- chine translation. Transactions of the Association for Computational Linguistics, 10:522-538.
Hyung Won Chung, Dan Garrette, Kiat Chuan Tan, and Jason Riesa. 2020. Improving multilingual models with annguag-cllusered vocabularies. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4536-4540 Online. Association for Computational Linguistics.
Ashim Gupta and Vivek Srikumar. 2021. X-fact: A new benchmark dataset for multilingual fact checking. Annual Meeting of the Association for Computational Linguistics.
Suchin Gururangan, Dallas Card, Sarah Dreier, Emily Gade, Leroy Wang, Zeyu Wang, Luke Zettlemoyer. and Noah A. Smith. 2022. Whose language counts as high quality? measuring language idcologies in text data selection. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 2562-2580, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, Al- bert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdh- ery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai,

---

Tahmid Hasan, Abhik Bhattacharjee. Wasi Uddin Ah- mad, Yuan-Fang Li. Yong-Bin Bing-Bin&oo Kang, and Rifat Shahriyar. 202 la. CrossSum: Beyond English- centric cross-lingual abstractive text summarization for 1500+ language pairs. ArXiv, abs/21 12.08804.
Sarin, Sokhar Samb, Benoî Sagot, Clara Rivera. Annette Rios, Isabel Papadimitriou, Salomey Osei. Pedro Ortiz Suarez. Iroro Orife, Kelechi Ogueji, An- dre Niyongabo Rubungo, Toan Q Nguyen, Math- ias Müller, André Müller, Shamsuddeen Hassan Muhammad, Nanda Muhammad, Ayanda Mnyak- eni, Jamshidbek Mirzakhalov, Tapiwanashe Matan- gira, Colin Leong, Nze Lawson, Sneha Kudugunta, Yacine Jernite, Mathias Jenny, Orhan Firat, Bonaven- ture F. P. Dossou, Sakhile Dlamini, Nisansa de Silva, Sakine Çabuk Balh, Stella Biderman, Alessia Bat- tisti, Ahmed Baruwa, Ankur Bapna, Pallavi Baljekar, Isracl Abebe Azime, Ayodele Awokoya, Duygu Ata- man, Orevaoghene Ahia, Oghenefego Ahia, Sweta Agrawal, and Mofetoluwa Adeyemi. 2022. Quality at glance: An audit of web-crawled multilingual datasets. Transactions of the Association for Compu- tational Linguistics, 10:50-72.
Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Samin, Yuan-Fang Li, Yong-Bin Kang, M. So- hel Rahman, and Rifat Shahriyar. 2021b. XI-sum: Large-scale multilingual abstractive summarization for 44 languages. In Findings.
Charles F. Hockett. 1997. The world's writing systems. Language, 73(2):379-385 Accessed 17 Oct. 2023.
Valentin Hofmann, Hinrich Schuetze, and Janet Pierre- humbert. 2022. An embarrassingly simple method to mitigate undesirable properties of pretrained lan- guage model tokenizers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 385-393, Dublin, Ireland. Association for Computational Lin- guistics.
Taku Kudo. 2018. Subword regularization: Improv- ing neural network translation models with multiple subword candidates. In Proceedings of the 56th An- nual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 66-75, Melbourne, Australia. Association for Computational Linguistics.
Haoyang Huang. Tianyi Tang, Dongdong Zhang, Wayne Xin Zhao, Ting Song, Yan Xia, and Furu Wei. 2023. Not all languages are created equal in LLMs: Improving multilingual capability by cross- tingual-thought prompting.
Viet Dac Lai, Nghia Trung Ngo, Amir Pouran Ben Vey- seh, Hieu Man, Franck Dernoncourt, Trung Bui, and Thien Huu Nguyen. 2023. ChatGPT beyond En- glish: Towards comprehensive evaluation of large language models in multilingual learning.
Ben Hutchinson, Vinodkumar Prabhakaran, Emily Den- ton, Kellie Webster, Yu Zhong, and Stephen Denuyl. 2020. Social biases in NLP models as barriers for persons with disabilities. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5491-5501, Online. Association for Computational I Linguistics.
Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, Jörg Frohberg, Mario Šaško, Quentin Lhoest, Angelina McMillan-Major, Gerard Dupont, Stella Biderman, Anna Rogers, Loubna Ben allal, Francesco De Toni, Giada Pistilli, Olivier Nguyen, Somaieh Nikpoor, Maraim Masoud, Pierre Colombo, Javier de la Rosa, Paulo Villegas. Tristan Thrush. Shayne Longpre, Se- bastian Nagel. Leon Weber, Manuel Muñoz, Jian Zhu, Daniel Van Strien, Zaid Alyafcai, Khalid Al- mubarak, Minh Chien Vu. Itziar Gonzalez-Dios, Aitor Soroa, Kyle Lo, Manan Dey, Pedro Ortiz Suarez, Aaron Gokaslan, Shamik Bose, David Ade- lani, Long Phan, Hieu Tran, Ian Yu, Suhas Pai, Jenny Chim, Violette Lepercq, Suzana Ilie, Margaret Mitchell, Sasha Alexandra Luccioni, and Yacine Jer- nite. 2023, The BigScience ROOTS Corpus: A 1.6tb composite multilingual dataset.
Melvin Johnson, Mike Schuster, Quoe V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas. Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2017. Google's multilingual neural machine translation system; En- abling zero-shot translation. Transactions the As- sociation for Computational Linguistics, 1339-351.
Danial Kamali, Behrooz Janfada, Mohammad Ebrahim Shenasa, and Behrouz Minaci-Bidgae Biddoli. 2022. Evalu- ating Persian tokenizers.
Jungo Kasai, Yuhei Kasai, Keisuke Sakaguchi, Yutaro Yamada, and Dragomir R. Radev. 2023. Evaluating GPT-4 and ChatGPT on Japanese medical licensing examinations.
Stav Klein and Reut Tsarfaty. 2020. Getting the #ilife out of living: How adequate are pirces for mod- elling complex morphology? In Proceedings of the 17th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 204-209, Online. Association for Computa- tional Linguisties.
Sara Laviosa. 2002. Corpus- translation studies: Theory, findings, applications.
Sabrina J. Mielke, Zaid Alyafeai, Elizabeth Salesky, Colin Raffel, Manan Dey, Matthias Gallé, Arun Raja. Chenglei Si, Wilson Y. Lee, Benoît Sagot, and Sam- son Tan. 2021. Between words and characters: A brief history of open-vocabulay modeling and tok- enization in NL.P.
Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orshik,, Allah- sera Tapo, Nishant Subramani, Artem Sokolov, Clay- tone Sikasote, Monang Setyawan, Supheakmungkol

---

Niklas Muennighoff. Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Rose Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir R. Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. 2022. Crosslingual gen- eralization through multitask finetuning. ArXiv, 1112221.0178...
Aslanides, Sarah Henderson, Roman Ring, Susan- nah Young, Eliza Rutherford, Tom Hennigan, Ja- cob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Mari- beth Rauh, Po-Sen Huang, Amelia Glaese Jo- hannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Anto- nia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Bud- den, Esme Sutherland, Karen Simonyan, Michela Pa- ganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Flena Gribovskaya, Domenic Donato, Angeliki I. Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsim- poukelli, Nikolai Grigorev, Doug Fritz. Thibault Sot- tiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d' Autume, Yujia Li. Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin. Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, Iason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis. Ko- ray Kavukcuoglu, and Geoffrey Irving. 2022. Scaling language models: Methods, analysis & insights from training gopher.
Benjamin Muller, Antonios Anastasopoulos. Benoît Sagot, and Djamé Seddah. 2021. When being un- seen from mBERT'S is just the beginning: Handling new languages with multilingual language models. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tiondl Linguistics: Human Language Technologies, pages 448-462, Online, Association for Computa- tional Linguistics.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pages 27730-27744. Curran Associates, Inc.
Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. 2021. How good is your tok- enizer? on the monolingual performance of multilin- gual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 31 3118-3135, Online. Association for Computational Linguistics.
Gunho Park, Baeseong Park. Minsub Kim, Sungjae Lee, Jeonghoon Kim, Beomseok Kwon, Se Jung Kwon, Byeongwook Kim, Youngjoo Lee, and Dongsoo Lee. 2023. LUT-GEMM: Quantized matrix multiplication based on luts for efficient inference in large-scale generative language models.
Aleksandar Petrov, Emanuele La Malfa, Philip H. S. Torr, and Adel Bibi. 2023. Language model tokeniz- ers introduce unfaireess between languages.
Teven Le Scao, Angela Fan, Christopher Akiki, Elizabeth-Jane Pavlick, Suzana 1lf' c, Daniel Hesslow, Roman Castagn'e, e. Alexandra Sasha Luccioni, Franc- cois Yvon, Matthias Gallé, Jonathan Tow, Alexan- der M. Rush, Stella Rose Biderman, Albert Web- son, Pawan Sasanka Ammanamanchi, Thomas Wang. Benoît Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan Iz Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Laurenccon, Yacine Jer- nite, Julien Launay, Margaret Mitchell, Colin Raf- fel, Aaron Gokaslan, Adi Simhi, Aitor Soroa Etx- abe, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris C. Emezue, Christopher Klamm, Colin Leong. Daniel Alexander van Strien, David Ifeoluwa Ade- lani, Dragomir R. Radev, Eduardo Gonz'ale, Pon- ferrada, Efrat Levkovizh. Ethan Kim, Eyal Bar Natan, Francesco De Toni, Gérard Dupont, Germán Kruszewski, Giada Pistilli, Hady ElSahar, Hamza Benyamina. Hieu Trung Tran, Ian Yu, Idris Abdul- mumin, Isaac Johnson, Itziar Gonzalez-Dios, Javier de la Rosa, Jenny Chim, Jesse Dodge, Jian Zhu, Jonathan Chang, Jorg Frohberg, Josephine L. To- bing, Joydeep Bhattacharjee, Khalid Almubarak,
Ofir Press, Noah Smith, and Mike Lewis. 2022. Train short, test long: Attention with linear biases enables input length extrapolation. In International Confer- ence on Learning Representations.
Ivan Provilkov, Dmitrii Emelianenko. and Elena Voita. 2020. dro-ootp Simple and effective subword regularization. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 1882-1892, Online. Association for Computational Linguistics.
Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is ChatGPT a general-purpor natural language process- ing task solver?
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John

---

Kimbo Chen, Kyle Lo. Leandro von Werra, Leon Weber, Long Phan, Loubna Ben Allal, Ludovic Tan- guy, Manan Dey, Manuel Romero Muñoz, Maraim Masoud, Mar'n ia Grandury, Mario vSavsko, Max Huang, Maximin Coavoux, Mayank Singh, Mike Tian-Jian Jiang, Minh Chien Vu, Mohammad Ali Jauhar, Mustafa Ghaleb, Nishant Subramani, Nora Kassner, Nurulaqilla Khamis, Olivier Nguyen, Omar Espejel, Ona de Gibert, Paulo Villegas, Peter Hen- derson, Pierre Colombo, Priscilla Amuok, Quentin Lhoest, Rheza Harliman, Rishi Bommasani, Roberto Oopee Rui Ribeiro, Salomey Osei, Sampo Pyysalo, Sebastian Nagel, Shamik Bose, Shamsuddeen Has- san Muhammad, Shanya Sharma, S. Longpre, So- maieh Nikpoor. Stanislav Silberberg. Suhas Pai, Syd- ney Zink, Tiago Timponi Torrent, Timo Schick, Tris- tan Thrush, Valentin Danchev, Vassilina Nikoulina, Veronika Laippala, Violette Lepereg. Vrinda Prabhu. Zaid Alyafeai, Zeerak Talat, Arun Raja, Benjamin Heinzerling, Chenglei Si, Elizabeth Salesky, Sab- rina Jiekle. Wilson Y. Lee, Abheesht Sharma, An- drea Santilli, Antoine Chaffin, Arnaud Stiegler, Deba- jyoti Datta, Eliza Szczechla, Gunjan Chhablani, Han Wang, Harshit Pandey, Hendrik Strobelt, Jason Alan Fries, Jos Rozen, Leo Gao, Lintang Sutawika, M Sai- ful Bari. Maged S. Al-shaibani, Matteo Manica, Ni- hal V. Nayak. Ryan Teehan. Samuel Albanie, Sheng Shen, Srulik Ben-David, Stephen H. Bach, Taewoon Kim, Tali Bers, Thibault Févry, Trishala Neeraj, Ur- mish Thakker, Vikas Raunak, Xiang Tang, Zheng Xin Yong. Zhiqing Sun, Shaked Brody, Y Uri, Hadar Tojarieh, Adam Roberts, Hyung Won Chung, Jae- sung Tae, Jason Phang. Ofrr Press, Conglong Li, Deepak Narayanan, Hatim Bourfoune, Jared Casper, Jeff Rasley, Max Ryabinin, Mayank Mishra, Minjia Zhang. Mohammad Shoeybi. Myriam Peyrounette. Nicolas Patry, Nouamane Tazi, Omar Sanseviero, Patrick von Platen, Pierre Cornette, Pierre Franc- cois Lev Rémi Lacroix, Samyam Rajbhan- dari, Sanchit Gandhi, Shaden Smith. Stéphane Re- quena, Suraj Patil, Tim Dettmers, Ahmed Baruwa, Amanpreet Singh. Anastasia Cheveleva, Anne-Laure Ligozat, Arjun Subramonian, Aur'elie N' N'ev'co eol, Charles Lovering, Daniel Garrette, Deepak R. Tunuguntla, Ehud Reiter, Ekaterina Taktasheva, Eka- terina Voloshina, Eli Bogdanov, Genta Indra Winata, Hailey Schoelkopf, Jan-Christoph Kalo, Jekate- rina Novikova, Jessica Zosa Forde, Xiangru Tang, Jungo Kasai, Ken Kawamura, Liam Hazan, Ma- rine Carpuat, Miruna Clinciu, Najoung Kim, New- ton Cheng. Oleg Serikov, Omer Antverg, Oskar van der Wal. Rui Zhang, Ruochen Zhang, Sebastian Gehrmann, S. Osher Pais, Tatiana Shavrina, Thomas Scialom, Tian Yun, Tomasz Limisiewicz, Verena Rieser, Vitaly Protasov. Vladislav Mikhailov, Yada Pruksachatkun, Yonatan Belinkov, Zachary Bam- berger. Zdenvek Kasner, Alice Rueda, Amanda Pes- tana, Amir Feizpour, Ammar Khan, Amy Faranak, Ananda Santa Rosa Santos, Anthony Hevia, Antig- ona Unidreaj. Arash Aghagol, Arezoo Abdollahi, Aycha Tammour, Azadeh HajiHosseini, Baharch Behroozi, Benjamin Olusola Ajibade, Bharat Kumar Saxena, Carlos Muñoz Ferrandis, Danish Contrac-
tor, David M. Lansky, Davis David, Douwe Kiela, Duong Anh Nguyen. Edward Tan, Emily Baylor, Ez- inwanne Ozoani, Fatim T Mirza, Frankline Onon- iwu, Habib Rezanejad, H.A. Jones, Indrani Bhat- tacharya, Irene Solaiman, Irina Sedenko, Isar Ne- jadgholi, Jan Passmore, Joshua Seltzer, Julio Bonis Sanz, Karen Fort, Lívia Macedo Dutra, Mairon Sama- gaio, Maraim Elbadri, Margot Mieskes, Marissa Ger- chick, Martha Akinlolu, Michael McKenna, Mike Qiu, M. K. K. Ghauri. Mykola Burynok, Nañis Abrar, Nazneen Rajani, Nour Elkott, Nourhan Fahmy, Olanrewaju Samuel, Ran An, R. P. Kromann, Ryan Hao, Samira Alizadeh, Sarmad Shubber, Silas L.. Wang. Sourav Roy. Sylvain Viguier, Thanh-Cong Cong Le. Tobi Oyebade, Trieu Nguyen Hai Le. Yoyo Yang. Zachary Kyle Nguyen, Abhinav Ramesh Kashyap, Alfredo Palasciano, Alison Callahan, Anima Shukla, Antonio Miranda-Escalade Ayush Kumar Singh, Benjamin Beilharz, Bo Wang. Caio Matheus Fon- seca de Brito, Chenxi Zhou, Chirag Jain, Chuxin Xu, Clémentine Fourrier, Daniel Le'on on Perin" an, Daniel Molano, Dian Yu, Enrique Manjavacas, Fabio Barth, Florian Fubrimann, Gabriel Altay, Giyased- din Bayrak, Gully A. Burns, Helena U. Vrabec. Iman LBB. Bello, Isha Dash, Ji Soo Kang, John Giorgi. Jonas Golde Jose David Posada, Karthi Sivaraman, Lokesh Bulchandani, Lu Liu, Luisa Shinzato, Madeleine Hahn de Bykhovetz, Maiko Takeuchi, Mare Pamies, María Andrea Castillo, Mar- ianna Nezhurina, Mario Sanger, Matthias Samwald, Michael Cullan, Michael Weinberg, M Wolf, Mina Mihaljcic, Minna Liu, Moritz Freidank, Myung- sun Kang, Natasha Seelam, Nathan Dahlberg. Nicholas Michio Broad, Nikolaus Muellner, Pascale Fung, Patricia Haller, R. Chandrasckhar, R. Eisen- berg, Robert Martin, Rodrigo L. Canalli, Rosaline Su, Ruisi Su, Samuel Cahyawijaya, Samuele Garda, Shlok S Deshmukh, Shubhanshu Mishra, Sid Ki- blawi, Simon Ott, Sinee Sang aroonsiro,iiii Srishti Ku- mar, Stefan Schweter, Sushil Pratap Bharati, T. A. Laud, Th's eo Gigant, Tomoya Kainuma, Wojeiech Kusa, Yanis Labrak, Yashasvi Bajaj. Y. Venkatra- man, Yifan Xu, Ying Xu, Yun chao Xu, Zhee Xao Tan, Zhongli Xie, Zifan Ye, Mathilde Bras, Younes Belkada, and Thomas Wolf. 2022. Bloom: A 176b- parameter open-access multilingual language model. ArXiv, abs/221 1.05
Rico Senarich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 1715-1725, Berlin, Germany. Association for Computational Lin- guistics.
Emily Sheng, Kai-We Aei-Wei Chang, Prem Natarajan, and Nanyun Peng. 2021. Societal biases in language generation: Progress and challenges. In Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 4275-4293, Online. Association for Computational Linguistics.

---

Xinying Song, Alex Salcianu, Yang Song, Dave Dopson, and Denny Zhou. 2021. Fast WordPiece tokenization. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 2089-2103, Online and Punta Cana, Dominican Re- public. Association for Computational Linguisties.
Safyyaa Saleem, Holger Schwenk, and Jeff Wang. 2022. No language left behind: Scaling human- centered machine translation.
Antti Virtanen, Jenna Kanerva, Rami lo,, Jouni Luoma, Juhani Luotolahti, Tapio Salakoski. Filip Ginter, and Sampo Pyysalo. 2019. Multilingual is not enough: BERT for Finnish.
Xinying Song, Alexandru Salcianu, Yang Song. Dave Dopson, and Denny Zhou. 2020. Fast WordPiece tokenization. In Conference on Empirical Methods in Natural Language Processing.
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Con- neau, Vishrav Chaudhary, Francisco Guzmán, Ar- mand Joulin, and Edouard Grave. 2020. CCNet: Extracting high quality monolingual datasets from web crawl data. In Proceedings of the Twelfth Lan- guage Resources and Evaluation Conference, pages 4003-4012, Marseille, France. European Language Resources Assoiiation.
Hongjin Su, Jungo Kasai. Chen Henry Wu, Weijia Shi, Tianlu Wang. Jiayi Xin, Rui Zhang. Mari Ostendorf, Luke Zettlemoyer. Noah A. Smith, and Tao Yu. 2023, Selective annotation makes language models better few-shot learners. In Proc. of ICLR.
Robert Wolfe and Aylin Caliskan. 2021. Low frequency names exhibit bias and overfitting in contextualizing language models. In Proceedings of the 2021 Con- ference on Empirical Methods in Natural Lnguage Processing, pages 518-532, Online and Punta Cana. Dominican Republic. Association for Computational Linguistics.
Jimin Sun, Patrick Fernandes, Xinyi Wang, and Gra- ham Neubig. 2023. A multi-dimennional evaluation of tokenizer-free multilingual pretrained models. In Findings of the Association for Computational Lin- guistics: EACL 2023, pages 1725-1735, Dubrovnik, Croatia, Association for Computational Linguistics.
Tianxiang Sun, Yunfan Shao, Hong Qian, Xuanjing Huang, and Xipeng Qiu. 2022. Black-box tnning for language- taaguage-model-as-ee model- In Proceedings of ICML.
Jingjing Xu, Hao Zhou, Chun Gan, Zaixiang Zheng, and Lei Li. 2021. Vocabulary learning via optimal transport for neural machine translation. In Proceed- ings of 59th Annual Meeting of the Association for Computational Linguistics and the I 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 7361-7373, Online. Association for Computational Linguistics.
Zeerak Talat, Joachim Bingel, and Isabelle Augenstein. 2021. Disembodied machine learning: On the illu- sion of objectivity in NLP. ArXiv, abs/2101.11974.
Zeerak Talat, Aurelie Névéol, Stella Biderman, Miruna Clinciu, Manan Dey, Shayne Longpre, Sasha Luc- cioni, Maraim Masoud, Margaret Mitchell, Dragomir Radev, Shanya Sharma, Arjun Subramonian, Jacsung Tac, Samson Tan, Deepak Tumuguntla, and Oskar Van Der Wal. 2022. You reap what you sow: On the chal- lenges of bias evaluation under multilingual settings. In Proceedings of BigScience Episode #5 Workshop on Challenges Perspectives in Creating Large Lan- guage Models, pages 26-41, virtual+ Dublin. Associ- ation for Computational Linguistics.
Shiyue Zhang, Vishrav Chaudhary, Naman Goyal, James Cross, Guillaume Wenzek, Mohit Bansal, and Francisco Guzman. 2022a. How robust is neural ma- chine translation to language imbalance in multilin- gual tokenizer training? In Proceedings of the 15th biennial conference of the Association for Machine Translation in the Americas (Volume 1: Research Track). pages 97-116, Orlando, USA. Association for Machine Translation in the Americas.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mi- haylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zeulemoyer. 2022b. Opt: Open pre-trained transformer language models.
Yi Tay, Vinh Q. Tran, Sebastian Ruder, Jai Gupta, Hyung Won Chung, Dara Bahri, Zhen Qin, Simon Baumgartner, Cong Yu, and Donald Metzler. 2022. Charformer: Fast character transformers via gradient- based subword tokenization. In International Con- ference on Learning Representutions.
Yiming Zhu, Peixian Zhang, Ehsan-Ul UI Haq, Pan Hui, and Gareth Tyson. 2023. Can ChatGPT reproduce human-generated labels? study of social computing tasks.
NLLB Team, Marta R. Costa-jusså, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Hef- fernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun. Skyler Wang. Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Bar- rault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan. Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan. Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers,
Judit Acs. 2019. Exploring BERT's vocabulary. http: / //juiitacs. tacs. gi thub. 1 2t0/219//2219/bbrttttt enization-se statiiionnssts..hee html.
A Prompt template
In Table 2, we provide the exact prompts we use for each respective task in our experiments.

---

B Cost analysis
Figure 11 we present the experimental cost rel- ative to English and Spanish for Crosssum and XFACT respectively. Figure 16 shows the esti- mated cost of GPT3.5 API access for all languages in CC100 relative to English.
C Extra analysis of fragmentation rate
In Figure 12 and Figure 13 we present the fragmen- tation rate across language families and scripts for both GPT3.5 and BLOOM respectively. Figure 17 shows the fragmentation rate for all languages in FLORES grouped by language script.
D Fragmentation rate vs pretraining data size
In Figure 14 we sort languages based on their size in CC100 corpus (Wenzek et al., 2020) and plot their fragmentation rate with GPT3.5 tokenizer. Figure 15 shows the same statistics for BLOOM's tokenizer based on language pretraining data size (ROOTS corpus; Laurençon et al., 2023).
Pretrained tokenizers on more languages
Figure 19 shows fragmentation rate across lan- guage scripts, when we train a BBPE tokenizer trained on parallel text in 30 languages.
F Fragmentation rate vs HDI
Figure 18 shows GPT3.5's fragmentation rate per language against the Human Development Index of the country with the largest amount of speak- ers of that language. We add more languages and countries here compared to the figure the main paper.

---

Task
Prompt Template
XLSUM
Write short summary sentence of the following text in
(language) Article: (rttilee Summary:
XQUAD
Context: context Question: question Answer: Template
XNLI
[Premise) Question [hypothesis} True, False, or Neither?
Answer:
CROSSUM
Write short summary sentence of the following text in
English. Article: article) Summary:
XFACT
Tell me whether the following claim (label or (label
2 or (label 3 ... given evidence (evidence }. (evidence
2 ) (evidence 3
Table 2: Prompt template used for each dataset.
Exprrtmet Cost matiiv Sporssh
Exper Cost reloove to Englih
10
de
pt
1t
tr
ka
13
3NV
ko
hi
ta
te
Language
Language
(a) XFACT
CROSSUM
Figure 11: Relative cost of prompt generated tokens for XFACT and PROSS-SUM evaluations.
Average number of tokens
40
30
Average number of token
200
20
10
Han (Simplified)
Devanagari
Arabic
Tamil
Bengali
Kannada
Telugu
Gujarati
Malayalam
Gurmukhi
Latin
Alro-Acitic
Adantic-Cong Congo
Austraasatic
Ausrooesian
Aymaran
Basque
Common Torkic
Constructd
Drovidian
Indo Europsan
Japonic
Kanvellon
Konanic
Mongalic- MMngsficKhhttn Khitan
Quechuan
Salaran
Tupion
Torkid
Smn- T
Language Family
Figure 12: Average number of tokens per language family after tokenizing Flores dataset with GPT3.5 tok- enizer. The fragmentation rate lower for Latin script languages and higher for other scripts.
Language Script
Figure 13: Average number of tokens per language script after tokenizing Flores dataset with BLOOM to- kenizer. The fragmentation rate higher on average for Latin script languages. This is because majority of the low-resourced languages are latin-script and have higher fragmentation rate.

---

300
Average number of tokens
200
100
eng
Rnd
Vi
ukn
sie
tha
ron
tra
kor
mam
kat
bep
mal
kaz
prd
mkdd
re
sin
kan
mal
kmnp
epo
amn
Language
Figure 14: Average number tokens per language after tokenizing FLORES with GPT3.5 tokenizer. Languages arranged in descending order based the size of pretraining data in Commoncrawl.
Average number of tokens
60
40
20
eng
zho
fra
spa
por
arb
vle
hin
inc
ben
cat
tam
ma ma
te
urd
eus
kan
mar
pan
guj
asm
swh
yor
kin
xoo
ibo
zul
şna
lug
wol
run
fon
n,50
lin
tsh
tw/
nya
sot
tso
aka
bam
kik
tum
Language
Figure 15: Average number of tokens per language after tokenizing FLORES with BLOOM tokenizer. Languages
are arranged in descending order based on the size of pretraining data in the ROOTS corpus.

---

12
Relative cost to English
8
4
5p3
dan
ita
aff
Ca3
e5t
ron
epo
bos
zhd
eus
tu
het
tha
hin
san
ma
U1g
bel
2517
pal
te
sin
kan
IT
Language
Figure 16: Estimated cost of GPT3.5 API access relative to English.
400
Arabic
Greek
Hebrew
Malaystam
Telugu
Gujorati
Myanmar
Thai
350
Bengali
Cunmukhi
Kannada
Of Chili
Tibetan
Average number of tokens
Cyrille
Han (Simplified)
Oriya
Trimngh
300
(Truditional)
Torkie
Goorgian
Laši
Tamil
250
200
150
Language
Figure 17: Average number of tokens by script after tokenizing all languages in the Flores dataset with GPT3.5 tokenizer.

---

Burmese
300
Average number of tokens
Armenigeorgian
250
KhmerKannada
SinAdaaayyllm
Telugu
Tamil Panjabi
200
Assamese
150
Bengali
Marathi
Uyghur
Greek
Urdu
Thai
100
Kyrgyz
Hebrew
Ukrainian
Vietnamesé pe
Maciddol
Rüssian
Somali
Japanese Hungarien Eenonia Portug Pettugueeshoo nermaan
50
Bosnian Spanish
English
0.4
0.5
0.6
0.7
0.8
0.9
Human Development Index
Figure 18: Fragmentation rate per language against the Human Development Index in the country with the largest amount of speakers of that language.
400
Arabic
Gujarati
Khmer
Sinhala
Armenian
Gurmukhi
Lao
Tamil
Bengali
Han (Simplified)
Latin
Telugu
350
Cyrillic
Han (Traditional)
Malayalam
Thai
Avg number of subwords
Devanagari
Hangul
Myanmar
Tibetan
300
Geez
Hebrew
Ol Chiki
Tifinagh
Georgian
Japanese
O Oriya
Turkic
250
Greek
Kannada
200
150
100
50
200
500
1000
5000
10000
25000
50000
Vocab size
Figure 19: BBPE tokenizer trained on parallel text from 30 language scripts with varying vocabulary sizes. It is impossible achieve uniform fragmnatation rate when have equal pretraining data sizes across all language scripts.


Reviews:
[{"review_id": "7Uu2UxfvPT", "reviewer": "Reviewer_YUzp", "summary": "", "questions": "- Why did you not use code for BloomZ to circumvent the API restrictions?\n- Why did you not evaluate Anthropic's Claude, Cohere, and GPT4?\n- Your results in figure 2 - are they generalizable given the small sample size of the scripts with the highest number of average tokens?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer YUzp for taking the time to review our paper, and noting that our work is critical and insightful in analyzing socio-economic implication of tokenization in LLMS. We are glad that they find our work particularly relevant to the wide NLP community. \n\n**Addressing your concerns**: \n\n**Circumventing BLOOMz API restrictions with code**: BloomZ is a 176B parameter model. Due to compute limitations in academic settings, we couldn’t load the model on our servers. We believe that is also the case with many researchers that cannot afford to deploy such systems locally. Further, many developers/ users that have little technical know-how and hence rely on querying the available APIs; we thus analyze the most common use-case for using BLOOMz. \n\n**Evaluating other LLMs**: At the time when we ran our experiments, Cohere models were not well optimized for multilingual in-context learning. In preliminary experiments, we saw poor results as the generated responses didn’t reflect any form of instruction following even in higher resourced languages beyond English. Also there was little information shared publicly regarding the multilingual capabilities of Claude, and our focus was on models that were explicitly stated to be multilingual. GPT4 was trained with the same tokenizer as ChatGPT, so we believe that results of our cost analysis will be similar.  We will add more discussion about this in the next version.\n\n**Are fragmentation-rate results generalizable?**: Our results in figure 2 show a trend of languages with non-Latin script having higher fragmentation rates. The majority of the languages that are included in the pretraining data of multilingual LMs are in Latin-script. Our focus here is to therefore highlight the tradeoffs of this data imbalance and how it could affect even higher-resourced languages with non-latin script. \n\n**Summary**: we thank the Reviewer again for their positive and helpful feedback! We will incorporate the above discussions in the final revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Exhaustive methodology along different axis - monolingual & multilingual models, model performance and socioeconomic implications.\n- Important findings on language properties and model utility based on tokenizer over-fragmentation of such languages.\n- Showcase the socioeconomic insidious implications of this.\n- Insightful discussions and recommendations for LLM practitioners and API builders \n- Evaluations on classification, span prediction and text generation", "reasons_to_reject": "- Only perform evaluation on ChatGPT and BloomZ. Missing Anthropic and Cohere.", "questions_to_authors": "- Why did you not use code for BloomZ to circumvent the API restrictions?\n- Why did you not evaluate Anthropic's Claude, Cohere, and GPT4?\n- Your results in figure 2 - are they generalizable given the small sample size of the scripts with the highest number of average tokens?", "paper_topic_and_main_contributions": "This paper investigates the disparate pricing among different languages on public LLMs APIs. They present an exhaustive methodology which yielded the following interesting findings:\n- Tokenizers of popular LLMs over-fragment texts in certain language scripts\n- They quantify the API cost disparity that this issue causes\n- They show that the tokenizer pretrainind data is not the sole culprits but also show that this stems from language properties and the way the language is represented in unicode\n- They show that such languages with over-fragmentation derive lower model utility from LLM APIs\n- They show that languages costing more come from countries and regions with lower Human Development Index (HDI), further exacerbating the economic divide. \n- Present discussions and suggests ways to remedy the situation.\n\nOverall, their findings present interesting and critical insights given the prevalence of public LLM APIs.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Missing references on LLM as APIs, e.g. https://arxiv.org/abs/2201.03514"}, "extra_text": {}}, {"review_id": "g0wjJQQt37", "reviewer": "Reviewer_JDu5", "summary": "", "questions": "A. I think one of the problems wrt testing closed source LLMs via web apis is total lack of reproducibility, but perhaps this is explained in the article and needs more underlining?\nB. Have you considered relating this research to morphological complexity and agglutinativity of the languages? E.g. maybe highly agglutinative as in almost concatenative languages still have more cost-friendly tokenisations?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer JDu5 for their time, thoughtful feedback and for noting that our work is important and relevant to the wide NLP community. \n\n**Addressing your concerns**:\n\n**Scientific concept of the Paper** : Our scientific contribution in this work is that we provide a general methodology to evaluate the fairness and utility of language model tokenizers across linguistic and economic axes. We believe this is particularly relevant given the recent commercialization of large language models, and because text tokenizers are used in every such model. Our methodology can therefore be applied across standard languages, dialects and model architectures. Our methods also rely on theoretically grounded socio-economic and linguistic indicators.  Further, the GPT series of models are one of the most widely user-adopted applications to date, and therefore we believe a study focusing solely on them has great scientific and practical value for the research community.\n\n**Lack of Reproducibility**:  Reproducibility is definitely a limitation with testing closed LLMs. Importantly, the tokenizers we evaluate are open-source, and our fragmentation analysis is therefore reproducible. We will release the code to reproduce our analyses. With regard to downstream evaluations, we will ensure to explicitly include this in the limitation section of the paper. \n\n**Morphological Complexity**: Thank you for your suggestion, yes we definitely believe that morphological complexity of a language plays a role in tokenization quality. In fact,  we compared fragmentation rates across language families with different morphological complexities. Uralic languages like Finnish and Hungarian for instance have slightly higher fragmentation rates than other Latin languages that might have less resources available. We will include plots from this analysis in the final paper. \n\n**Summary**: we greatly appreciate your time and the careful review! We will incorporate discussions about scientific contributions, reproducibility, and morphological complexity-related analyses in the final revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The points that are brought up in this article are highly important and relevnat tothe context of this conference. \n* extents of testing in terms of language range are good and will provide good basis for discussion", "reasons_to_reject": "* I am a bit torn on the scientific concept of an article based on just testing two off-the-shelf systems and providing mainly non-linguistic cost evaluation, however, I think this article is important and unique enough to warrant inclusion, especially as the focus of EMNLP does tend towards this direction anyways", "questions_to_authors": "A. I think one of the problems wrt testing closed source LLMs via web apis is total lack of reproducibility, but perhaps this is explained in the article and needs more underlining?\nB. Have you considered relating this research to morphological complexity and agglutinativity of the languages? E.g. maybe highly agglutinative as in almost concatenative languages still have more cost-friendly tokenisations?", "paper_topic_and_main_contributions": "Paper offers an extensive evaluation of disadvantages related to large language models in two external services. The evaluated services are the popular open AI model chatgpt and an open source alternative bloomz over huggingface api. The impacts are evaluated from the point of view of tokenisation with bbpe and overall use case. The evaluations highlight a very important aspect of inequality between languages written in latin alphabet and others. The tests are performed on typologically varied range of languages.", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The article reads well and is easy to follow. \n\nSome formulations are a bit hard to follow, e.g.:\n* on L240 \"Cyrillic and Japanese scripts[...]languages with their own script e.g. Telugu\", I guess Japanese is not strictly language with its own script but I am not sure if this expression is actually relevant to point being made here."}}, {"review_id": "c7EdetIW7F", "reviewer": "Reviewer_8QsM", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer 8QsM for taking the time to review our work and offering valuable comments. We acknowledge their suggestions and will incorporate them. \n\n**Addressing your concerns** :\n\n**Conceptual-theoretical foundation of the paper**: Our scientific contribution in this work is methodological:  we provide a general methodology to evaluate the fairness and utility of language model tokenizers across linguistic and economic axes. Our methods rely on theoretically grounded socio-economic indicators and are linguistically informed. We believe such general methods are particularly relevant now, given the recent commercialization of large language models. Our methodology can be applied across standard languages, dialects and model architectures, and can inform future development of more equitable text tokenizers.   \n\n**Fairness in LM API pricing**: In the paper, we highlight that the LLMs we study are advertised as capable of performing tasks in several languages. There are many factors that contribute to performance and the overall satisfaction users derive from these APIs. Our goal is to shed light on an often neglected factor, tokenization. Of course, we believe that token-ratios would often correlate with amount of pre-training data, we however believe that it is largely misleading for LLM companies to extol the multilingual capabilities of their models without considering the socio-economic impacts on users that speak lower-resourced languages. \n\n**Computing tokenizer fertility**:  For the purpose of our analysis, we define fragmentation rate as the average number of tokens after tokenizing a corpus. While fertility reflects the average number of subwords that correspond to a single word, we are instead interested in the total cost of tokenizing the same content, but in different languages. Since LLM companies charge users per token, we believe that this approach enables a fair comparison across languages, and is most relevant to our research question\n\n**Description of ChatgGPT and BLOOM tokenizers**: In line 297 we state that we hypothesize that disparity in ChatGPT’s tokenization could be attributed to training data imbalance. Given that ChatGPT is a closed API, we cannot ascertain the training setup. We only infer from previous studies that speculate that ChatGPT was trained on 90 languages. Upon acceptance, we will explicitly make this clear in the description of the tokenizer.  The BLOOMz tokenizer was trained on a corpus in which sentences from different languages are sampled according to a multinomial distribution, thereby increasing the number of tokens associated with low-resource languages. We will include this detailed description in the final version upon acceptance. \n\n**Results from analyzing all 200 languages in FLORES**: We conducted our analysis on all languages in the FLORES 200 dataset and decided to present an aggregated plot in the main paper due to space constraints. However our findings still hold across all languages in the FLORES 200 dataset.  We generally see that non-Latin-script languages have higher fragmentation rates irrespective of their level of resourcedness (we categorize resourcedness based on the FLORES 200 paper). Given additional space upon acceptance, we will include these charts in the main paper. \nRegarding analyzing BLOOM’s tokenizer on all languages, we see the same trends as above, except the fact that BLOOM has lower fragmentation rate on Arabic and some Dravidian languages present in its pre-training data. We will include these findings in the final version. \n\n**Controlled experiment reflected in Fig 3, 18**: When building BPE tokenizers, different languages have different allocations in the vocabulary and this has been largely attributed to pretraining data size and not inherent properties of the language. Our goal with this experiment is to actually see what you get when you have parallel data in all languages, this is our approach to control content. Of course we see that different amounts of bytes are required to yield the same size of vocabulary in different languages.  We highlight this to prove that data imbalance isn’t the sole cause of disparate fragmentation rates across languages. In the final version, we will include the actual language selected per script for this analysis. \n\n**HDI correlation**s: We made the decision to use HDI as a proxy for socio-economic indicators, following prior work [1]. Notably, GDP already has a high correlation with HDI and we thus expect the same correlations with fragmentation rates to hold for GDP as well. We will include this analysis in the final version.\n[1] Incorporating Dialectal Variability for Socially Equitable Language Identification](https://aclanthology.org/P17-2009) (Jurgens et al., ACL 2017) \n\n**Subset of FLORES used for RQ1 experiments** : We used the validation set of FLORES 200, and will make this explicitly clear in the final version of the paper.\n\n**Vocabulary sizes of ChatGPT and Bloomz**: The vocab sizes of BLOOM and ChatGPT are significantly different. ChatGPT has a vocabulary size of 50257 compared to BLOOM that has 250080.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A series of measurements relating to the number of tokens and utility per token across different languages/scripts in\ncommercialized LLM web APIs.", "reasons_to_reject": "Conceptual-theoretical foundation of the paper is weak and the author(s) do not seem to know the languages they are studying\nso the discussion is suboptimal. Some key parts do not have a satisfactory description and some crucial experiements are poorly conducted.\n\n\nMajor:\n* The author(s) use value judgements (\"fair\", \"unfair\" etc) with the\ntacit assumption that fair is that each language should have the same\nprice. They provide no philosophical or economical grounding for the\nunderlying assumption. But the token ratios depend on the amount of\ntraining data and companies in question operate in a supply-demand\neconomy so it's natural that languages for which data is in less supply\nis more expensive and that English is cheap.\n\n* \"Second, many languages we analyze are understudied and do not have word tokenizers 242\navailable which are required to compute fertility.\"\n\nNone of the languages analyzed is understudied in the relevant sense (i.e., wrt to tokenization). See, e.g., glottolog.org for descriptions of the phonology and writing systems of all. Nearly all have word tokenizers (only a few, e.g., Burmese is problematic in this domain).\n\n* The results re RQ1 do not feature 101 or 200 languages either in the text (well, one grouped figure is there) or the appendix. Yet the authors assert correlations of various kinds where the remaining 150-ish languages are highly relevant. (To exlcude them in the presentation or analsysi is what is really unfair...!)\n\n* The crucial Fig 2, 14-15 leave some things to desired. First, I want the\nfull list of 200 languages and the 13 programming languages to be included\nin the BLOOMZ case (they are relevant because the tokenization rate of one\nlanguage crucially depends on what other languages there are in the training\ndata). Second, informal rank correlations are asserted whereas it seems\nobvious a Pearson correlation is expected (maybe with the log of the data\nsize). Do formal Spearman or Pearson test if a correlation is asserted or\ndon't assert one. Third, the crucial observation \"low-resourced languages\nof Latin script appear to be less fragmented compared to other mid-resourced\nlanguages of non-Latin scripts.\" is problematic since is it not said where\nthe border between mid and low resources languages go and why you are\ncompared low-resource latin with mid-resource non-latin rather that\nmid to mid or low to low). Fig 14-15 (not Fig 2) is what should be in the\npaper rather than the appendix because the RQ1 asks about languages *not*\nscripts.\n\n* Re the BLOOMZ and ChatGPT tokenizers. The referenced desc of the ChatGPT\ntokenizer says it's just a BPE so the training setup is known, just not\nthe input data. The authors never explain what the Bloomz tokenizer does\nbut it seems it's doing something more than just (B)BPE because of the\nmuch smaller discrepancies in tok rate, not quite expected from the disparate\ninput ROOTS corpus. Do explain.\n\n* The crucial experiment reflected in Fig 3, 18 is completely\ninadequate.  The comparison is *per sentence* not acknowledging that\nsentence length can differ across languages, so what you might have\nfound is not diff tokenization rates but different sentence\nlength. (NB: Not all scripts indicate sentence boundaries but the\nsubset of languages studied appear to have such boundaries imposed in\nthe data as used). Presumably the vocabulary size is the vocabulary of\nthe BBPE tokenizer but different amount of bytes in different\nlanguages are required to yield the same size vocabulary. The authors\nsay content is controlled for, ok, but HOW? Fig 18 has one language\nper script, ok, but which language in each case? Finally it would be\nnice to know how many sentences are required to reach a vocabulary of\n50000?\n\n* An important limitation is that the Flores corpus is translations ***from English*** so one can expect the other languages to be a little longer than the English original.\n\n* \"In particular, most scripts are very sensitive to small vocabulary sizes compared to Latin and Hangul scripts\"\n\nNot quite, in Fig 18 the logosyllabic Chinese are before Hangul. Anyway, it's not clear why this is relevant because the vocabulary sizes are probably very big in the Bloomz and ChatGPT tokenizers.\n\n* The authors related the language-wide differences with HDI which is a composite of life expectancy, economic power etc. But the interpretations only relate to the economic power of the speakers (\"afford\", \"higher cost\" etc) and not the other factors in HDI. So the authors should have correlated directly with GDP and rather the entire purchasing power of the language not the average pp of a speaker. The data use is also substandard as only the \"top\" (in which sense?) country is selected, whereas speaker numbers across all countries are available in, e.g., Ethnologue.\n\n\n\n  \n\nMinor:\n* \"Previous work has shown that tokenization in multilingual models\nis usually biased towards high-resourced languages in the pretraining\ndata (Ács, 2019; Rust et al., 2021);\"\n\nIt follows theoretically from BBPE that frequent sequences in the\npretraining data get chunked more so if the pretraining data is\ndominated by some language then it follows that the tokenization will\nbe \"biased\" towards that language. There is no need to invoke an\nempirical study unless one is interested in the specific numbers.\n\n* We use a subset of FLORES-200 (Goyal et al., 2022), a multilingual parallel corpus containing examples in over 200 languages.\n\nWhat subset? The ref to Goyal et al. 2022 is for Flores-101 and not Flores-200. The difference is significant because the translation quality\nfor the addional 99 language will be quite different.\n\n* The authors confuse \"language\", \"script\" and \"script as used in the orthography for a language\" throughout. Do fix. It's often ok to have a shorthand but not in this case where the relevant differences are explored.\n\n* The ISO 639-3 code zho denotes a set of languages. Probably only cmn is intended.\n\n* Give the size of the FLORES corpus (3000 sentences/language)\n\n* It might be useful to look at pairs of languages that differ minimally\nexcept for the script (e.g, croatian/serbian, uzbek-uigur at different times, hindi-urdu etc)\n\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper provides a series of measurements relating to the number of tokens and utility per token across different languages/scripts in commercialized LLM web APIs. These measurements are novel and interesting but conceptual-theoretical foundation of the paper is weak and the author(s) do not seem to know the languages they are studying so the discussion is suboptimal. Some key parts do not have a satisfactory description and some crucial experiments are poorly conducted.\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "\n* \nClaims like \"There are hundreds of distinct writing systems in the world.\" should be backed up by a reference, e.g.\n\nDaniels, Peter T. & William Bright (eds.). (1996) The world's writing systems. Oxford: Oxford University Press.\n\n* \nCorrelations between computational language resources and the economic power of speakers has been done before, with somewhat better theoretical foundation, see e.g., pp 107-132 of\n\nHammarström, Harald. (2009) Unsupervised Learning of Morphology and the Languages of the World. Chalmers University of Technology and University of Gothenburg doctoral dissertation.\n", "Typos_Grammar_Style_and_Presentation_Improvements": "Sec 2 includes the answers to the RQs, duplicating this info from the answers that follow in the results checking.\n\nFigure 12 is orphaned (not referred to and whose relevant is not explained. Also it has errors like Turkic vs Common Turkic --- they are the same family, CT is the main branch of Turkic)\n\n\n"}}]

Author rebuttal:

Meta-review: *Summary*: It is fairly well-known that the suboptimal tokenization of text across different languages leads to large variance in the number of tokens required to convey the same content in different languages. In this work, the authors systematically analyze the cost and utility of OpenAI's API on 22 typologically diverse languages. The authors find that text in many languages tend to be heavily fragmented, and consequently incur higher costs and derive lower model utility from the OpenAI APIs. This is an analysis-driven paper critiquing the API pricing policy of charging by tokens that puts over-fragmented languages at a clear disadvantage and suggests ways to move forward.

*Evaluations*: R1 and R2 were very favourable in their review of this work (5/4 soundness scores, 4/4 excitement scores). R3 diverges in their review and has listed many concerns including clarifications needed about the main results in both Figs 2 and 3, details about the BLOOM-Z tokenizer, details about the datasets (FLORES corpus, etc.), the more careful use of ``script" whenever invoked, and a few other minor points. The authors have addressed most of these concerns.
Decision: Accept-Main