SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Unicode is a computing industry standard for representing text characters (The Unicode Consortium, 2022). Unicode supports virtually all languages (including many ancient ones, emojis and special characters) by assigning every grapheme, modifier, punctuation mark, control character or formatting character one of 1,114,112 integer codepoints. The codepoints can be represented in binary as the variable-width encoding UTF-8, which encodes every codepoint with one to four bytes, or the fixed-width UTF-32 which encodes all codepoints with four bytes (see Figure 4).
UTF-8 can therefore represent any string in any language as a string of bytes. As each byte can take only one out of 256 values, 256 tokens can be sufficient to encode all texts. In practice this is usually combined with the BPE tokenizer. At first, the corpus is en
Figure 4: Comparison of variable width Unicode encoding (UTF-8) and fixed width encoding (UTF-32). Image adapted from (The Unicode Consortium, 2022).
coded as UTF-8 bytes and then BPE is ran on top of it. As most characters occur frequently, BPE would assign them a dedicated token. If the model encounters a character that didn't exist in the training corpus (e.g., the medium skin tone waving hand), it can still represent it byte-by-byte (F0+9F+91+8B for the waving hand and F0+9F+8F+BD for the skin tone modifier). This allows the vocabulary to efficiently represent frequently occurring words and rare characters. For example, the sentence "I love açaí" could be tokenized as "I" + "love" + "a" + C3 + A7 + "a" + C3 + AD.
Byte-level and character-level tokenization. If we can represent any input with just 256 characters, then why bother with subword tokens? A key consideration is sequence length. This is since transformers (Vaswani et al., 2017), the currently predominant deep learning architecture for language models, have attention layers with a quadratic complexity in the input length. Hence, as the number of characters is much longer than the sub-word tokenization, working on the character level has been traditionally considered computationally inefficient. However, Chung et al. (2016), Lee et al. (2017), Gao et al. (2020), Clark et al. (2022) and Xue et al. (2022) proposed various architectures working around this issue and operating directly on characters or UTF-8 bytes.
B Parity for Linguistic Varieties
A language can vary according to factors such as geography, history, social class and culture. As a result, different dialects, pidgin and creole language variations emerge, each with its own distinct set of grammar, vocabulary and pronunciation rules.[9] Unequal treatment of certain dialects or languages can lead to social and economic disadvantages for those who speak them. Therefore, it is important to also study the tokenization differences between the "standard" language and its varieties.[10] Unfortunately, parallel corpora for dialects, pidgin and creole language variations are far and few in between. In this section, however, we show results on regional Swiss German varieties, Arabic and Japanese dialects, as well as Haitian and Mauritian creoles.
Swiss German dialects. Swiss German is a dialect continuum which significantly differs from the formal High German. German-speaking Switzerland is diglossic:11 High German is used alongside regional dialects (Hogg et al., 1984). In contrast to other dialects, the use of Swiss dialects is increasing (Sieber and Sitta, 1987) especially online (Ludi, 2007). Swiss German dialects are often considered unintelligible to High German speakers and sometimes even speakers of different dialects may find difficulty understanding each other (Russ, 1990). Therefore, ensuring that German-targeting NLP applications can process Swiss German dialects is important.
To this end, we compare the tokenization parity relative to High German of GottBERT (Scheible et al., 2020) on the regional dialects of Aargau, Bern, Basel, Graubünden, Luzern,
9While no standard definitions exist, dialects are usually considered to be regional variations of a language, whereas pidgin and creole languages are contact languages that emerge from the interaction of speakers of different languages (Muysken and Smith, 1994).
<sup>10</sup>We refer to the language that the datasets label as "standard", "official" or "dominant" without necessarily endorsing this designation.
<sup>11</sup>Diglossia is the situation of two dialects or languages being used by a single language community (Kaye, 2001).
Table 6: GottBERT tokenizer premiums on the SwissDial dataset for Swiss German dialects. The premium is computed with respect to High German.
Region	GottBERT parity
High German	1.00
Zürich	1.38
St. Gallen	1.40
Basel	1.41
Graubünden	1.44
Luzern	1.52
Aargau	1.53
Wallis	1.58
Bern	1.59
Table 7: ArabicBERT tokenizer premiums on the MADAR dataset for Arabic dialects. The premium is computed relative to Standard Arabic.
City	ArabicBERT	City	ArabicBERT
Jeddah	0.91	Sanaa	1.01
Doha	0.92	Beirut	1.02
Riyadh	0.92	Benghazi	1.02
Muscat	0.94	Cairo	1.03
Basra	0.95	Sfax	1.03
Salt	0.95	Tripoli	1.05
Baghdad	0.96	Aswan	1.06
Damascus	0.97	Alexandria	1.06
Aleppo	0.97	Tunis	1.06
Jerusalem	0.97	Algiers	1.07
Khartoum	0.98	Mosul	1.10
Amman	0.99	Fes	1.11
Std. Arabic	1.00	Rabat	1.17
St. Gallen, Wallis and Zürich. We use SwissDial, a parallel multidialectal corpus, as the basis of comparison (Dogan-Schönberger et al., 2021). It is worth noting, that the dialect of each city and its corresponding region may differ significantly. Therefore there might be large variations within regions as well.
The results in Table 6 show a disparity between the tokenization lengths for High German and the Swiss dialects with a premium ranging from 1.38 for the Zürich dialect, or Züritütsch, to 1.59 for the Bernese Bärndütsch. In fact, English has a lower premium than any Swiss dialect (1.35 on FLORES-200, Table 2) and the premium for Bernese German is close to the linguistically further Swedish (1.64) and Norwegian Bokmål (1.65). The following example from SwissDial shows how the sentence "Like he's waiting for something" has almost twice as long tokenization in Bernese German compared to High German:
The fact that the GottBERT tokenizer results in better parity for English, Swedish and Norwegian Bokmål than for Swiss German dialects highlights that it does not likely pick out stable linguistic constructs.
Arabic dialects. Similarly to Swiss German, Arabic is usually spoken in diglossic speech communities, where Modern Standard Arabic is spoken alongside at least one prestigious vernacular particular to the country or region (Bassiouney, 2009). As both Standard Arabic
Table 8: BERT Japanese tokenizer premiums on the CPJD dataset for Japanese dialects. The premium is computed with respect to Standard Japanese. The CPJD dataset consists of two parallel corpora with the dialects split across the two. Hence, we have also indicated the corpus for each dialect. Nara-ben has two entries as the dataset has transcriptions for two separate speakers. The suffix "-ben" (弁) means "speech" or "dialect".
Dialect	Corpus Parity	Dialect	Corpus Parity
Akita-ben	2	1.09	Miyazaki-ben	1	1.05
Awa-ben	2	1.09	Morokata-ben	1	1.15
Fukui-ben	2	1.04	Nara-ben	2	1.09
Fukuoka-ben	1	1.03	Nara-ben	2	1.03
Hiroshima-ben	1	1.02	Okayama-ben	1	1.15
Hokkaido-ben	2	1.06	Oosaka-ben	2	1.03
Iwaki-ben	2	1.08	Saitama-ben	1	1.01
Iyo-ben	1	1.05	Tosa-ben	1	1.03
Izumo-ben	1	1.10	Toshu-ben	1	1.06
Kanazawa-ben	2	1.11	Tsugaru-ben	1	1.09
Kyokotoba	2	1.07
and its dialects are commonly used in written communication, it is vital that tokenizers handle them equally well.
To assess the performance of Arabic tokenizers, we compare the tokenization lengths of ArabicBERT (Safaya et al., 2020) across 25 Arabic dialects. To this end, we use the MADAR parallel corpus of Arabic dialects (Bouamor et al., 2018).
Table 7 shows the premiums relative to Standard Modern Arabic. The premium varies from 0.91 for the Jeddah dialect to 1.17 for the Rabat dialect. This is significantly lower than the premium for English (1.83 on FLORES-200 Table 2). The range is also much smaller than for the Swiss German dialects and approximately half of the considered dialects have a lower premium than Standard Modern Arabic. Therefore, one could say that the tokenizer of ArabicBERT achieves tokenization parity for these 25 Arabic vernaculars. This is likely because the corpus and vocabulary set on which ArabicBERT was trained contained dialectical Arabic. It is also possible that Arabic dialects are closer to Modern Standard Arabic and more mutually intelligible than Swiss German dialects are to High German (Čeplö et al., 2016; Trentman and Shiri, 2020). Still, this difference between the parity for Swiss and Arabic dialects indicates that including a broader set of vernaculars and dialects in the corpus results in improved tokenization parity.
Japanese dialects. Japanese also has a number of regional dialects (Hattori, 1973). We compare the tokenization parity of BERT Japanese (Tohoku NLP Group, 2019) across them. We employ the CPJD dataset by Takamichi and Saruwatari (2018) which contains transcriptions of the voice recordings of 250 sentences across 20 dialects.
The results in Table 8 show that the premium compared to Standard Japanese (Tokyo dialect) ranges from 1.01 (for Saitama prefecture, neighbouring Tokyo) to 1.15 (for Morokataben and Okayama-ben). These all are significantly lower than the premium for English (1.49, as shown in Table 2). Therefore, similarly to ArabicBERT, this is an example of the tokenizer being relatively well-aligned with the dialects. This is likely because Japanese dialects are more closely related (and intelligible (Yamagiwa, 1967) to Standard Japanese speakers) than the Swiss dialects are to High German speakers.
Mauritian and Haitian Creoles. While creoles often have some similarities with a high-resource language (usually English or French), the differences are significant to necessitate special attention to their support (Lent et al., 2021, 2022). This is especially critical for emergency services and disaster management (Munro, 2010).
Mauritian Creole is based on French as well as the languages of slaves imported from Madagascar and East Africa. As the British gained control of Mauritius, they brought indentured labourers from India who further had an effect on the formation of the modern Mauritian
Creole (Seuren, 1995). Similarly, Haitian Creole (Kreyol) emerged from the interaction of French and the various Niger-Congo languages spoken by the Africans brought as slaves (DeGraff, 2007).
Considering that both languages have their basis in French, one would expect that tokenizers targeting French would have low tokenization parities for Mauritian and Haitian Creoles. However, taking the tokenizer of CamemBERT (Martin et al., 2020), the premium for Mauritian Creole is 1.20 using the MorisienMT parallel corpus (Dabre and Sukhoo, 2022). The premium for Haitian Creole is 1.64 when using the QEDv2 corpus (Tiedemann, 2012; Abdelali et al., 2014). Haitian Creole is also represented in the FLORES-200 dataset where the premium relative to French is 1.58. This is significantly larger than linguistically further languages such as English (1.20), Pangasinan (1.49) and Nigerian Fulfulde (1.54). Therefore, CamemBERT is not well-placed to tokenize French-related creoles despite the model being trained for French.
C Extended Tables of Tokenization Premiums
In addition to the models presented in the main text, these extended tables also include LLAMA (Touvron et al., 2023), MBart50 (Liu et al., 2020; Tang et al., 2020), SeamlessM4T (Barrault et al., 2023) and Qwen-VL (Bai et al., 2023).
Language	HBrew	112	122	135	159	-	235	100	128	172	116	100	-	122	122	148
Hindi	141	160	-	246	097	180	136	-	134	097	-	126	126	426	186
Chattisgarhi	141	160	-	246	097	180	136	-	134	097	-	126	126	426	186
Gricoanian	141	160	-	246	097	180	136	-	134	097	-	126	126	426	186
Hungarian	141	160	-	246	097	180	136	-	134	097	-	126	126	426	186
Iberoanien	141	160	-	246	097	180	136	-	134	097	-	126	126	426	186
Ibericus	141	160	-	246	097	180	136	-	134	097	-	126	126	426	186
Javanese	141	160	-	246	097	180	136	-	134	097	-	126	126	426	186
Kabayie	141	160	-	246	097	180	136	-	134	097	-	126	126	426	186
Jingpho	141	160	-	246	097	180	136	-	134	097	-	126	126	426	186
Kashmiri (Arabic script)	141	160	-	246	097	180	136	-	134	097	-	126	126	426	186
Kashmiri (Danaugargari script)	141	160	-	246	097	180	136	-	134	097	-	126	126	426	186
Georgian	141	160	-	246	097	180	136	-	134	097	-	126	126	426	186
Kazakh	141	160	-	246	097	180	136	-	134	097	-	126	126	426	186
Kazakh	141	160	-	246	097	180	136	-	134	097	-
Kazakh	141	160	-	246	097	180	136	-	134	097	-
Kazakh	141	160	-	246	097	180	136	-	134	097	-
Langauge	Eastern Padjabi	9.44	7.00	7.00	7.00	7.00	7.00	7.87	7.00	8.47	—	1.25	1.11	1.00	1.29	1.23
LLAMA	Papamento	1.65	1.98	1.98	1.98	1.98	1.98	1.75	1.98	1.33	1.37	1.25	1.03	1.37	1.32
GPT-2	Southern Pasho	4.27	5.39	5.39	5.39	5.39	5.39	3.83	3.39	5.47	—	—	—	—	1.38	1.47
p50k_base	RoBERTA	1.70	2.69	2.69	2.69	2.69	1.91	2.69	1.79	1.71	1.38	1.00	1.19	1.26
p50k_edit	cl100k_base	1.42	1.94	1.94	1.94	1.94	1.94	1.48	1.91	1.69	1.79	1.71	1.38	1.00	1.19
RoBERTA	GottBERT	1.96	2.20	2.20	2.20	2.20	2.20	2.20	1.61	1.54	1.40	1.14	1.59	1.54
CamemBERT	PhoBERT	1.70	2.48	2.48	2.48	2.48	1.88	2.48	1.69	1.34	1.46	1.13	1.71	1.29
RoCBert	XLM-RoBERTa	1.64	3.74	3.74	3.74	3.74	2.43	3.74	3.67	1.59	1.49	1.26	1.57	1.49
M2M100	Yale Chinese (Traditional)	1.21	2.58	2.58	2.58	2.58	2.26	2.26	1.21	1.21	1.21	1.21	1.21	1.21
Northern Malay	2.00	2.00	2.00	2.00	2.00	2.00	2.00	2.00	1.63	1.63	1.49	1.09	1.92	1.80
Zhu	2.09	2.41	2.41	2.41	2.41	2.41	2.41	2.41	1.67	1.61	1.51	1.47	1.39	1.39