Tokenization Impacts Multilingual Language Modeling: Assessing Vocabulary Allocation and Overlap Across Languages
Year: 2023

Abstract:
Multilingual language models have recently gained attention as a promising solution for representing multiple languages in a single model. In this paper, we propose new criteria to evaluate the quality of lexical representation and vocabulary overlap observed in sub-word tokenizers. Our findings show that the overlap of vocabulary across languages can be actually detrimental to certain downstream tasks (POS, dependency tree labeling). In contrast, NER and sentence-level tasks (cross-lingual retrieval, NLI) benefit from sharing vocabulary. We also observe that the coverage of the language-specific tokens in the multilingual vocabulary significantly impacts the wordlevel tasks. Our study offers a deeper understanding of the role of tokenizers in multilingual language models and guidelines for future model developers to choose the most suitable tokenizer for their specific application before undertaking costly model pre-training.1

heading: Future Work
text: ByT5: Towards a TokenFree Future With Pre-Trained Byte-to-Byte Models. ArXiv:2105. 13626 [cs]. Bo Zheng, Li Dong, Shaohan Huang, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, and Furu Wei. 2021. Allocating Large Vocabulary Capacity for Cross-Lingual Language Model Pre-Training. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 3203–3215. Association for Computational Linguistics. A Technical Details
A. 1 Tokenizer training details
We use the Huggingface Tokenizers library for training the Unigram and BPE tokenizers. We kept the default values for the training parameters. Namely, for Unigram, we use a maximum piece length of 16 and a shrinking factor of 0. 75. For BPE, we use alphabet size 1000 and minimum merge frequency 2. For all languages, we use SentencePiece (Kudo and Richardson, 2018) for word segmentation techniques instead of languagespecific word tokenizers. A. 2 Model Architecture and Pre-Training
In this study, we employed the Huggingface library (Wolf et al. , 2020) to conduct all experiments. The model architecture is based on XLM-Roberta, although for our purposes, it was scaled down. Specifically, the size of the embeddings is 768, the number of attention layers is 8, and the number of attention heads is 6. The maximum sentence length is 128, and the vocabulary size is 120000. The number of parameters is 150M and, therefore, roughly 2 times smaller than the XLM-Roberta base model. The model was pre-trained for 10 epochs with a batch size of 1024. The learning rate was 5e-5 with linear decay and weight decay and 1% warm-up steps. In pretraining, we used AdamW optimizer (Loshchilov and Hutter, 2019). In total, we pretrained 7 models. The models were trained on 3 Nvidia GPUs. The probing experiments were run on 1 Nvidia GPU with 40GB of memory (Nvidia A40). The pretraining took about 17 hours for each 6-language model and 60 hours for the models trained on the full set of 20 languages. We didn’t pursue any extensive hyperparameter search efforts as this was not the focus of our work. We selected the best batch size and learning rates for the pre-training based on a few trials. A. 3 Downstream Data and Training
The probes were for 30 epochs with early stopping and batch size 16. We used an initial learning rate of 2e-5. Other training parameters were the same as in pretraining. Probing experiments took between 5 to 180 minutes to complete on the same infrastructure as used for pretraining. We ran around 360 probe trainings. POS We use Part of Speech annotations from Universal Dependencies (de Marneffe et al. , 2021). The dataset is available for 17 languages analyzed by us (not covered: Swahili, Thai, Georgian). Each word is assigned one of the 17 coarse POS tags. NER We use Wikiann dataset (Pan et al. , 2017) consisting of Wikipedias article with annotated named entities of three types: location, person, and organization in IOB2. Following XTREME, we use balanced data splits from (Rahimi et al. , 2019). Dependency labeling As in Part of Speech, we use Universal Dependencies (de Marneffe et al. , 2021) for the dependency relation annotations. We use the largest UD treebank available for each language. For each word we predict one of the 37 universal relations to its head word. Because the relation is between two words, we use the concatenation of the two word representations along with their element-wise product as an input to the probe ([hw1;hw2;hw1 ⊙ hw2]). NLI We use XNLI dataset (Conneau et al. , 2018b) for Natural Language Inference. We train the linear classification probe on top of the concatenation of two sentence vectors and their elementwise product: [hs1;hs2;hs1 ⊙ hs2]. We predict one of two relations between the first of sentences (called premise): contradicts, entails, or is neutral to the second sentence (called a hypothesis). We evaluate XNLI with the accuracy of classification. XNLI contains data for 15 languages (not covered: te, ta, mr, he, ka). Sentence Retrieval We use up to 1,000 sentences aligned for pairs of languages from Tatoeba dataset (Artetxe and Schwenk, 2019). For the pairs including English, we use the same sample as in XTREME data collection. For other pairs, we perform sampling ourselves. We compute the cosine similarity between sentence representations across languages and find the best alignment with the Hungarian algorithm(Kuhn, 1955). We compute the accuracy as the number of correctly aligned sentences divided by the total number of sentences. B In-depth Tokenizers Analysis
In Figure 4, we present the probabilities of vocabulary units, computed on concatenate six languages corpora, learned by different tokenization
algorithms. Unigram and NOOVERLAP use a bigger fraction of the vocabulary for rarely appearing tokens (with probability lower than 10−6). BPE and TOKMIX produce a vast set of tokens with probabilities in the range between 10−5 and 10−6. Interestingly, the former algorithm allocates about 6000 vocabulary entries to tokens not appearing in the corpora. BPE is better than Unigram in vocabulary allocation throughout languages. To support this claim, we train Unigram and BPE tokenizers for different vocabulary sizes. We observe that both the average rank (Figure 5) and CPT (Figure 6) stop rising for vocab sizes above 250,000 (except for Chinese). For BPE, the metrics still steadily rise after this threshold, which makes it overperform Unigram for most languages. We think that the reason why Unigram does not learn valuable tokens after this point is the way the
initial vocabulary is constructed, i. e. , it is the set of all character n-grams appearing in the corpus with n lower than 16. In contrast to BPR, Unigram’s vocabulary won’t cover longer words than 16 characters, which are useful in modeling some languages. We believe that further work on identifying optimal strategies for multilingual tokenization is needed. Vocabulary units preferred by tokenizers. In Table 6, we show the tokens with the highest differences in empirical probabilities obtained with BPE and Unigram tokenizers for three languages. We see that Unigram prefers suffixes to prefixes. Also, it splits text more often into single, possibly due to lower vocabulary allocation. C Supplementary Results
C. 1 Visualizations
We present the additional visualization for the results for transfers across six languages for the tasks not presented in the main text: Dependency labeling 7a and NLI cross-lingual accuracy 7b, Sentence
retrieval accuracy 7c. The results of experiments for 20 languages: Jensen-Shanon Divergences 8, and cross-lingual transfers for POS 10a, NER 10b, dependency tree labeling 10c, XNLI 9a, sentence alignment 9b. C. 2 Results for All Languages We also include detailed results for the in-language experiments along with the proposed tokenizer metrics. In Table 7, we present the results for the six languages. C. 3 Correlation Analysis We present paired correlation plots for in-language metrics in Figure 11. We use the results from 20 language settings to increase the number of observations. In this analysis, we focus on the differences between the tokenization methods and want to marginalize the language-specific features (such as the pre-training and fine-tuning data size or the model’s preference for Indo-European languages). Therefore, for vocabulary allocation measures (AR, CPT) and downstream tasks, we subtract the mean for each language. For vocabulary overlap measure (JSD) and transfer values, we subtract the mean value for each pair of languages. In both cases, means are computed across all tokenizers. We present Spearman’s correlation coefficient and associated p-value. ar tr zh el es en All metric tokenizer
V. Allocation (AR) Unigram 2129 2719 5919 2070 1439 1513 2042 BPE 2972 3226 4294 2907 2220 2143 2193 NoOverlap 2537 2653 2090 2065 1661 1597 1829 TokMix 3485 4167 3961 2639 1999 1898 2198
V. Allocation (CPT) Unigram 3. 16 4. 01 1. 84 3. 5 3. 88 3. 91 3. 17 BPE 3. 7 4. 19 2. 03 3. 97 4. 34 4. 22 4. 47 NoOverlap 3. 53 4. 19 1. 56 3. 81 4. 15 4. 15 3. 16 TokMix 3. 7 4. 45 1. 73 3. 9 4. 24 4. 18 3. 34
MLM (MRR)
Unigram 36. 0 36. 0 34. 2 46. 3 49. 7 49. 6 42. 0 BPE 28. 7 33. 6 28. 6 38. 6 43. 1 41. 0 35. 6 NoOverlap 38. 1 39. 6 41. 4 42. 8 47. 5 46. 6 42. 7 TokMix 31. 5 30. 6 38. 2 41. 2 45. 3 45. 6 38. 7
NER (F1)
Unigram 66. 4 ±0. 1 73. 0 ±0. 1 35. 1 ±0. 1 68. 0 ±0. 1 68. 0 ±0. 1 66. 1 ±0. 2 62. 8 ±0. 1 BPE 76. 1 ±0. 0 76. 7 ±0. 0 54. 2 ±0. 1 70. 3 ±0. 1 75. 2 ±0. 1 70. 0 ±0. 0 70. 4 ±0. 1 NoOverlap 76. 5 ±0. 1 72. 8 ±0. 0 58. 4 ±0. 1 69. 6 ±0. 1 71. 6 ±0. 1 67. 3 ±0. 1 69. 4 ±0. 1 TokMix 76. 6 ±0. 1 76. 2 ±0. 1 56. 1 ±0. 0 70. 1 ±0. 1 74. 3 ±0. 1 68. 1 ±0. 1 70. 2 ±0. 1
POS (F1)
Unigram 54. 8 ±0. 1 46. 9 ±0. 2 29. 3 ±0. 1 52. 9 ±0. 3 76. 5 ±0. 2 81. 9 ±0. 1 57. 1 ±0. 2 BPE 66. 7 ±0. 1 52. 1 ±0. 1 62. 2 ±0. 0 63. 4 ±0. 1 81. 7 ±0. 4 87. 4 ±0. 1 68. 9 ±0. 2 NoOverlap 66. 5 ±0. 1 52. 5 ±0. 2 60. 6 ±0. 1 67. 5 ±0. 1 81. 3 ±0. 6 86. 7 ±0. 1 69. 2 ±0. 2 TokMix 66. 0 ±0. 1 52. 1 ±0. 2 56. 2 ±0. 0 61. 7 ±0. 2 81. 3 ±0. 2 86. 3 ±0. 1 67. 3 ±0. 1
Dep. labeling (F1) Unigram 13. 5 ±0. 6 58. 6 ±0. 8 20. 7 ±0. 1 58. 4 ±0. 4 71. 9 ±0. 1 65. 7 ±0. 2 48. 1 ±0. 4 BPE 13. 8 ±0. 0 63. 7 ±1. 2 59. 5 ±0. 1 68. 2 ±0. 8 77. 0 ±0. 2 70. 3 ±0. 4 58. 7 ±0. 4 NoOverlap 13. 2 ±0. 0 65. 0 ±0. 5 60. 5 ±0. 2 67. 7 ±0. 2 77. 1 ±0. 3 69. 2 ±0. 3 58. 8 ±0. 3 TokMix 14. 1 ±0. 0 62. 9 ±1. 2 53. 8 ±0. 1 67. 3 ±0. 5 76. 5 ±0. 1 69. 1 ±0. 2 57. 3 ±0. 4
NLI (Acc)
Unigram 52. 5 ±0. 3 52. 9 ±0. 3 47. 5 ±1. 4 55. 0 ±0. 2 55. 3 ±0. 3 57. 4 ±0. 5 53. 4 ±0. 5 BPE 52. 2 ±0. 3 53. 6 ±0. 5 45. 2 ±0. 4 55. 6 ±0. 3 55. 7 ±0. 2 57. 8 ±0. 2 53. 3 ±0. 3 NoOverlap 52. 9 ±0. 7 54. 0 ±0. 2 44. 0 ±0. 8 54. 8 ±0. 1 54. 9 ±0. 3 57. 3 ±0. 3 53. 0 ±0. 4 TokMix 52. 0 ±0. 2 53. 6 ±0. 5 46. 2 ±1. 0 55. 4 ±0. 3 55. 3 ±0. 1 57. 5 ±0. 2 53. 3 ±0. 4
Table 7: Results of evaluation for in-language properties and tasks for six diverse languages. We observe significant changes for different tokenization methods. The results for MRR, POS, NER, XNLI are in percent. For the downstream task, we show average and standard deviations computed for five runs of probing. ACL 2023 Responsible NLP Checklist
A For every submission: 3 A1. Did you describe the limitations of your work?
Yes, section Limitations. 3 A2. Did you discuss any potential risks of your work? Yes. we cannot think of many risks. One of possible risks might lie in under-representing the low-resource languages but actually we propose possible improvements on that. 3 A3. Do the abstract and introduction summarize the paper’s main claims? Yes. Section 1
3 A4. Have you used AI writing assistants when working on this paper? For reformulation of our English text (this does not need to be disclosed) we also report the use of coding assistants in the README we never use long coding assistant suggestions in verbatim and check the outputs closely. B 3 Did you use or create scientific artifacts? > Yes, we use SentencePiece, Huggingface - cited in section 2 and appendix. Datasets used are cited in
section 4. 2. We do not publish the models we trained but we publish the code to reproduce the results along with a metric-computation utility package
3 B1. Did you cite the creators of artifacts you used? Yes, in the same sections as above
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? We do not discuss the license terms in the paper. The libraries we use are licensed under the Apache 2. 0 license which allows the use of the tools for research. The datasets are released for public use. 7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No, We don’t discuss the use of the existing artifacts, nevertheless our use is consistent with their intended use. We will specify the license under which we release our code. B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. We use existing, publicly released datasets. We therefore assume that these steps were already taken. 3 B5. Did you provide documentation of the artifacts, e. g. , coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc. ? Yes, Section 4. 2. 1
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Yes, Section 4. 2
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance. C 3 Did you run computational experiments? Yes, Section 5
3 C1. Did you report the number of parameters in the models used, the total computational budget (e. g. , GPU hours), and computing infrastructure used? Yes, Appendix A. 1
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Yes, Appendix A. 1
3 C3. Did you report descriptive statistics about your results (e. g. , error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Yes, Section 5
3 C4. If you used existing packages (e. g. , for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e. g. , NLTK, Spacy, ROUGE, etc. )? Yes, Appendix A. 1
D 7 Did you use human annotators (e. g. , crowdworkers) or research with human participants? Left blank. D1. Did you report the full text of instructions given to participants, including e. g. , screenshots, disclaimers of any risks to participants or annotators, etc. ? No response. D2. Did you report information about how you recruited (e. g. , crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e. g. , country of residence)? No response. D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response. D4