XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Models
Authors: Davis Liang
Hila Gonen
Yuning Mao
Rui Hou
Naman Goyal
Marjan Ghazvininejad
Luke Zettlemoyer
Madian Khabsa
Venue: EMNLP 2023 Main (2023)
Keywords: Multilingual
Masked Language Models
TL;DR: We introduce a new approach for scaling to large multilingual vocabularies by de-emphasizing token sharing between languages with little lexical overlap and assigning vocabulary capacity to achieve sufficient coverage for each individual language.

Abstract: Large multilingual language models typically rely on a single vocabulary shared across 100+ languages. As these models have increased in parameter count and depth, vocabulary size has remained largely unchanged. This \textit{vocabulary bottleneck} limits the representational capabilities of multilingual models like XLM-R. In this paper, we introduce a new approach for scaling to very large multilingual vocabularies by de-emphasizing token sharing between languages with little lexical overlap and assigning vocabulary capacity to achieve sufficient coverage for each individual language. Tokenizations using our vocabulary are typically more semantically meaningful and shorter compared to XLM-R. Leveraging this improved vocabulary, we train XLM-V, a multilingual language model with a one million token vocabulary. XLM-V outperforms XLM-R on every task we tested on ranging from natural language inference (XNLI), question answering (MLQA, XQuAD, TyDiQA), to named entity recognition (WikiAnn). XLM-V is particularly effective on low-resource language tasks and outperforms XLM-R by 11.2\% and 5.8\% absolute on MasakhaNER and Americas NLI, respectively.

XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Models
Davis Liang
Hila Gonen
Yuning Mao
Rui Hou
Naman Goyal
Marjan Ghazvininejad
Luke Zettlemoyer
Madian Khabsa
Meta Al
Abstract
2,500 unique tokens per language, calling into ques- tion the vocabulary's s ability represent the diverse selection of languages that it was intended to model. For example, there are 8, 8,105 characters in the Ta- ble of General Standard Chinese characters and over 100,000 unique characters in total; the num- ber of commonly used Chinese words (consisting of multiple characters) is even larger (Wikipedia, 2023). In fact, prior work has already shown that this vocabulary bottleneck hinders the performance of multilingual models on question answering and sequence labeling where in-depth token-level and sequence-level understanding is essential (Wang et al., 2019).
Large multilingual language models typically rely on a single vocabulary shared across 100+ languages. As these models have increased in parameter count and depth, vocabulary size has remained largely unchanged. This vocabulary bottleneck limits the representational capabil- ities of multilingual models like XLM-R. In this paper, introduce new approach for scaling to very large multilingual vocabular- ies by de-emphasizing token sharing between languages with little lexical overlap and assign- ing vocabulary capacity to achieve sufficient coverage for each individual language. Tok- enizations using vocabulary typically more semantically meaningful and shorter com- pared XLM-R. Leveraging this improved vocabulary, we train XLM-V, multilingual language model with a one million token vo- cabulary. XLM-V outperforms XLM-R ery task we tested on ranging from natural lan- guage inference (XNL.), question answering (MLQA, XQuAD, TyDiQA), named entity recognition (WikiAnn). XLM-V is particularly effective on low-resource language tasks and outperforms XLM-R by 1 11.2% and 5.8% ab- solute on MasakhaNER and Americas NLI, re- spectively.
In this paper, we construct a large multilingual vocabulary by attending to two core principles: (1) vocabularies be improved by de-emphasizing token sharing between languages with little lex- ical overlap and (2) proper vocabulary capacity allocation for individual languages is crucial for en- suring that diverse languages are well-represented. Then, we show that our new vocabulary exhibits favorable characteristics including the ability to fre- quently output semantically meaningful tokeniza- tions while reducing ver-tokenizztion for low- resource languages. Finally, present XLM-V, the first multilingual language model with one million token vocabulary trained on 2.5TB of data from Common Crawl (Conneau et al., 2019).
1 Introduction
While multilingual language models have increased in parameter count and depth over time, vocabu- lary size has largely remained unchanged: mBART (680M parameters; Liu et al. 2020), XGLM (7.5B
Our main contributions are as follows:
In Section 3, we present our method for constructing large multilingual vocabularies. Specifically, improve upon the language clustering algorithm from Chung al. (2020) by constructing better vector representations for individual languages and leverage Zheng et al. (2021) to improve the vocabulary capac- ity assignments for each cluster.
parameters, Lin et al. 20 XLM-R XXL (10.7B
parameters; Goyal et al. 2021). mT5 XXL (13B parameters; Xue et al. 2020); and BLOOM (176B parameters; Scao et al. 2022) all share the same 250K token vocabulary size as XLM-R base (Con- neau al., 2019), 250M parameter model. For models like mT5 and XLM-R, this 250K vocabulary is shared across 100+ languages. Dis- counting shared tokens, this results in an average of
In Section 5, we demonstrate that XLM-V outperforms comparable baselines that have

---

the same vocabulary size on XNLI. Addition- ally, XLM-V outperforms XLM-R on every multilinguul language understanding task we tested on (including XNLI, WikiAnn, MLQA, XQuAD, and TyDiQA) by an average of 3.5 points absolute. XLM-V performs especially well on low-resource evaluation datasets like AmericasNLI and MasakhaNER, outperform- ing XLM-R by 5.8% absolute accuracy and 11.2% absolute FI, respectively.
present in the vocabulary of I. An illustration of this step is shown in Figure Then, the authors cluster the binary vectors to group lexically similar languages together. Finally, they construct a vocab- ulary for each cluster and combine the per-cluster vocabularies together to form a unified multilingual vocabulary.
2.3 Vocabulary allocation
Zheng et al. (2021) proposed the average log proba- bility (ALP) to evaluate the ability of a vocabulary to represent a particular language. Specifically, given a monolingual corpus composed of sentences D; = {$1: 8 s(D,1) from the i-th language and tokenized with vocabulary V, the average log prob- ability is defined as;
Finally, in Section 6, we provide examples and quantitative analysis to compare our new vocabulary to various baselines. Most notably, provide evidence showing that expanding the vocabulary beyond 1M tokens can degrade performance on downstream tasks.
D2 ΣΣ Σ
2 Background
VL [D.
log Puns (5)) (1)
2.1
Sentencepiece
The Unigram Language Model (ULM) from Kudo and Richardson (2018) is a popular subword seg- mentation algorithm used to construct vocabularies. ULM begins with large initial vocabulary that is iteratively pruned to maximize the likelihood of the training corpus (under unigram language model of the tokens) until the number of tokens falls be- low some pre-determined vocabulary size thresh- old, [V]. During tokenization, ULM decodes the most probable segmentation of a sequence through the Viterbi algorithm (Viterbi, 1967). This method is used by both XLM-R and our work.
where st. is the k-th subword of the sentence sj and Puns(.) (.) is the unigram distribution counted on the monolingual corpus D, The authors first show that ALP is highly correlated with downstream task performance and then propose greedy algorithm to determine the desired vocabulary capacity for in- dividual languages in the multilingual vocabulary.
Methodology
3.1 Building the vocabulary
In this subsection, we describe our method for con- structing multilingual vocabularies. At high level, we (1) train individual monolingual sentencepiece models (SPM) for each language in our dataset using the Unigram Language Model (ULM) algo- rithm (Kudo and Richardson, 2018), (2) use the per-language vocabularies to construct lexical rep- resentation vectors for each language, (3) cluster the lexical representation vectors using K-Means, assign vocabulary capacities for each cluster using the ALP, and then construct per-cluster vocabular- ies using the ULM algorithm, and (4) create the final multilingual vocabulary by taking the union of the vocabularies for each cluster.
2.2 Clustering
Chung et al. (2020) proposed an approach to multi- lingual vocabulary construction that balances the trade-off between optimizing for cross-lingual sub- word sharing and the need for robust representation of individual languages.
Their procedure for building multilingual vo- cabulary contains several steps. First, the authors train individual sentencepiece models for each lan- guage: for each language I in the set of languages L, a vocabulary V is generated, Then, they cre- ate the shared lexicon V4 by taking the union of each sanguag vocabulary, VL Uter ! Next, for each language 1, they construct binary vector of of dimension [V which represents the lexicon of 1. Each component of of corresponds to a subword in V other words, the binary vec- tor of contains a I corresponding to each subword
Training monolingual SPMs To acquire the data for building the vocabulary, we perform sampling with temperature
2 to sample 1 billion lines of text from CC100 (up-sampling lesercresource and bown-sampling data from high resource lan- guages). Then, for each language in CC100, we

---

Cluster
[V
Languages
C1
174,504
Ga, pu, si, kas ur. lo, my, ne, am, te, my, lh,, in, ko, bn, ml. he, sd. as, hi, km. gu. kn, sú, yi, me, ps, or, ah, or, ug
102,722
1R, 2b-TW, Zh-CN
43
186,881
fi, sk, oen, sw, az, lp, uee so, by, ss, Iu, la. HT, ot ta, wo, H9, to, se, ol. pl,
110,148
pt, eu, gl. gn, it, ca, es
19,801
hir, sl, be
101,485
he, kk, mi.
279,702
su, jN, so, in, no. id. ig. bu. ns, mg, es, mss To, ur, rin hu, gu, is, eu, gd. br. hi, en, ey. ft, vi, da, yo, de,
Table Lexical clustering results for XLM-V with number of clusters is = 8 and a total vocabulary capacity of 1M.
train language-speciffs sentencepiece model with a vocabulary size of 30,000 (per language) using this data.
method for assigning vocabulary capacities used by Chung et al. (2020) (i.e. proportionally to the union of the per-language vocabularies in each cluster) resulted in several clusters with deficient vocabulary capacity. For example, cluster e2 Table 1 (a smaller cluster that contains lcxically diverse languages: Chinese Simplified, Chinese Traditional. and Japanese), was assigned a capacity of just 28,593 tokens.
(the, of, land, le, 103
Generate vodab eech Language
Enoode
en [they of, and]
(1,
1,
0,
ft Lle, la,
(C,
1, 1. 19
Chung
We instead use the per-language vocabulary ca- pacity allocations (Zheng al., 2021) optimized for the CC100 dataset. By doing so, the vocabulary capacity assigned 02 was increased 102,722. For each tail-end (low-resource) language that was not covered in Zheng et al. (2021), we allocate a 2,000 token vocabulary budget. Rather than use the vocabulary allocations directly, we take their relative values and rescale them to sum up to the vocabulary capacity of our choosing (e.g. 1M, 2M, etc.). Finally, perform K-Means clustering with 8, based on experiments from Chung et al. (2020) showing that 8 results the best per- formance on downstream tasks. We expect the ideal number of clusters to vary not based on the number of languages but rather on the identity of those languages and their respective similarities to one another.
Unlon
Generate vooab for each language
(they ot,
and,
Le,
1a3
Enced:
en [the, of, and)
01
7.4, 8.5.
8.71
Ouns
Figure I: Similar to Chung et al. (2020), also lever- age the per-language sentencepiece vocabularies as "lexical fingerprint" for clustering. However, instead of using binary vectors, the unigram log probability instead.
Constructing lexical fingerprints We then con- struct a vector representation of each language us- ing the vocabularies of each language as shown in Figure 1. Unlike Chung et al. (2020), where language is represented by binary vector contain- ing a corresponding each subword present in the vocabulary of that language, we instead use the negative log probability that each token appears the respective language's monolingual corpus. We hypothesize that weighting each token by its likelihood occurring better represents the lexical fingerprint of language.
The final vocabulary For each resulting cluster, we train per-clusted sentencepiece models and com- bine the vocabularies each cluster into a single multilingual vocabulary. The final vocabulary con- sists of 901,629 tokens (remaining 98,371 tokens overlapped between the clusters), meaning that average over 90% of the tokens learned in each cluster unique.
Clustering and capacity allocation Next, we construct language clusters and train sentencepiece models for each cluster in order to discourage the vocabulary sharing between lexically dissimilar languages. Before training per-cluster sentence- piece models, we need to first decide on the vo- cabulary size, or vocabulary capacity, to allocate to each cluster. Unfortunately, we found that the
3.2 Training the model
To pretrain our model, follow the same train- ing procedure from XLM-R (Conneau al., 2019). Specifically, we use the CC100 dataset with sam- pling temperature 0.3 to increase the amount of low- and resourrresource resource language examples

---

seen during training. We use the Adam optimizer (Kingma and Ba. 2014) with the default (81, 32) and parameters of (0.9, 0.98) and le-6, respec- tively. We use a learning rate of 6e-4, a warmup of 15,000 steps, a batch size of 8, 1,192 distributed across 256 A100 GPUs, and train for total of 1.5M iterations. Each batch consists of examples concatenated up to the maximum sequence length of 512. We pretrain the model using the Masked Language Model (MLM) task (Devlin et al., 2018) with the standard masking rate of 15%.
Chung et al. (2020) (1M) We create 1M token vocabulary using the lexical clustering approach from Chung et al. 2020 as described in Section 2.
4.2 Datasets
CC100 (Conneau et al.. 2019) is a multilingual corpus created from one Common Crawl dump for English and twelve dumps for all other languages. The resulting corpus contains 2.5 TB of data split between 16 languages. We use this dataset exclu- sively for constructing vocabularies and pretraining our models.
Increasing the vocabulary size can significantly increase pretraining time due to the computation- ally intensive softmax layer. To address this, prior works have leveraged approximation tricks such as adaptive softmax (Baevski and Auli, 2018) and adaptive inputs (Joulin al., 2017). However, we found that these tricks require non-trivial amounts of tuning and resulted in slower convergence and increased training instability. In this paper, we per- form pretraining without any approximation tricks noting that this method may not be feasible when the vocabulary is scaled beyond 2M.'
FooRes-2000 (Goyal et al., 2022) is an evalua- tion corpus consisting of 3,001 sentences extracted from 842 English Wikipedia articles and covering variety of different topics and domains. These sentences have been translated into 200 languages by professional translators through a carefully con- trolled process.
XNLI (Conneau et al., 2018) asks whether a premise sentence entails, contradicts, or is neu- tral toward hypothesis sentence. Crowd-sourced English data is translated to 10 other languages by professional human translators and used for evalua- tion, while the Multi-Genre Natural Language In- ference Corpus (MultiNLI) (Williams et al., 2018) data is used for training.
4 Experiment setup
4.1 Baselines
Aside from training XLM-V, we also construct sev- eral baselines to compare our model against. To construct our baselines, we first create the respec- tive vocabularies and then pretrain transformer en- coders (12-layers, equivalent to XLM-R basc) us- ing these vocabularies. For the rest of the paper, we will use the following names to refer to the vocabulary and the model interchangeably.
MIQA (Lewis et al., 2019) is QA evalua- tion dataset created by mining target language sen- tences that are parallel sentences in English from Wikipedia, crowd- -ourcing annotations in English, and translating the question and aligning the answer spans in one of the 6 target languages. It consists of over 12K QA instances in English and 5K in each other language. The training set of MLQA is SQuAD v1.1 (Rajpurkar et al., 2016).
XLM-R (250K) The XLM-R vocabulary is cre- ated using the same procedure from (Conneau et al., 2019) by applying the ULM algorithm described in Section 2 on a corpus of IB lines of text sampled from CC100. The result is a multilingual vocabu- lary with 250,002 tokens. For our experiments, we simply re-use the publicly available XLM-R sen- tencepiece model and pretrained model checkpoint from fairseq (Ott al., 2019).
XQuAD (Artetxe et al., 2019) translates the dev set of SQuAD v1.1 into 10 other languages through professional translators. The resulting dataset is used for evaluation. The training set of XQuAD is SQuAD vl.1.
TyDiQA-GoldP (Clark et al., 2020) is question answering (QA) dataset covering 11 typologically diverse languages with 200K QA pairs. Questions
XLM-R (1M) We construct a IM token vocab- ulary by following the same approach as XLM-R (250K) with an increased vocabulary capacity.
'For the question answering tasks, instead of validating the selected spans after retrieving the n-best answers, we pro- pose to only reirieve n-bext answer spans that are valid (e.g. span start and end indices are part of the passage context). This change improves QA performance for both the baseline models and XLM-V.
For a model with a vocabulary size of IM, each iteration of ML.M pretraining took 2.5 times longer than the same model with a 250K token vocabulary.

---

Model
XNLI
NER
MLQA EM/FI
TyDiQA EM/FI
XQuAD EM/FI
ANLI
MNER Average
Acc.
Acc.
F1
XLM
69.1
32.6/48.5
29.1/43.6
54.3/
XLM-R
76.2
46.3 63.7
38.5
XLM-R reimpl.
74.9
61.3
46.7 / 64.4
38.3/56.0 39.7 56.9
56.0 71.3
39.6
20.9
55.5
XLM-V
76.0
64.7
47.7 / 66.0
56.3 71.9
45.4
32.1
59.0
Table 2: Overall results across multiple multilingual datasets comparing model against the XLM and XLM-R baselines. All results are based on crosslingual transfer after fine-tuming on English data. We computed the average result using the accuracy or F1 of each task. "reinpl" is our re-implementation finetuning, used by both XLM-R and XLM-V. Please refer the appendix for specific hyperparameters reproduce each result. EM stands for exact match. ANLI refers to AmericasNLI and MNER refers to MasakhaNER.
in TyDiQA are written without seeing the answers leading significantly less lexical overlap than XQuAD or MLQA. The languages of TyDiQA are selected to be diverse with regard to their typology. We use the gold passage version of the Typologi- cally Diverse Question Answering dataset.
Results
5.1 Comparisons using partial training
We first perform a study to measure the impact of our new vocabulary on downstream performance. Specifically, we pretrain 12-layer transformer en- coder model using Masked Language Modeling on the CC100 corpus for each baseline as well for our proposed method. Because pretraining is expensive, we limit the batch size to 2,048 and the number of total steps to 300,000 for these experi- ments. The results Figure show that our model outperforms all baselines on XNLI including XLM- R (IM) by 1.34% and Chung et al. (2020) by 1.1 1.1% absolute accuracy.
NER (Pan al., 2017) consists of 48 languages and based on the WikiAnn (PAN-X) dataset. Named entities were automatically annotated with LOC, PER, and ORG tags through knowledge base properties, crosslingual and anchor links, self- training, and data selection. Similar to (Hu al., 2020), use the balanced dev and test splits from Rahimi et al. (2019).
Comparing Sentenceplece Models @300K iterations
73.5
73.12
Americas NLI (Ebrahimi et al., 2021) is an ex- tension of XNLI to 10 indigenous languages of the Americas constructed by translating subset of XNLI using human translators. These languages contain interesting linguistic features such as a rich system of applicative suffixes (Asháninka), direc- tional verbs (Bribri), and nominal incorporation (Wixarika). Presently, these languages are written, spoken, and used in an official capacity by tens of thousands to several million people in Central and Southern America. The training set of Americas NLI is MultiNLI.
73.0
72.72
XNLI Accuracy (dev)
72.5
72.0
71.78
71.5
71.0
70.5
70.13
70.0
XLM-R
XLM-R Chung
XLMV (1.5M)
XLM-V (1M)
2020 (1
Sentencapiace
Figure 2: We compare the performance of the same model trained with different sentencepiece vocabularies. The models are all trained for 300K iterations with batch size of 2,048 on the CC100 corpus.
MasakhaNER (Adelani et al., 2021) is the first large, publicly available, and high-quality dataset for named entity recognition (NER) in ten African languages including Amharic, Hausa, Igbo, and others. The languages covered in this dataset have varied seripts and range from 4M to 98M speakers in regions across East, West, Central, and North- west Africa.
5.2 Fully trained model
We evaluate an XLM-V (1M) model, trained on CC100 for 1.5M iterations with a batch size of 8,192, on several tasks including natural language inference (XNLI), question answering (MLQA,

---

Model
fr
de
bg
tr
vi
th
zh
hi
ur
AVG
Finetune multilingual model on English training set (Cross-lingual Transfer)
XLM-R reimpl. 85.4
78.5 79.1 77.7 76.1 78.1 76.3 73.9 72.3 75.6 73.0 74.9
70.5
65.8
66.5 74.9
XLM-V
85.6
79.6 79.5 78.4 76.9 79.6 76.6 74.0 73.1 76.2 73.0 75.1 72.0
70.5
69.4
76.0
Finetune multilingual model on all training sets (Translate- Train-All)
XLM-R reimpl.
85.4
81.5
82.0 80.7 80.2 81.2 78.9 78.4 77.6 79.9 77.6 79.5
75.8
73.4 72.3
79.0
XLM-V
85.6
81.5 82,1 81.5 80.7 81.5 79.6 78.7 77.6 80.0 77.7 79.5 77.0
74.3 73.9 79.4
Table 3: XLM-V outperforms XLM-R cross- lingual transfer every language in XNLI with outsized improve- ments on the lower-resource languages, Swahili and Urdu. We observe similar improvements on translate-train-at The model trained for 12 epochs epochs for translate-train-all) on 8 A100 GPUs with floatl6 precision. We use a learning rate of 7.5e-6 with max sequence length of 256, a batch size of 16, no weight decay, and no warmup.
Model
aym
bzd
cni
hch
nah
oto
shp
tar
AVG
XLM-R reimpl.
36.6
39.6
40.5
41.6
38.8
40.2
39.4
38.7
42.7
37.6
39.6
XLM-V
39.9
41,5
41.7
58.8
40.7
44.7
42.1
56.9
46.5
41.2
45,4
Tok. Length (rel.)
-108%%
-1166%
-11.9%
-1.5%
-6.5%
-10.7%
-8.4%
-18.4%
10.9%
-9.1%
-11.5%
Table 4: We show the zero-shot cross-lingual transfer results Americas NLI (trained on English and evaluated the unseen languages). Our model, XLM-V, outperforms XLM-R by wide margin with outsized improvements on Quechua and Guaraní. Tok. Length (rel.) refers to the relative difference in the average number of tokens (ootttookenization) between XLM-R and XLM-V. XLM-V consistently outputs shorter sequences post-tokenization. The model is trained for 12 epochs on 8 A 100 GPUs with f precision. We use a learning rate of 7.5e-6 with a max sequence length of 256, batch size of 16, no weight decay, and no warmup.
TyDiQA, and XQuAD), named enitity recogni- tion (WikiAnn), and low resource language tasks (AmericasNLI, MasakhaNER). All tasks leverage crosslingual transfer from English-only finetuning and trained using 6 precision with the AdamW optimizer (Loshchilov and Hutter, 2017). We use hyperparameters selected based on the best English performance on the dev set, and finally evaluate on the test set. We compile all of our re- sults in Table for XLM-V and XLM-R. We also include results for XLM (Lample and Conneau, 2019) for additional context.
racy improvement on Swahili (sw) and Urdu (ur) on XNLI. Similarly, we show an average gain of 11.2% F1 on MasakhaNER, a low-resource African language NER dataset.
In Table 4 we show that XLM-V not only con- sistently outperforms XLM-R on Americas NLI in zero-shot crosslingual transfer but is able to ob- tain 18.2% absolute F1 improvement Quechua (quy) and 17.2% absolute improvement on Guaraní (gn). Interestingly, Quechua and Guaraní also the two languages with the largest relative drop in average token count per sentence suggesting that these languages are over-tokenized by XLM-R.
Table shows that XLM-V outperforms our re- implementation of XLM-R on all datasets by an average of 3.5 points absolute (we compute the av- erage result using either the accuracy Fl of each task). In Table 3, we show that XLM-V outper- forms XLM-R on all languages in cross-lingual transfer (training
6 Analysis
6.1 The Zipf ceiling
We explored training models with vocabulary sizes greater than 1M tokens but found that these models perform comparatively worse on downstream tasks. We visualize the diminishing utility of increasing the vocabulary size in Figure 3. Specifically, we create vocabularies with 500K, 1M, 1.5M, and 2M tokens using our methodology. Then, we use these vocabularies tokenize the FLoRes-200 dataset. For vocabulary sizes of 500K, IM, and 2M, we find that 99% of the content is covered by just 140,337, 197,817, and 243,832 unique tokens, respectively.
English and evaluating on other languages) with similar improvements on translate-train train-all (finetuning the model on both the English and translated training sets). In particu- lar, we find that XLM-V consistently outperforms XLM-R on low-resource languages. For exam- ple, in Table 3, we observe 4.7% and 2.9% accu-
'For tasks trained on MNLI we follow Conncau et al. (2019) and select the checkpoint with the best average perfor- mance across all languages.

---

Language
Tokenizer
Tokenized Output
Original Sentence
XLMR (250K)
rar. B.. "XP 2n 40 "f, 16244 rar, ralea. * SPP
XLM R (IM)
*F7, 'BE2". 1977
Chung al. (2020) (IN)
XLM-V (IM)
ral "48E. A9, WAEE
Original Sentence
nareolepsy murcolepsie nacolepsia
XI M-M (250K)
I'na', ir, ''oole, "pay'] I'na", 'r", 'cole,, "pai", 'e'l I'na", 'oole', ppar,, I'ma', rrole'. pay'l ['ma,, 'roole'. "psie" I'naa. 'roole", "psial] I'ma'. 'rolle. "pay'] ["nareo", "lepsi. e'If'ma. 'reale". "psia'l l'muco'. "lo". 'pay I'naroo", 'lepai" 'e'] ["naroo 'leps"" 'a'l
en, es
XLM R (IM)
Chung al. (2020) (IM)
XLM-V (IM)
Original Sentence
XLM (250K)
['Be'. "W",, 'ubb. 'ungs", "miteel, 'ver', 'schreibung'. 's. 'ver', "ordnung'1
XLM-R (1M)
['Be', "B8", "ub", "ungsmitte"" "ver', "schreibung" 's', "verordhung]
Ching al. (2020) (IM)
['Ber', 'aub', 'ungsmirtel", 'ver', 'schreibung", sverodninngg ['Bet. 'jub', "umgsamitee"" 'ver.. 'schreibung' 'sverorduung'
XLM-V
Table 5: We provide examples comparing tokenization using the XLM-V vocabulary against baselines. We find that our sentencepiece model reduces overtokenization and can be surprisingly good at splitting sentences into pscudo-meaningee segments out-of-the-box.
Model
vi
zh
de
en
xho
tel
AVG
XLM-R (250K)
34.3
28.5 37.5 33,9 29.1 43.9 38.8 43.6
XLM-R (IM)
33.5
31.7
34.8 31
26.8 40.3
41.6
41.4
Chung et al. (2020) (IM)
32.7
24.4
32.9
29.1
27.8
29.2
25.7
37.7
XLM-V (IM)
32.4
23.4
32.2
28.3
25.5
37.4
33.2 38.6
Table 6: Average number of tokens after tokenization on the FLoRes- dataset for several high, medium, and low resource languages. AVG denotes the average tokenized lengths per sentence across all 200 languages in Flores-200.
Token Utilization for Different Vocabulary Sizes (XLM-V)
token embeddings are problematic because they are trained significantly less data during the course of MLM pretraining and will Iearn sub-optimal rep- resentations as result. As a consequence, vocab- ularies past a certain size will cease to improve model performance and can potentially degrade it. A clear example of this is shown in Figure 2 where our model with 1M token vocabulary out- performs its 1.5M token counterpart trained using equivalent amount of data.
2M Teken Vocab
12
1.5M Vocab
IM Taken Vocab
10
500k Taken Vocab
Log Frquuency
8-
6-
4-
2-
6.2 Qualitative improvements in tokenization
0-
Table 5 shows a few tokenized examples from Chinese (zh), English (en), French (fr), Span- ish (es), and German (de). For languages in the same cluster (en, fr, es), our method can sep- arate shared roots (e.g. narco) from the same word in different languages. Notably, our method demonstrates a surprising ability to segment Chi- nese out-of-the-bon parsing out individual entities in the original phrase. For example, the XLM- V tokenizer is able to mcaningfully break down the phrase A # 0F trans- lated as Cambridge University undergraduates and postgraduates. Specifically, the output of the XLM-V tokenizer is sity), $ # (undergraduut))))))) FIl(and), and OF
50000 100000 150000 200000 250000 300000 Taken Index (sorted by frequency)
Figure 3: We compare the token utilization of each sentencepiece vocabulary on the FLoRes-200 200 dataset. We diminishing returns as the size of the vocabulary is increased beyond 1M tokens.
We hypothesize that since the Unigram LM (Kudo and Richardson, 2018) algorithm used to construct the vocabulary iteratively prunes large initial set, as discussed in Section 2, further expand- ing the vocabulary is equivalent to inheriting tokens from the long tail of a Zipfian distribution. These
Univer-

---

([postgraduates) Qualitatively, our tokenizer frequently performs tokenizations that are se- mantically meaningful, one possible contributor to the improved downstream performance.
increase the number of trainable parameters by 3.3x with just 25% increase in training time.
7 Related work
7.1 Vocabulary-free models
6.3 Over-tokenization
In recent years, vocabulary-free models like ByT4 (Xue et al., 2022) and CANINE (Clark al., 2022) have demonstrated on-par or better perfor- mance compared their subword tokenization- based counterparts. However, one consistent draw- back of these models is slower training and infer- ence speed. For example, ByT5 is 6.4 to 9.5 times slower than mT5 (Xue et al., 2020) on classifica- tion tasks like XNLI. CANINE fares better, leverag- ing optimizations like lower input character dimen- sions and heavy down sampling, but still remains approximately 1.6 times slower than a comparable BERT baseline. On the other hand, simply using a larger sentencepiece vocabulary can improve down- stream performance, increase the capacity of the model, and reduce the over-tokenization and cov- erage of low-resource languages all with smaller impact on inference latency. We believe that both directions are useful areas of research and can be explored simultaneously.
Representing input data with fewer tokens can speed up inference, allow the model make use of longer context, and help with over-tokenization for low-resource languages (Rust et al., 2020). Ta- ble shows the average number of resulting to- kens (post-tokenization) for several languages in 200Res-200. On average, the XLM-V tokenizer returns fewer tokens for high and medium resource languages while Chung et al. (2020) returns the fewest tokens for low-resource languages. Overall, XLM-V returns 11.5% fewer tokens compared the baseline XLM-R tokenizer, meaning that input sequences are on average 11.5% % shorter.
Training Speed vs. Vocab Size
0.30
ti per iteration [in secondal
0.25
5M
0.20
500K
0.15
0.10
7.2 Building larger vocabularies
0.05
Prior work on vocabulary expansion (Wang et al., 2019) sought to augment the vocabulary of exist- ing models to address out-of-vocabulle problems in multilingual settings. While these re- sults are potentially useful in augmenting subword models like BERT, sentencepiece models by nature encounter significantly fewer OOVs.
0.00
200
400
800
1000
1200
1400
number parameters (in millions)
(OOV)
Figure 4: We track training speed vocabulary size using a typical training setup on XNLI: one A100 GPU, batch size 16, sequence length of 128, and float16 precision. The text above each point denotes the vocab- ulary size.
More recent work on building larger vocabular- ies (Chung et al., 2020; Zheng et al., 2021) leverage tricks like lexical clustering and more principled methods for vocabulary allocation have tackled is- sues with over-tokenization and vocabulary cover- age for low-resource languages. While compelling, these works are unfortunately limited by data (the models are trained on Wikipedia, a relatively small pretraining corpus) and scale (the largest vocab- ulary explored was 500K, only twice the size of the vocabulary in XLM-R). As such, the result- ing models significantly under-perform the pub- lic XLM-R baseline. Our work seeks to combine and improve upon existing methods for building large-scale vocabularies, pretrain with substantially bigger datasets, and explore vocabularies of 1M tokens and beyond.
6.4 Speed vs. size
For XLM-R, which has vocabulary size of 250K tokens, the vocabulary embedding matrix contains 77% of the model's trainable parameters. For XLM- V, the IM token vocabulary accounts for 93% of the model's trainable parameters. While scaling the vocabulary can markedly increase the number of trainable parameters in a model, we can treat it efficient form of conditional compute (Ben- gio al., 2015): only small fraction of the em- bedding matrix is used for any given input. We illustrate the relationship between the vocabulary size and training speed in Figure 4. By increasing the vocabulary from 250K to 1M tokens, we can

---

8 Conclusion
representations for neural language modeling. arXiv preprint 1XXiv:809.1185..
In this paper, we presented XLM-V, a multilingual language model with a IM token vocabulary. We showed that our model outperforms XLM-R, has outsized gains on tasks in low-resource languages, results in semantically meaningful tokenizations, reduces average sequence length, and serves as an efficient form of conditional compute. In the fu- ture, we would like to further investigate the Zipf ceiling discussed in Section 6 by increasing the vo- cabulary beyond 2M tokens while also using more data. Another possible direction for future work is explore larger multilingual vocabularies for autoregressive language models. Finally, further exploration with different clustering methods such as hierarchical clustering may prove both interest- ing and effective.
Emmanuel Bengio, Pierre-Luc Bacon, Joelle Pineau, and Doina Precup. 2015. Conditional computation in neural networks for faster models. arXiv preprint 111.v:1511.06297.
Hyung Won Chung, Dan Garrette, Kiat Chuan Tan, and Jason Riesa. 2020. Improving multilingual models with language-clustered vocabularies. EMNLP.
Jonathan H Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. Tydi qa: A benchmark for information-seekine question answering in typo- logically diverse languages. Transactions of the As- sociation for Computational Linguistics, 8:454-470.
Jonathan H Clark, Dan Garrette, Tulia Turc, and John Wieting. 2022. Canine: Pre-training an efficient tokenization-fres encoder for language representa- tion. Transactions of the Association for Computa- tional Linguistics, 10:73-91.
Limitations
While the strengths of XLM-V clear, there re- mains several scalability issues that are notable. First, while scaling the vocabulary is an efficient form of conditional compute, it can result in in- creased pre-training times due to the computational complexity of the softmax over the entire vocab- ulary. We believe these issues can be solved by adopting approximation techniques like adaptive softmax (Joulin et al., 2017) and adaptive inputs (Baevski and Auli, 2018). Additionally, scaling the vocabulary can also significantly increase the memory footprint of a model. However, we believe memory-related issues become less a problem as we begin to work with larger models, where the number of non-embedding parameters vastly outweigh the size of the vocabulary embedding matrix.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2019, Unsupervised cross-lingual representation learning at scale, arXiv preprint 1oXx:2991100211
Alexis Conneau, Guillaume Lample, Ruty Rinott, Ad- ina Williams, Samuel R Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. Xnli: Evaluating cross- lingual sentence representations. arXiv preprint
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing, arXiv preprint arXiv: 10X.:v88110048800 04800.
Abteen Ebrahimi, Manuel Mager, Arturo Oncevay, Vishrav Chaudhary, Luis Chiruzzo, Angela Fan, John Ortega, Ricardo Ramos, Annette Rios, Ivan Vladimir, et al. 2021. Americasnli: Evaluating zero-shot nat- ural language understanding of pretrained multilin- gual models in truly low-resource languages. arXiv preprint arXiv:2104.08720
References
David Ifeoluwa Adelani, Jade Abbott, Graham Neubig, Daniel D' souza, Julia Kreutzer, Constantine Lignos, Chester Palen-Michel, Happy Buzaaba, Shruti Rijh- wani, Sebastian Ruder, et al. 2021. Masakhaner: named entity recognition for african languages. Transactions of the Association for Computational Linguistics, 9:1116-1131.
Naman Goyal, Jingfei Du, Myle Ott, Giri Ananthara- man, and Alexis Conneau. 2021. Larger-scale trans- formers for multilingual masked language modeling. arXiv preprint 2 200200000002
Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng- Jen Chen, Guillaume Wenzek, Da Ju. Sanjana Kr- ishnan, Mare' Aurelio Ranzato. Francisco Guzman. and Angela Fan. 2022. The flores- evaluation benchmark for low-resource and multilingual ma- chine translation. Transactions of the Association for Computational Linguistics, 10:522-538.
Mikel Artetxe, Sebastian Ruder, and Dani Yo- gatama. 2019. On the cross lonsulinggaa transferabil- ity of monolingual representations. arXiv preprint arXiv: 11856.
Alexei Baeyski and Michael Auli. 2018. Adaptive input

---

Junjie Hu, Sebastian Ruder, Aditya Siddhant, Gra- ham Neubig, Orhan Firat, and Melvin Johnson. 2020. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisa- tion. In International Conference on Machine Learn- ing, pages 441 1-441. PMLR.
Phillip Rust, Jonas Pfeiffer, Ivan Vulić. Sebastian Ruder, and Iryna Gurevych. 2020. How good is your tokenizer? on the monolingual performance of multilingual language models. arXiv preprint arXiv:2012.1500 15613.
Teven Le Scao, Angela Fan, Christopher Akiki, El- lie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni. François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b- parameter open-access multilingual language model. arXiv preprint arXiv:221
Armand Joulin, Moustapha Cissé, David Grangier, Hervé Jégou, al. 2017. Efficient softmax approx- imation for gpus. In International conference on machine learning, pages 1302-1310. PMLR.
Diederik P Kingma and Jimmy Ba. 2014. Adam: method for stochastic optimization. arXiv preprint urXiv:141..6980.
Andrew Viterbi. 1967. Error bounds for convolutional codes and an asymptotically optimum decoding al- gorithm. IEEE transactions on Information Theory, 13(2):260-269.
Taku Kudo and John Richardson. 2018. Sentencepiece: simple and language independent subword tok- enizer and detokenizer for neural text processing. arXiv preprint arXiv: 1808.0626..
Hai Wang. Dian Yu, Kai Sun, Janshu Chen, and Dong Yu. 2019. Improving pre-trained multilingual models with vocabulary expansion. arXiv preprint 1xXiv:999.11244
Guillaume Lample and Alexis Conneau. 2019. Cross- lingual language model pretraining, arXiv preprint ar..9:190..0729.
Wikipedia. 2023. Table general standard chinese characters - wikipedia, the free encyclope- dia. http://en. http://en.wikiieedaa. wikipedia wwkipeeda..ogg///iidde php?
Patrick Lewis. Barlas Oğuz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2019. Miqa: Eval- uating cross-lingual extractive question answering. arXiv preprint 190.0:79..
[Online; accessed 05- Januaryuyy
Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang. Shuohui Chen, Daniel Simig, Myle Ott, Na- man Goyal, Shruti Bhosale, Jingfei Du, et al. 2021. Few-shot learning with multilingual language models. arXiv preprint arXiv:2 106.688...
Adina Williams, Nikita Nangia, and Samuel R Bowman. 2018. The multi-genre nli corpus.
Linting Xue, Aditya Barua, Noah Constant, Rami Al- Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2022. Byt5: Towards a token-free future with pre-trained byte-to-byte models. Transac- tions of the Association for Computational Linguis- tics, 10:291-306.
Yinhan Liu, Jiatao Gu. Naman Goyal, Xian Li. Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilinuual denoising pre- training for neural machine translation. Transac- tions of the Association for Computational Linguis- tics, 8:726-742.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2020. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010. 11944.
Ilya Loshchilov and Frank Hutter. 2017. Decou- pled weight decay regularization. arXiv preprint arXiv:/711.0510f.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng. David Grangier, and Michael Auli. 2019. Fairseq: A fast, extensible toolkit for se- quence modeling. arXiv preprint Xrxiv: 1904.01038.
Bo Zheng, Li Dong. Shaohan Huang, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, and Furu Wei. 2021. Allocating large vocabulary capacity for cross- lingual language model pre- vrettrannng. EMNLP.
Xiaoman Pan, Boliang Zhang, Jonathan May. Joel Noth- man, Kevin Knight, and Heng Ji. 2017. Cross- -ingual Pro-
A
Appendix
name tagging and linking for 282 languages. ceedings of the 55th Annual Meeting of the Associa- tion for Computational Linguistics (Volume I: Long Papers), pages 1946-1958.
B
Appendix
We show the per-language results for each task we tested on. For the sake of reproducibility, we also provide the hyperparameters that we used to finetune the model for each task.
Afshin Rahimi, Yuan Li, and Trevor Cohn. 2019. Mas- sively multilingual transfer for ner. arXiv preprint 1902.0990200199
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Perey Liang. 2016. Squad: 100,000+ questions for machine comprehension of text, arXiv preprint arXiv:1600.05250.

---

Model
de
hi
vi
zh
AVG
XLM-R reimpl. 65.9/78..
50).4 67.7 47.6/ 62.2 36.8 55.8 42.1 / 59.3
45.2 65.2
37.8/ 60.7
46.5 64.2
XLM-V
/ 80.4 51.1 69.4 49.8 64.3 38.1 1 / 58.2 44.5 62.7 46.4
67.2 67.2 36.3
59.9 47.7 66.0
Table MLQA results (EM/FI), The model is trained for epochs on single A100 GPU with float16 precision. We use a learning rate of 3e-5 with a max sequence length of 512, batch size of 6, no weight decay, and no warmup.
Model
en
ar
ko
ru
SW
te
AVG
XLM-R reimpl. 55.5/68.6
42.0/63.9 18.6/37.6 42.8/61.6 54.7/73.1 23.6/39.9
31.5/59.9 30.7/54.1 27.5/44.3
36.3/55.9
XLM-V
52.3/66.9
45.4/65.5 27.4/42.7 46.0163.6 56.1/72.3 22.8/37.4 31.5/59.3 43.1/61.4 32.4/43.2 39.7/56.9
Table 8: TyDiQA-GoldP results (EM/FI). The model trained for 8 epochs on a single A 100 GPU with float precision. We use a learning rate of 3e-5 with max sequence length 512, batch size no weight decay, and warmup.
Model
en
e8
ru
XLM-R reimpl.
72.1 / 83.5
58.5 / 76.5
57.6 73.0
55.4 72.2 56.6 /
73.1 52.2 68.3
XLM-V
72.9/ 84.2
60.3 / 78.1 57.3 / 75.1 53.5 / 72.4 56.0 73.2 51.8 67.55
ar XLM-R reimpl.
AVG
49.2 65.9
56.0 / 71.3 53.5 / 72.9 55.7 / 66.3 55.5 / 65.3 49.8/57.7
XLM-V
51.2 67.5
53.7 / 73.1 56.9/ / 67.0 53.5 / 63.1 51.9/ / 69.4 56.3 / 71.9
Table 9: XQuAD Results (EM/F1). The model is trained for epochs single A100 GPU with float16 precision. We use a learning rate of 3e-5 with a max sequence length of 512, batch size of 6, no weight decay, and no warmup.
Model
gu
pa
It
uk
qu
hu
et
tr
kk
zh
my
XLM-R reimpl.
73.5 62.9 53.6
72.7
72.4
77.5 60.4 75.8 74.4 71.2 75.4 42.2 25.3 48,9 33.6
XLM-V
73.8 66.4 48.7
75.6
66.7 65.7 79.5 70.0 79.5 78.7
75.0
77.3 50.4 30.2 61.5 54.2 72.4
ko
ka
ja
bg
pt
fr
fa
hi
bn
el
de
XLMMR reimpl.
5.2 49.4 65.4
21.0 63.1 76.1 70.2 77.0 76.9 76.5 44.6
51.4 61.5 67.2 69.0 73.8
74.4
XLM-V
3.3 53.0 69.5
22.4 68.1 79.8 74.5 80.5 78.7 77.6 50.6 48.9 59.8 67.3 72.6 76.7 76.8
af
te
ml
ms
jv
id
vi
he
AVG
XLMMR reimpl.
83.0 80.0 75.83 49.2 56.3 61.9 57.2 69.8 68.3 59.4 48.6 67.7 53.2 43.8 61.3
XLM-V
83.4 81.4 78.3
51.8 54.9 63.1 67.1 75.6 70.0 67.5 52.6 67.1 60.1 45.8 64.7
Table 10: NER Results. The model is trained for 10 epochs on a single A100 GPU with float16 precision. We use learning rate 2e-5 with max sequence length of 128, batch size of 32, no weight decay, and no warmup.
Model
amh hau ibo kin
lug
luo
pem swa wol yor AVG
XLM-R R reimpl.
25.1 43.5 11.6 9.4 9.5 8.4 36.8 48.9 5.3 10.0 20.9
XLM-V
20.6 35,9 45,9 25.0 48.7 10.4 38.2 44.0 16.7 35.8 32.1
Table We show the zero-shot cross-lingual transfer results on MasakhaNER (trained on English and evaluated the unseen languages). The model is trained for 10 epochs single A100 GPU with floatló precision. We use a learning rate of 2c-5 with max sequence length of 128, batch size of 32, no weight decay, and no warmup.


Reviews:
[{"review_id": "3mmA0fVO8Q", "reviewer": "Reviewer_BY56", "summary": "", "questions": "A. lines 217-220: First, a clarification. I assume that the set of languages here is the same as for Chung et al., namely those from CC100 -- is this accurate? It might be worth pointing out that the ideal number of clusters should be expected to vary not based on the number of languages in the set of 100+ languages, but rather on the identity of those languages and their respective similarities with one another. How do you expect this clustering approach might scale as more languages get included? And more generally, how do you expect the model might scale as you reach beyond 100 languages? \n\nB. Overall, it would be nice to have a table showing not only the number of languages covered by each data set, but also which languages are in each data set, and where they do and do not overlap. \n\nC. Regarding bold-facing of results in all results tables - there are some inconsistencies. For example, in Table 2, the bold-faced value for XNLI is not in fact the highest score. In some cases, matching scores are both highlighted, in some only the XLM-V row is highlighted. More generally, has any significance testing been done?\n\nD. Figure 3 is hard to interpret visually - it made sense to me only after reading the textual description.\n\nE. The table for MNER in the appendix shows a different picture from what the text suggests -- there are quite a few languages for which XLM-V is significantly outperformed by XLM-R. It would be nice to see this addressed, analyzed, and discussed in the body of the paper.\n\nF. line 451: do you actually mean to say that 'sverordnung' is the correct German word? In case this is new information, the noun is 'Verordnung', and the 's' is commonly used for linking words within German compound nouns. I'm assuming that this is not new information, though, in which case I think the text around this example should be softened. Minor quibble.\n\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is a strong paper presenting sensible developments for multilingual MLMs, and with improved performance for many tasks and languages. \n\n1. The approach addresses an ongoing problem associated with multilingual models and has some promise for combatting the vocabulary bottleneck issue.\n\n2. The experimental work is carefully and thoroughly carried out and reported. In addition, the paper includes a reasonable amount of analysis (we always like more!).\n\n3. The paper is well-written and well-organized.\n\n4. The results on low-resource data sets are especially exciting and encouraging!", "reasons_to_reject": "Mostly I only have small quibbles with this paper (see comments and questions below) and would be happy to see it accepted. However, there's one claim I find unconvincing (or at least not especially convincingly demonstrated). This is the claim that the tokenizations produced by XLM-V are qualitatively better, linguistically more sensible, and/or semantically more meaningful than those produced by earlier models. \n\nThe only demonstration given for this claim comes from the three examples presented in Table 5. Of these examples, I see mixed results (with the caveat that I'm not able to assess the Chinese tokenization). While I would agree that the segmentations produced for variant forms of the word 'narcolepsy' look better for XLM-V than for other models, I'm not sure the same is true for the German example. For the German example, XLM-V does better toward the end of the compound than earlier models (arguably -- why segment off 'ver' for 'verschreibung' but not for 'verordnung'?), its segmentation for the beginning of the compound isn't much better (e.g. 'ungsmittel' makes little sense). \n\nMore generally, though, I'm not quite willing to accept this claim based on a few selected examples. The paper could really be improved with a more extensive analysis, perhaps comparing to outputs of morphological analyzers for some languages, and to word segmentation systems for others (such as Chinese)? \n\nAll this said, this alone is not a rejection-worthy shortcoming.", "questions_to_authors": "A. lines 217-220: First, a clarification. I assume that the set of languages here is the same as for Chung et al., namely those from CC100 -- is this accurate? It might be worth pointing out that the ideal number of clusters should be expected to vary not based on the number of languages in the set of 100+ languages, but rather on the identity of those languages and their respective similarities with one another. How do you expect this clustering approach might scale as more languages get included? And more generally, how do you expect the model might scale as you reach beyond 100 languages? \n\nB. Overall, it would be nice to have a table showing not only the number of languages covered by each data set, but also which languages are in each data set, and where they do and do not overlap. \n\nC. Regarding bold-facing of results in all results tables - there are some inconsistencies. For example, in Table 2, the bold-faced value for XNLI is not in fact the highest score. In some cases, matching scores are both highlighted, in some only the XLM-V row is highlighted. More generally, has any significance testing been done?\n\nD. Figure 3 is hard to interpret visually - it made sense to me only after reading the textual description.\n\nE. The table for MNER in the appendix shows a different picture from what the text suggests -- there are quite a few languages for which XLM-V is significantly outperformed by XLM-R. It would be nice to see this addressed, analyzed, and discussed in the body of the paper.\n\nF. line 451: do you actually mean to say that 'sverordnung' is the correct German word? In case this is new information, the noun is 'Verordnung', and the 's' is commonly used for linking words within German compound nouns. I'm assuming that this is not new information, though, in which case I think the text around this example should be softened. Minor quibble.\n\n", "paper_topic_and_main_contributions": "This paper describes XLM-V, a new entry in the field of multilingual MLMs, with innovations in both the size of the shared vocabulary and the ways in which that vocabulary is shared (or, importantly, sometimes not shared). The key innovations are: \n\n1 -- an expansion of the model's shared subword vocabulary to roughly 1M tokens.\n2 -- incorporation of a lexical clustering approach (following Chung et al. 2020) which is used to first identify clusters of languages with a higher-than-average degree of lexical overlap, and then to encourage vocab sharing between clustered languages, and discourage vocab sharing between non-clustered languages [roughly put]\n3 -- incorporating of a vocabulary allocation measure (from Zheng et al. 2021) in order to decide how much of the overall vocabulary to devote to each language cluster\n\nFor a range of tasks, on a range of languages, the model outperforms both smaller vocabulary models and reimplementations of earlier models at a similar vocabulary size, but either without the clustering and allocation steps described above, or with only one or the other.\n\nIncreasing the vocabulary size of course increases both the memory and compute demands of the model, but the authors find that compute time does not in fact scale at the same rate of the increase vocabulary size, largely because the development of specific shared vocabularies for lexically-similar languages means that most of the vocabulary is unused for any given input sample. \n\nThis work presents a logical next step in multilingual language modeling and should be useful for many researchers, particularly those working on low-resource languages, as this approach promises to combat some of the problems associated with poor vocabulary representation for diverse languages.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "* lines 306-314: please add # of languages in MLQA\n\n* Table 2: please add description of what 'EM' stands for\n\n* lines 329-335: please add # of languages for NER\n\n* lines 336-340: it would be nice to have a description of ANLI languages similar to that provided for MNER\n\n* References: please fix capitalization issues!\n\n"}}, {"review_id": "0fQiGBSOKx", "reviewer": "Reviewer_rngv", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The idea is good, and the method is straightforward. The paper is easy to follow and well-written. The authors tackle the vocabulary bottleneck and handle the over-tokenization for low-resource languages. ", "reasons_to_reject": "I do not have a reason to reject it. But I am curious about the performance of the XLM-V model on the GLUE benchmark dataset. The authors evaluate XLM-V on 3 types of downstream tasks (e.g., NLI, NER and QA) with different datasets, however, as XLM-R is the model that they direct compare with, I wonder if XLM-V can also beat XLM-R on GLUE benchmark datasets. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors propose XLM-V, which is a multilingual language model with a larger vocabulary size compared to the XLM-R model. Their approach scales up the size of the vocabulary by reducing the emphasis on sharing tokens between languages that have limited word lexical overlap and allocating vocabulary space to ensure sufficient coverage for each individual language.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "GqOL4eDCT8", "reviewer": "Reviewer_hL9S", "summary": "", "questions": "Thanks for this great work, I have the following questions for you:\n\n1: except the lexical clustering, do you tried some more naive method such as directly compute the vocabulary overall all the corpus?\n\n2: to cluster the language, except the k-means, have we tried any other clusters algorithm? or have we changed the cluster number to other number?\n\n3: do you have any insights why the approximated softmax doesn't work too much or unstable for pre-training? I think if we want to move to 2 million tokens, then we have to solve the softmax problem first.\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper explore an interesting idea: scaling the vocabulary in LLM to one million. And experimental results demonstrate the superiority of their new model.", "reasons_to_reject": "I don't find any obvious reasons to reject this paper.", "questions_to_authors": "Thanks for this great work, I have the following questions for you:\n\n1: except the lexical clustering, do you tried some more naive method such as directly compute the vocabulary overall all the corpus?\n\n2: to cluster the language, except the k-means, have we tried any other clusters algorithm? or have we changed the cluster number to other number?\n\n3: do you have any insights why the approximated softmax doesn't work too much or unstable for pre-training? I think if we want to move to 2 million tokens, then we have to solve the softmax problem first.\n", "paper_topic_and_main_contributions": "This paper focus on multilingual large language models. Specifically, they improve the previous released masked language model XLM-R by using a much larger vocabulary that has 1 million tokens. They also did various experiments on various multilingual benchmarks to show that the proposed model is much better than previous XLM-R.\n\nContribution:\n\n(1): propose a new method to construct the large (one million) vocabulary with lexical clustering.\n\n(2): based on this newly constructed vocabulary, they followed previous XLM and trained a new model called XLM-V.\n\n(3): experimental results on various benchmarks including XNLI, FloRes-200, MLQA etc demonstrate the superiority of their new model. \n\n\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work studies the impact of the vocabulary size on a XLM  model, together with a lexical cluster approach and encouraging vocabulary share between clustered languages. This leads to overall performance boosts on a large variety of benchmarks. It is also remarkable that the authors did the experiments on truly low resource languages. The approach was exciting for all reviewers and no major objections to include this paper in to the main conference were made.
Decision: Accept-Main