Efficient Multilingual Language Model Compression through Vocabulary Trimming
Authors: Asahi Ushio
Yi Zhou
Jose Camacho-Collados
Venue: EMNLP 2023 Findings (2023)
Keywords: multilingual
language model
model compression
efficiency

Abstract: Multilingual language models (LMs) have become a powerful tool in NLP, especially for non-English languages. Nevertheless, model parameters of multilingual LMs remain large due to the larger embedding matrix of the vocabulary covering tokens in different languages. Instead, monolingual LMs can be trained in a target language with the language-specific vocabulary only. In this paper, we propose vocabulary-trimming (VT), a method to reduce a multilingual LM vocabulary to a target language by deleting potentially irrelevant tokens from its vocabulary. In theory, VT can compress any existing multilingual LM to any language covered by the original model. In our experiments, we show that VT can retain the original performance of the multilingual LM, while being considerably smaller in size than the original multilingual LM. The evaluation is performed over four NLP tasks (two generative and two classification tasks) among four widely used multilingual LMs in seven languages. The results show that this methodology can keep the best of both monolingual and multilingual worlds by keeping a small size as monolingual models without the need for specifically retraining them, and can even help limit potentially harmful social biases.

Efficient Multilingual Language Model Compression through Vocabulary Trimming
Asahi Ushio and Yi Zhou and Jose Camacho-Collados
Cardiff NLP, School of Computer Science and Informatics, Cardiff University, UK (UshioA, Zhouy131 CamachoCol lecoooooooddo ff ac. uk
Abstract
models, multilingual LMs can be used for zero/few-- shot learning (Scao al., 2022) by increasing the model size and, more frequently, can be specialized to different tasks by fine-tuning to specific data. practice, however, there are a few practical issues when training multilingual LM such as the curse of multilinguality (Conneau et al., 2019; Pfeiffer et al.. 2022), trade-off between the number of languages and individual performance in single language. or the multilingual vocabulary construction, which requires a careful design for better generalization (Chung et al., 2020; Zheng et al., 2021; Liang et al., 2023).
Multilingual language models (LMs) have be- come powerful tool in NLP, especially for non-English languages. Nevertheless, model parameters of multilingual LMs remain large due to the larger embedding matrix of the vo- cabulary covering tokens different languages. Instead, monolingual LMS can be trained in a target language with the language-specifii vocabulary only. this paper, we propose vocabulary-trimmmnng (VT), method reduce a multilingual LM vocabulary to target lan- guage by deleting potentially irrelevant tokens from its vocabulary. In theory, VT can com- press any existing multilingual LM to any lan- guage covered by the original model. In experiments, we show that VT can retain the original performance the multilingual LM, while being considerably smaller in size than the original multilingual LM. The evaluation is performed over four NLP tasks (two generative and two classification tasks) among four widely used multilingual LMs in seven languages. The results show that this methodology can keep the best of both monolingual and multilingual worlds by keeping a small size as monolingual models without the need for specifically retrain- ing them, and can even help limit potentially harmful social biases.
Besides such generalization concerns, multilin- gual LMs usually consist of larger parameters than their monolingual counterparts due to the need for large vocabulary covering multiple languages. This becomes an important issue in practice when the resources to host models are limited. For in- stance, while using the same configuration (i.e., same number of layers and hidden units), the pa- rameter size of T5 (Raffel et al., 2020) and mT5SMALL (Xue al., 2021) are 140M and 300M, respectively. This is only due their difference in vocabulary size, with T5 being 50k and mT55 250k. In fact, the embedding matrix stemming from the LM vocabulary can occupy a large portion the parameter space. For instance, the ratio of the em- bedding matrix to the full model's parameter size in multilingual LMs can be higher than 80% as T5 (see Figure I).
1 Introduction
Multilingual language model (LM) pre-training (Devlin al., 2019; Conneau et al., 2019; Liu et al., 2020; Xue et al., 2021) has been shown be an ef- ficient mechanism to store information from many languages into single model, without the need for training multiple sunguage-specife models. More- over, it has been proven reliable for cross-lingual tasks (Pires et al., 2019; Conneau and Lample, 2019) and provide competitive performance in most settings, generally similar its monolin- gual counterparts (Goyal et al., 2021), while being bi-
In this paper, we propose a simple vocabulary trimming (VT) method to remove tokens from the vocabulary of multilingual LMs that may be irrel- evant to the target language." This achieved by automatically identifying language-specific to- kens from an underlying text corpus. We consider two VT strategies of pre-FT VT (VT before fine- tuning) and Post-FT FT VT (VT after fine- (ine-tuning) and
generally less affected by culturally-dependee ases (Ahn and Oh, 2021). Similarly to monolingual
We release Python library for VT at https: //gi thub. com/as /ww/asahiaaa 417 117/mm/ocabb vocab-trimmer. trimmer.

---

m SOMALL (M6M)
XLM-RRsE (2184)
mBARTLAGC (611M)
2 Related Work
Several studies have explored the possibility to modify or adapt the vocabulary of LMs. For in- stance, Artetxe et al. (2020) and Marchisio et al. (2022) adapted a mono-lingual LM into another language by learning the embedding matrix the new language, while fixing the other weights. Sim- ilarly, Wang al. (2019) augmented the vocabu- lary of a multilingual LM to new languages with multilingual word alignment (Lample et al., 2018). Zheng et al. (2021) proposed to evaluate the ability of a vocabulary to represent a particular language, and Chung et al. (2020) proposed multilingual vocabulary construction that balances the trade- off between optimizing for cross moos-lingual sub-word sharing and the need for robust representation of individual languages. XLM-V (Liang al., 2023) combines the idea of Zheng et al. (2021) and Chung et al. (2020) to efficiently enlarge the vocabulary size along with the model size scaling. Ostendorff and Rehm (2023) used a multi-stage fine-tuning to obtain LM in the target language from other LM in the source language. These prior works modify existing mono/multi-lingee LMs to include new languages, i.e. augmenting the multilinguality of the LMs. In contrast, our study focuses on com- pressing multilingual LMs into the target language to effectively achieve smaller monolingual LMs, i.e. reducing the multilingual representation of the LMs while retaining the capability in specific target language.
AGM
256M (4E%)
354M
256M (80%)
192M (70%)
Futeddinng Marri
Olher Weights
Figure The ratio the embedding matrix to the number of parameters for each multilingual LM.
analyse them by varying the final vocabulary size. We conduct experiments on two generation tasks, question answering (QA) and question generation (QG), and two classification tasks, sentiment anal- ysis and natural language inference (NLI), across seven different languages. The experimental results show that both pre and post fine-tuning VT can rc- duce the model size while retaining the original performance in generation tasks (QA and QG), and particularly in classification tasks (sentiment and NLI) where the results are close to being identi- cal despite the significant reduction in vocabulary size. In all tasks, the original performance can be generally maintained with less than 40% of the full model parameters for all languages.
Finally, even though pre-trained LMs have re- ported impressive performance on various NLP downstream tasks (Kenton and Toutanova, 2019; Liu et al., 2019; Conneau et al., 2019), such LMs also demonstrate worrying levels of social biases in certain situations (May et al., 2019; Kurita et al., 2019; Kaneko and Bollegala, 2021). One natural question that arises is whether VT can have an influence the bias level in multilingual LMs, in- cluding fine-tuned models. For this purpose, evaluate social bias multilingual LMs after ap- plying VT with different settings and compare against its monolingual counterpart. Experimen- tal results show that the monolingual LM tends to contain more bias than its multilingual versions. Moreover, compared to the original multilingual LM, the bias level has significant change after applying VT. These results suggest that mono- lingual LM can be induced by applying VT to its corresponding multilingual LM, thereby obtaining a less biased monolingual LM compared to its orig- inal monolingual counterpart.
The work of Abdaoui et al. (2020) is the most relevant our study as, the best of our knowl- edge, they introduced the idea of VT for the first time. However, their analysis limited to NLI with pre-finetuunin VT with mBERT (Devlin et al., 2019) only, as well as a fixed vocabulary size af- ter VT. In contrast, our study compares two VT strategies, before and after fine-tuning, and show how this latter strategy, not considered in Abdaoui et al. (2020), be more effective compression technique in some settings. Furthermore, we ex- tend the experiments to generation tasks as well classification tasks with more recent LMS such as mBART (Lewis et al., 2020) and mT5 (Xue et al., 2021), and provide an exhaustive analysis on the effect of VT.
3 Vocabulary Trimming
To perform vocabulary trimming (VT), we first need a multilingual LM as an input. The idea is to

---

inguui
Pre-FT
VT (Fxench)
FF (Pzench)
Bareddiing
Multilingual
IM
Rost-PT 12
FT (French)
(Fzench)
Enalish
Xecson
Fiendh
Figure Comparisons Pre-FT Post-FT VT in example of fine-tuning task in French.
Forman-Drimmed
Prensh-Trirmed IM
Ochez Ne glt
wabed
4.1 Experimental Setting
Tasks and datasets. In order to test the efficacy of VT. we consider two generation tasks, ques- tion answering (QA) and question generation (QG), and two classification tasks, sentiment analysis and natural language inference (NLI). As the datasets for QA, we use SQuAD (Rajpurkar et al., 2016) (English), Spanish SQuAD (Casimiro Pio et al., 2019) (Spanish), FQuAD (d' (d'Hoffschmidt et al., 2020) (French), Italian SQuAD (Croce et al., 2018) (Italian), JAQuAD (So et al., 2022) (Japanese), Ko- rQuAD (Lim al., 2019) (Korean), and SberQuAd (Efimov et al., 2020) (Russian). For QG, the same datasets adapted for QG via QG-Bench (Ushio al., 2022). For sentiment analysis, we use Twitter-based datasets for English (Rosenthal al.. 2017), Arabic (Rosenthal et al., 2017), French (Be- namara et al., 2017), Italian (Barbieri et al., 2016), German (Cieliebak et al., 2017), Portuguese (Brum and Nunes, 2017), and Spanish (Díaz-Galiano al.. 2018) from UMSAB (Unified Multilingual Senti- ment Analysis Benchmark) (Barbieri et al., 2022). All the sentiment analysis datasets contain three labels: positive, neutral and negative. For NLI, we use XNLI (Conncau et al., 2018), a multilingual NLI dataset, including English, French, German, Spanish and Arabic, which are the languages in- cluded in the sentiment analysis experiment. We fine-tune LMs on the training sets of each language. which translated automatically from English and released in the original paper.
French
Figure 2:
An illustration of vocabulary trimming for Korean and French.
tailor model to particular target language I, which in principle belong to the same set of languages used to trained the input multilingual LM. For the target language 1, VT first identifies language- specific tokens on language-specific corpus C, and remove all the tokens along with their embed- dings except for those appeared in Cj as described in Figure 2. In our analysis ($ 5), we also con- sider to keep the top-n most frequent tokens in C₁ to further reduce the model size by removing less frequent tokens. We consider two VT strategies: (I) before fine-tuning and (2) after fine-tuning.
The difference between these two strategies is whether to perform VT before or after fine-tur tin-ttnnng, as shown in Figure 3. Both VTs have advantages and drawbacks: while pre-FT VT can reduce the time of fine-tuning as the trimmed LM is smaller than the original LM, PO VT only need fine- tuned multilingual LM this way, post-FT VT can be used as a postprocessing step and no additional language-specific training is required.
Finally, we release simple LM vocabulary trimming starting package to apply our proposed technique any input multilingual transformer- based LM, along with all the models and code needed reproduce our experiments, at https: vocab- tr inmer.
Evaluation metrics. For the evaluation, we use the following standard metrics: answer span FI score (Ans-F1) and exact match (EM) are used for QA; METEOR (MTR) and BERTScore (BS) for QG, which have been shown to be the most cor- related metries to human judgment (Ushio et al., 2022); macro-F1 score for sentiment following (Barbieri et al., 2022); and accuracy for NLI. As the language-specific corpus C₁ to extract vocabu- lary counts for VT, we use mC4 (Xue et al., 2021), one of the largest public multilingual corpora.
4 Evaluation
In this section, we present our experimental results test the reliability of our proposed VT methodol- ogy in NLP tasks.
In theory, vocabulary trimming could be applied to any language model, even monolingual, but this analysis is out of the scope of this paper.

---

QA
QG
Lang.
Vocabulary
Parameters
No-Trim
Post-FT
Pre-FT
No- No-Trim
Post-FT
Pre-FT
EN
209K (83.6%)
258M (86.1%)
70.1 / 55.5
70.2
70.1 / 56.4
23.8 90.0
23.8 90.0
24.0/ 90.1
ES
13IK (52.4%)
178M (59.4%)
55.9 34.7
55.9 34.7
57.8/ 37.5
22.7 84.1
22.7 84.1
22.3 84.2
FR
13IK (52.4%)
178M (59.4%) 4%)
50.0 50.0/30.9
48.6/
17.5 80.7
17.5 / 80.7
/
IT
111K (44.4%)
157M (52.6%)
53.2/
53.4/37.8
51.5
17.6/ 80.8
17.6/ 80.8
17.5/ 80.6
JA
125K (50.0%)
172M (57.6%) 37.6%
65.7 65.7
65.7/65..
63.0/63.0
29.0/ 80.9
29.0/ 80.9
28.6 81.0
KO
73K (29.2%)
119M (39.7%)
77.1/70.6
77.1/70.5
/
27.5/ 82,9
27.5 83.0
28.0 83.7
RU
147K (58.8%)
195M (65.1%)
73.7 51.4
73.8/514
74.8/53.4
26.4/ 84.3
26.4/ 84.3
28.9 86.4
173K (69.2%)
532M (87.1%)
76.9/62.66
62.0
78.4/ 65.7
25.1 / 99.4
25.1 90.4
24.7 1
ES
87K (34.8%)
443M (72.7%)
/4.1/
64,5/42.8
63.7/43.9
22.9/ 83.6
22.8/ / 83.6
22.8 / 84.0
mBART
FR
85K (34.0%)
442M (72.5%)
60.4/39.3
61.0/39.8
66.4/45.1
19.8/81.7
19.8/ 81.7
18.4 79.7
IT
67K (26.8%)
424M (69.5%)
/5.7
64.9/50.2
65.8/49.8
18.0/ 80.6
17.9 80.7
18.9 81.1
JA
77K (30.8%)
434M (71.1%)
68.2/
68.2/68.2
70.6 70.6
30.0/ 82.3
29.7 / 82.1
29.1 80.8
KO
46K (18.4%)
402M (65.9%)
79.3
83.2/77.3 79.2/72.1
30.2 83.9
30.3 84.0
30.2 83.8
99K (39.6%)
456M (74.8%)
78.7/58.
79.0/58.1
79.5
29.3/
28.7 / 87.0
28.3 86.7
Table I: Results on QA (Ans-FI/EM) and QG (MTR/BS), including both the vocabulary size and the number of parameters after VT with the ratio to the original model (%) The best results in each LM and language are in bold characters. Note that the parameter size of the original mT5 and mBART (No-Trim) is 300M and 61 61, respectively, both with a vocabulary size of 250K.
Base language models. As the base LMs, given computational constraints we chose the smallest mT5 and mBART to fine-tune QA and QG, and XLM-R and XLM-V (Liang et al., 2023) for sen- timent analysis and NLI. All these models have a vocabulary size of 250K, except for XLM-V which has a vocabulary size of 901K subword to- kens. For our experiments, we compare the results of pre/post-FT VT against vanilla LM fine-tuning without VT, which we refer to as No-Trim.
the vanilla fine-tuning for all the languages for both LMs in QA and QG, except for a few cases such as mBART QA in Korean and mBART QG in Rus- sian, although the decrease is more than 0.5%. Mcanwhile, pre-FT VT outperforms its vanilla fine- tuning model with a relatively important margin in some cases, such as mBART French QA and mT5 Spanish QA. In contrast, there are a few models where pre-FT VT degrades the performance of the original model such as mT5 QA in Korean (2.6% decrease in Ans-F1) or mBART QA in Russian (3.2% decrease in Ans-F1).
Fine-tuning. For model fine-tuning, we rely on Imag (Ushio et al., 2023) for QA/QG, and Ray Tune? for sentiment analysis. In both cases, we use the default search space for hyperparameter search. For NLI, we follow the same hyperparameters used in Liang et al. (2023). All the resulting models and code will released upon acceptance of the paper.
Since we keep all the vocabulary that appeared in the language-specific corpus Ct, the percent- age of reduced parameter depends on the lan- guage, and generally VT can reduce the model size for Asian (Japanesc/Korean) and European (Span- ish/French/Italian languages efficiently (50% for mT5 and 70% for mBART), but remains high in other languages (English/Russian)
4.2 Results
We present the results for the generation and clas- sification tasks in section 4.2.1 and section 4.2.2, respectively.
4.2.2 Classification Tasks: Sentiment & NLI
Table 2 shows the results on sentiment analysis and NLI. In this case, post-FT VT can robustly preserve the original performance of the original No-Trim baseline in both tasks for XLM-R and XLM-V, while being no more than 40% and 60% in vocabu- lary and overall parameter size, respectively, of the original XLM-V and XLM-R models in all the non- English datasets. XLM-V PT sentiment model is the only post-FT VT where a slight decrease can be observed (0.1 %).. On the other hand, the accuracy
4.2.1 Generation Tasks: QA & QG
Table shows the overall results on QA and QG. The results confirm that both of pre/post-FT VT can maintain the original performance in most cases, while being smaller than the original models by sig- nificantly reducing the vocabulary size. First, post- FT VT achieves at least the same performance as
Phttps://docss ray. ht//n//atest//uneeeeeee html

---

Sentiment Post-FT
NLA
Lang.
Vocabulary
Parameter
No-Trim
Pre-FT
No-Trim
Post FT
Pre-FT
AR
49K (19.6%)
124M (44.7%)
66.3
66.3
60.9
75.7
75.7
73.8
91K (36.4%)
156M (56.3%)
73.2
73.3
73.5
79.9
79.9
78.3
XLM-R
EN
173K (69,25)
219M (78, 7%)
68.4
68.5
67.9
84.6
84.6
70.6
ES
87K (34.8%)
153M (55.0%)
69.0
69.0
65.0
79.8
79.8
67.2
FR
(34.0%)
151M (54.6%)
71.8
71.8
72.1
80.1
79.6
II
67K (26.8%)
138M (49.7%)
62.9
62.9
70.8
PT
66K (26,4%)
137M (49.3%)
70.7
70.8
70.2
92K (11,8%)
157M (20.2%)
59.8
59.8
64.7
75.5
75.6
76.1
DE
239K (30,7%)
269M (34.7%)
73.5
73.5
73.0
78.9
78.9
79.0
XLM-V
484K (62.2%)
458M (58.9%)
63.9
63.9
61.3
84.4
84.4
84.5
ES
243K (31.2%)
279M (35.1%)
60.7
60.7
66.6
80.7
80.7
80.6
FR
218K (28.0%)
253M (32.6%)
68.8
68.8
59.5
78.6
78.6
79.0
IT
184K (23.7%)
227M (29.3%)
70.2
70.2
74.2
18IK (23.3%)
225M (28.9%)
66.6
66.5
52.8
Table Results sentiment analysis (macro F1) and XNLI (accuracy) including both the vocabulary size and the number of parameters after VT with the ratio the original model (%).. The best results in each LM and language are in bold characters. Note that the overall parameter size of the original XLM-R and XLM-V (No-Trim) is 278M and 778M, respectively, with the vocabulary size being 250K and 901K vocabulary each case.
of Pre-FT VT appears to be sensitive to the lan- guage and task, where it improves the performance in some languages such as Italian (XLM-R and XLM-V achieve 7.9% and 3.8% increase for sen- timent analysis). but it decreases the performance with non-trivial margin in other languages such as Arabic, where XLM-R decreases 5% for sentiment analysis and 2% for XNLI. Since XLM-V has a larger vocabulary size, the percentage of reduced parameters at VT is more prominent in XLM-V. V, as seen in Arabic (20.2%) and Portuguese (28.9%) for example.
5.1 Generation Tasks: QA & QG
Figure 4 shows the results of mT55 on QA and QG, Noticeably, post-FT VT can reduce the vocabulary size to 60K for both QA and QG in all the lan- guages with a trivial gap (0.3% decrease of EM in Russian QA and 0.1% % decrease of BS in French QG), and that is 35% of the original mT5 in the pa- rameter size. Furthermore, post-FT VT can further reduce the vocabulary to 5K tokens with more than 0.4% decrease in each metric for both QA and QG in English, French, and Italian, which 16% of the original mT5 the parameter size. Mean- while, pre-FT VT outperforms the No-Trim result in all the languages in QA, and the majority the languages in QG (English, Italian, Korean, and Russian), but the result is sensitive to the choice of n. For example, Japanese/Korean QA and Russian QG with pre-FT VT for top-5K (16% of the origi- nal mT5) outperforms No-Trim well post-FT VT, but Japanese QG with pre-FT VT worse any choice of on contrary. This larger variation of results may also due to the parameter size space, as the optimal parameters for the original multilingual LM (which is the one trained for post- FT VT) may differ. We leave this extended analysis for future work.
5 Vocabulary Size Analysis
our main experimental results ($ 4.2), all the unique tokens that appeared the monolingual corpus were kept, which resulted in a low compres- sion ratio for some languages such as English and Russian. In this analysis, we constrain the number of vocabulary and choose the top-n vocabulary at VT in terms of the frequency in the corpus (see $ 3). For QA and QG, we compare mT55MALL results with from [SK, 10K, 15K, 30K, 60K, 90K, 120K], which correspond an overall param- eter size of [49M, 54M, 59M, 74M, 105M, 136M, 166MJ, respectively. For sentiment analysis and NLI. we experiment with XLM-RBASE with n from [SK, 10K, 15K, 30K, 60K], which correspond to [89M, 93M, 97M, 109M, 132M] of parameter size, respectively
5.2 Classification Tasks: Sentiment & NLI
Figure 5 and Figure 6 show the results of XLM- on sentiment and NLI. NLI, we can see that post/pre-FT VT both can reduce the vocabulary to 30K (39% of the original XLM-R R in the parameter
'The full results with top-n can be found at Appendix and Appendix A.

---

English (QA)
English (QG)
Arabic
German
72.5
METEOR
24.5
65-
73
No-Trm
70.0
24.0
0
Pie-IT
67.5
23.5
60
FT
100K
200K
100K
200K
100K
200K
Spanish (QA)
Spanish (QG)
English
Spanish
Nro-Fl
60
NETEOR
12
68
63
66
100K
200K
100K
200K
100K
200K
60-
100K
200K
French (QA)
French (QG)
French
Italien
50
17
74
kro-F1
NETEOR
40
72
15
100K
200K
100K
200K
100K
200K
0
100K 200K
Italian (QA)
Italian (QG)
Portuguese
60
METEOR
70
Arr-F1
17.5
65
100K
200K
100K
200K
100K
200K
Japanese (QA)
Japanese (QG)
Vocabulary
Arn-F1
METEOR
30
Figure 5: Sentiment analysis macro-FI results of XLMM R with pre/post-IT VT for different vocabulary sizes compared to No-Trim.
80
100K
200K
100K
200K
Korean (QA)
Korean (QG)
Arabic
German
Anr-11
80
METEOR
Accurecy
66
28.0
No Trlm
Fro-FT
27.5
60
71
100K
200K
100K
200K
100K
200K
100K
200K
Russian (QA)
Russian (QG)
Engliihh
Spanish
Ans-1
METEOR
Pre-IT
lecurecy
68
67.5
Post FT
65.0
66
70
62.5
100K
200K
100K Vacabulary
200K
200K
100K
200K
Vocabulary
French
Figure 4: QG (METEOR) and QA (Ans-FI) results for mT5 with pre/post-FT VT for different vocabulary sizes compared to the original multilingual LM (No-Trim).
Accuracy
100K
200K
Figure 6: NLI accuracy of XLM-R R with pre/post- VT for different vocabulary sizes compared to No-Trim.
size) without any decrease except 0.3% of pre-FT VT for German, and there is no decrease more than 0.4% even with 15p-15K of post-FT VT. In senti- ment analysis, Fre-FT VT with t0p-10K (33% of the original XLM-R in the parameter size) can re- tain the accuracy of the No-Trim baseline in French and Italian. Moreover, post-FT VT with 30K can retain the original Fl score without a major drop in sentiment analysis, yet the decrease in Fl score is slightly more prominent than NLI (1.1% in Arabic sentiment analysis).
contain tokens from other languages. This can explain the effectiveness of top-n VT in NLI com- pared with sentiment analysis, smaller values of should result in a vocabulary with fewer tokens from the other languages, which limits the ability of the models to handle foreign tokens.
Monolingual vs. Multilingual LMs: The Case of Social Bias
The sentiment analysis datasets are collected from Twitter, one dataset in single language can contain tokens from other languages (hash- tags or named-entities, or even code-switching). In contrast, XNLI translates English NLI into other languages, so there is less chance for a dataset to
There has been extensive literature in NLP com- paring monolingual and multilingual LMs (Muller et al., 2021; Goyal et al., 2021). As for the perfor- mance, there is no clear consensus on which type is better for certain languages, tasks or settings. How-

---

ever, there are other important factors that play a role in this comparison. First, monolingual mod- els tend to have a smaller vocabulary size, which makes them more practical. In contrast, a single multilingual model can be used for a large number of languages. Moreover, multilingual LMs are less prone to capture and carry cultural- or language- dependant biases. This is due to the combination of languages and cultures into a single model, which makes it less biased toward specific cultures (Liang et al., 2020; Ahn and Oh, 2021). Prior works have shown that different types of biases consistently appear in gegguage--peciffi models (Nadeem et al., 2021; Nangia et al., 2020; Blodgett et al.. 2021; Dhamala et al., 2021; Kaneko et al., 2022; Zhou et al., 2022). While the comparison of monolingual and multilingual LMs is not the main focus of this paper, this analysis is certainly relevant. Trimming the vocabulary of a multilingual model essentially makes the model smaller, and therefore alleviates one of the main drawbacks of using multilingual language models on language-specific tasks, which is its larger size. On top of that, this strategy en- ables the usage of monolingual models with poten- tially less social bias. In the following, we present comparison of monolingual and multilingual LMs (both trimmed and not trimmed) in terms of social bias and general performance.
Mohammad, 2018)² considering two bias types: gender and race. The EEC dataset was specifically with the aim to examine social bias for sentiment analysis systems.
Evaluation metrics. We compare the pseudo- likelihood scores returned by each model for stereotypical and anti-stereotypica sentences us- ing AULA (All Unmasked Likelihood with At- tention weights) (Kaneko and Bollegala, 2022). R. AULA has been shown be robust against the frequency biases of the masked tokens and pro- vides a more reliable assessment in contrast to al- ternative metrics when evaluating social biases in masked language models (MLMs). Given a sen- tence pair in the test dataset: "My mom spent all day cooking for Thanksgiving" vs. "My dad spent all day cooking for Thanksgiving", the first sen- tence is considered stereotypical while the sec- ond one is anti-stereotypical. AULA computes the percentage of stereotypical sentences preferred by the MLM over anti-stereotypical ones as the bias score. An MLM is considered to be unfairly biased if it returns higher pscudo-loglicclioood scores for stereotypical sentences than the corresponding anti- stereotypical sentences. The AULA score falls within the range of [0, and an unbiased model would return a bias score close to 50. On the other hand, bias score greater than or less than 50 in- dicates the bias direction towards the stereotype or anti-stereotype, respectively. Since the original AULA is not fitted to evaluate fine-tuned models, we adapt AULA to the EEC dataset obtain the bias score for the LMs fine-tuned on sentiment analysis, and denote this metric EEC-AULA. Specifically, given model that predicts sentiment labels (e.g.. positive, neutral, negative) to sentences, we con- sider the percentage of stereotypical test sentences with a more negative label over unti-stereotypical ones as the corresponding bias evaluation measure.
6.1 Experimental setting
Social bias datasets. To study the effect of VT on social bias existing in pre-trained LMs, we first conduct experiments on two commonly used social bias evaluation datasets for masked LMs: Stere- oSet (SS; Nadeem et al., 2021 and crowdsourced stereotype pairs benchmark (CP; Nangia et al., 2020)6. SS consists of associative contexts cov- ering four types of social biases: race, gender, reli- gion, and profession; while CP is crowdsourced and annotated by workers in the United States, which contains nine types of social biases: race, gender, sexual orientation, religion, age, nationality, disability, physical appearance, and socioeconomic status. In order to further investigate the impact of pre/post-IFT VT on LMs, we trim and fine-tune models on sentiment analysis with different orders and evaluate the social bias such models on the Equity Evaluation Corpus (EEC; Kiritchenko and
General performance. As a proxy to test the general performance, we use the general language understanding evaluation (GLUE; Wang et al., 2018) benchmark. We acknowledge the limita- tions of using this benchmark draw reliable con- clusions at large (Ethayarajh and Jurafsky, 2020)
html
" iias_in_mim
htttps://gtthub.. httpss///ggtuub...o/m/mmmmeemee hwt..//miiiiaaeeettteeeeeeee 'https: httpp///w.cccmoooool//////s/appa
"https: http:///lubbeehhhrrr cam/; Models are tested on the development sets of each task.

---

Social Bias
General Performance
Vocabulary
Buse Model
Vocab Trimming
AULA
EEC-AULA
GLUE
Pre-FT
Post-FT
CP
SS
Gender
Race
Monolingual (RoBERTa)
50K
50K
58.1
58.8
64.8
85.7
78.0
250K
250K
49.5
54.9
44.3
62.0
77.9
Pre-FT VT (EN)
173K
173K
49.5
54.9
42.5
56.9
78.0
Multilingual (XLM-R)
Post-FT VT (EN)
250K
173k
49.5
54.9
44.3
62.0
77.9
Pre-FT VT (50K)
50K
50K
49.3
55.0
41.0
62.4
76.9
Post-FT VT (50K)
250K
50K
49.5
54.9
44.3
62.0
77.9
Table 3: Results of pre/post-FT VT models compared with the original monolingual and multilingual models on two social bias analysis benchmarks (AULA for pre-trained masked language models and REC-AULA models fine-tuned on sentiment analysis) and the GLUE benchmark. The VT models are trimmed on English vocabulary with different vocabulary sizes: EN (full English vocabulary) and 50K (top 50K subword tokens). Note that for post-FT VT, the results on AULA are exactly the same the original XLM-R. The green and red colours represent the social bias towards anti-stereotypical sentences (scores lower than 50) and stereotypical sentences (scores higher than 50), respectively. The lighter colour indicates less social bias observed in the LM.
but it nevertheless provides good proxy for un- derstanding the overall performance of comparable models in standard NLP tasks. Moreover, these experiments are only aimed analysing the effect of vocabulary trimming on general performance.
gual models with different settings. In particular, RoBERTa obtains the overall highest bias score the EEC dataset after fine-tuning, with an alarm- ingly high 85.7 score on race." On the other hand, compared to the original XLM-R, there is no signif- icant change in performance on social biases and GLUE evaluation tasks for pre-FT VT and post- FT VT models. This is important as we can apply the proposed VT method to any multilingual LM, obtaining a monolingual one with consistent per- formance the GLUE benchmark and less social biases than the original monolingual model pre- trained in the target language, without using any ad-hoc debiasing methods.
Models. We compute the bias scores of RoBERTa (Liu et al., 2019) as base monolingual LM, and XLM-R (Conneau et al., 2019) as its mul- tilingual counterpart (they have been trained with the same architecture and in overlapping cor- pus. We explore two VT settings be applied XLM-R: XLM-R with the standard VT includ- ing the full English vocabulary (VT XLM-R) and XLM-R with VT for top-50K English vocabulary (top-50K VT XLM-R), which is the same vocabu- lary size as the monolingual RoBERTa model. Our experiments are based both on masked language models on AULA (in which the post- VT does not have any effect) and models fine-tuned on the senti- ment analysis presented in $ 4.1 on EEC-AULA, as well as on the corresponding GLUE training sets.
Discussion
Vocabulary trimming before and after fine- tuning. According to the results, pre-FT VT ap- pears be generally more effective in classification tasks (see section 4.2.2). For generation tasks, both pre/post-IT VT robustly retain the original perfor- mance while being able to considerably reduce the model size (see section 4.2.1). As a guideline to choose the type of VT in such a case, post-FT VT should be more suitable if one already has fine- tuned model, as additional training is needed for this case. Moreover, post-FT is more robust as a compression mechanism as the performance is largely maintained with respect to that of the original multilingual LM. On the other hand, one needs to fine-tune a model from scratch and the computation resources are limited, we recommend
6.2 Results
Table shows the performance of pre-FT and post- FT VT models against the original monolingual and multilingual LMs on social bias evaluation datasets and the GLUE benchmark. Both AULA and GLUE results are computed using the LMs without fine-tuning (i.c., RoBERTa, XLM-R, VT XLM-R, and top-50K VT XLM-R), whereas the BEC-AULA results are computed using the models applying VT and fine-tuning strategies. We observe that the monolingual model contains the highest levels of social biases compared to the multilin-
Appenddi C includes a breakdown by type of the social hias results.

---

exploring pre-IT VT, as fine-tuning on trimmed LM should be more efficient due to its smaller vo- cabulary and parameters and, in some cases, can lead to better overall results. However, this process has to be done carefully as the set of optimal pa- rameters could differ from the original multilingual LM fine-tuning process.
compressed LM can even achieve better results than the original larger model when trimmed be- fore fine-tuning. Since the main goal of the paper was to compress a multilingual LM while keeping its original performance, we leave the analysis of this behaviour for future work.
Limitations
Monolingual and multilingual LM comparison. While in this paper we have not compared monolin- gual and multilingual models, the question would be whether we need vocabulary trimming strate- gies in world where monolingual LMs exist. In this case, a monolingual model may perform simi- larly to multilingual model (Goyal et al., 2021). However, the multilingual model is often larger mainly due to larger vocabulary storage require- ments. In contrast, our proposed VT technique does not require any extra LM training or com- putational resources. Indeed, only a multilingual LM is needed and we can induce multiple smaller language-specific monolingual models. This may reduce the carbon footprint overall and especially help with less-resource languages when high- quality monolingual model does not exist. Finally, our social bias analysis see $ shows how monolin- gual models exhibit larger social biases (especially racial) than a VT-induced multilingual LMM. This is consistent with prior work suggesting that a multi- lingual LM has been trained with more languages, and hence more cultural variety, and these diverg- ing viewpoints compensate cach other (Ahn and Oh, 2021).
We have not tested our methodology in truly low- resource languages. Because of this, there could be different behaviour when we apply VT to language with lower resources or that is poorly rep- resented in the underlying training corpus. The LMs we used in the paper limited their size up to 600M, and we have not considered larger models such as mT5xXL or BLOOM (Scao et al., 2022), due to our limited computational resources. As the danguage-specifie corpus to compute frequency, we employ mC4, which is one of the largest multilin- gual corpora. Nonetheless, this is used proxy and having access to the full multilingual model could give potentially better results.
Similarly, we acknowledge the limitations of the analysis comparing multilingual and monolingual models in terms of social bias. Due to the evalua- tion data available and the existence of comparable monolingual and multilingual LMs, the evaluation is focused on English only and the results could dif- fer for other languages. Moreover, there are other types of biases not covered this evaluation.
Ethics Statement
Pre-trained LMS are known to contain undesirable biases to generate toxic contents in some edge cases (Schick et al., 2021), so the resulting models could inherit such biases. While we have not analysed in detail the output of all models in the tasks evaluated, in this paper we have made an attempt study this effect in terms of social biases for both base pre- trained LMs and fine-tuned LMs.
8 Conclusion
In this paper, we proposed vocabulary-trimming (VT), a method to reduce the vocabulary of a mul- tilingual LM to a vocabulary specific to any given target language. VT induce monolingual LM in the target language by leveraging an exist- ing multilingual LM. The main advantage of this filtering step is the reduced size, as well as avoid- ing having to train monolingual LMs from scratch, which would be computationally demanding. Our experiments show how VT can retain the high per- formance of the original multilingual LM, while largely reducing the model size. For all languages evaluated, 35% compression rate proves suffi- cient keep the original performance of the larger mT5 multilingual LM both QA and QG, with a similar 39% in NLI and 55% in sentiment analy- sis with XLM-R. Interestingly, in some cases, the
A Acknowledgements
Jose Camacho-Collados and Yi Zhou are supported by UKRI Future Leaders Fellowship.
References
Amine Abdaoui, Camille Pradel, and Grégoire Sigel. 2020. Load what you need: Smaller versions of mutililingual BERT. Proceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language

---

Processing, pages 119-123, Online, Association for Computational Linguistics.
Mark Cieliebak, Jan Milan Deriu, Dominic Egger, and Fatih Uzdilli. 2017. A twitter corpus and benchmark resources for german sentiment analysis. In 5th Inter- national Workshop on Natural Language Processing for Social Media, Boston MA, USA, 11 December 2017, pages 45-51. Association for Computational Linguisties.
Jaimeen Ahn and Alice Oh. 2021. Mitigating language- dependent ethnic bias in BERT. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 533-549, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal. Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Oit, Luke Zettle- moyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning scale. arXiv preprint arXiv: 191 1.02/16.
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross- -ingual transferability of mono- lingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pages 4623-4637, Online. Association for Computational Linguisties.
Alexis Conneau and Guillaume Lample. 2019. Cross- lingual language model pretraining. Advances in neural information processing systems, 32.
Francesco Barbieri, Valerio Basile, Danilo Croce, Malv- ina Nissim, Nicole Novielli, and Viviana Patti. 2016.
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross- lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Nat- urul Language Processing, pages 2475-2485, Brus- sels, Belgium. Association for Computational Lin- Francesco Barbieri, Luis Espinosa Anke, and Jose Camacho- Camaaco-Collados. 2022. XLM-T: Multilingual language models in Twitter for sentiment analysis and beyond. In Proceedings of the Thirteenth Lan- guage Resources and Evaluation Conference, pages 258 266. Marseille, France. European Language Re- sources Association.
Overview of the evalita 2016 sentiment polarity clas- sification task. In Proceedings of third Italian con- ference computational linguistics ir 2016) fifth evaluation campaign of natural language pro- cessing and speech tools for Malian. Final Workshop (EVALITA 2016).
guistics.
Danilo Croce, Alexandra Zelenanska, and Roberto Basili. 2018. Neural learning for question answering in italian. In AI*IA 2018 Advances in Artificial Intelligence, pages 389 402, Cham. Springer Inter- national Publishing.
Farah Benamara, Cyril Grouin, Jihen Karoui, Véronique Moriceau, and Isabelle Robba. 2017. Analyse d' d'opinion et langage figuratif dans des tweets: présen- tation et résultats du défi fouille de textes deft2017.
Jacob Devlin, Ming-Wai Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- mologies, Volume (Long and Short Pupers), pages 4171-4186 Minneapolis, Minnesota. Association for Computational Linguistics.
Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, and Hanna Wallach. 2021. Stereotyping norwegian salmon: An inventory of pitfalls in fair- ness benchmark datasets. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguisties and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 1015-
Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun. Kai-Wei Chang, and Rahul Gupta. 2021. Bold: Dataset and metrics for measuring biases in open-ended language genera- tion. In Proceedings of the 2021 ACM conference fairness, accountability, and transparency, pages 862-872.
Henrico Bertini Brum and Maria das Graças Volpe Nunes. 2017. Building a sentiment corpus of tweets in brazilian portuguese. arXiv preprint 17200217..
Carrino Casimiro Pio, Costa-jussa Marta R., and Fonol- losa Jose A. R. 2019. Automatic Spanish Translation of the SQuAD Dataset for Multilingual Question An- swering. arXiv e-prints, page arXiv: 1..2.02200vl.
Martin d' d'Hoffschmidt, Wacim Belblidia, Quentin Heinrich, Tom Brendlé, and Maxime Vidal. 2020. FQuAD: French question answering dataset. In Find- ings of the Association for Computational Linguistics: EMNLP 2020, pages 1193-1208, Online. Association for Computational Linguistics.
Hyung Won Chung, Dan Garrette, Kiat Chuan Tan, and Jason Riesa. 2020. Improving multilingual models with language-clusten cluustrrd vocabularies. Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4536 4546, Online. Association for Computational Linguistics.
Manuel C Díaz-Galiano, Eugenio Martfnez-Cámara, M Ángel Gareía Cumbreras, Manuel Gareía Vega, and Julio Villena Román. 2018 The democratization of deep learning in tass 2017. Procesamiento del Lenguaje Natural, 60:37-44.

---

Pavel Efimov, Andrey Chertok, Leonid Boytsov, and Pavel Braslavski. 2020. Sberquad-russin reading comprehension dataset: Description and analysis. In Internationaa Conference of the Cross-Language Evaluation Forum for European Languages, pages 3-15. Springer.
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad. Abdelrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequeneceto-sseqeeee pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Länguistics, pages 7871-7888 Online. Association for Computa- tional Linguisties.
Kawin Ethayarajh and Dan Jurafsky. 2020. Utility is in the eye of the user: A critique of NL.P leaderboards. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4846-4853, Online, Association for Computa- tional Linguistics.
Davis Liang, Hila Gonen, Yuning Mao, Rui Hou, Na- man Goyal, Marjan Ghazvininejad, Luke Zettle- moyer, and Madian Khabsa. 2023. XIm-v: Overcom- ing the vocabulary bottleneck in multilingual masked language models. arXiv preprini arXiv:2301.1047
Naman Goyal, Jingfei Du, Myle Ott, Giri Ananthara- man, and Alexis Conneau. 2021. Larger-scale trans- formers for multilingual masked language modeling. In Proceedings of the 6th Workshop on Represen- lation Leurning for NLP (RepLANLP-2021), pages 29-33, Online. Association for Computational Lin- guistics.
Sheng Liang. Philipp Dufter, and Hinrich Schütze. 2020. Monolingual and multilingual reduction of gender bias in contextualized representations. In Proceed- ings of the 28th International Conference on Com- putational Linguisties, pages 5082-5093 Barcelona, Spain (Online). International Committee on Compu- tational Linguistics.
Masahiro Kaneko and Danushka Bollegala. 2021. De- biasing pre-trained contextualised embeddings. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Lin- guistics: Main Volume, pages 1256-1266 Online. Association for Computational L Linguistics.
Seungyoung Lim, Myungji Kim, and Jooyoul Lec. 2019. Korquad OO: Korean qa dataset for machine reading comprehension. arXiv preprint arXiv: (90::099900..
Yinhan Liu, Jiatao Gu. Naman Goyal, Xian Li. Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre- training for neural machine translation. Transac- tions of the Association for Computational Linguis- tics, 8:726-742.
Masahiro Kaneko and Damushka Bollegala. 2022. Un- masking the mask-evaluating social biases masked language models. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 36, pages 11954-11962.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint 1907. 16692.
Masahiro Kaneko, Aizhan Imankulova, Danushka Bol- legala, and Naoaki Okazaki. 2022. Gender bias in masked language models for multiple languages. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Länguistics: Human Language Technologies. pages 2740-2750, Seattle, United States. Association for Computational Linguistics.
Kelly Marchisio, Patrick Lewis, Yihong Chen, and Mikel Artetxe. 2022. Mini-model adaptation: Ef- ficiently extending pretrained models to new lan- guages via aligned shallow training. arXiv preprint 12.
Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019. Bert: Pre-training of deep bidirec- tional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171-41888
Chandler May, Alex Wang, Shikha Bordia, Samuel Bow- man, and Rachel Rudinger. 2019. On measuring so- cial biases in sentence encoders. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 622-628.
Svetlana Kiritchenko and Saif M Mohammad. 2018. Examining gender and race bias in two hundred sen- timent analysis systems. NAACL HLT 2018, page 43.
Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black, and Yulia Tsvetkov. 2019. Mcasuring bias in contex- tualized word representations. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 166-172.
Benjamin Muller, Antonios Anastasopoulos, Benoît Sagot, and Djamé Seddah. 2021. When being un- seen from mBERT is just the beginning: Handling new languages with multilingual language models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 448-462, Online. Association for Computa- tional Linguistics.
Guillaume I Lample, Alexis Conneau, Marc' Aurelio Ran- zato, Ludovie Denoyer, and Hervé Jégou. 2018. Word translation without parallel data. In Interna- tional Conference on Learning Representations.

---

Moin Nadeem, Anna Bethke, and Siva Reddy, 2021. StereoSet: Measuring stereotypical bias in pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 5356-5371, Online. Association for Computational Linguistics.
Timo Schick. Sahana Udupa, and Hinrich Schüizze. 2021. Self diagnosis and self-debiasing: A proposal for re- ducing corpus-based bias in NI.P. Transactions of the Association for Computational Linguistics, 9:1 9:1408- 1424.
ByungHoon So, Kyuhong Byun, Kyungwon Kang, and Seongjin Cho. 2022. Jaquad: Japanese question an- swering dataset for machine reading comprehension. arXiv prepriat arXiv:2202.01720
Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020. CrowS-pairs: A chal- lenge dataset for measuring social biases in masked language models. Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 1953-1967, Online. As- sociation for Computational Linguistics.
Asahi Ushio, Fernando Alva-Manchego, and Jose Camacho-Collados, 2022. Generative language mod- els for paragraph-Ievel question generation. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 670-688, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Malte Ostendorff and Georg Rehm. 2023. Effi- cient language model training through cross-lingual and progressive transfer learning. arXiv preprint arXiv:2301.09626.
Asahi Ushio, Fernando Alva-Manchego, and Jose Camacho-Collados 2023. A practical toolkit for mul- tilingual question and answer generation. In Proceed- ings of the 61th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, Toronto, Canada. Association for Computational Lin- guistics.
Jonas Pfeiffer, Naman Goyal, Xi Lin, Xian Li, James Cross, Sebastian Riedel, and Mikel Artetxe. 2022. Lifting the curse of multilinguality by pre-training modular transformers. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Seattle,
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018, Glue A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNI.P. Analyz- ing and Interpreting Neural Networks for NLP, pages 353-355.
Language Technologies, pages United States. Association for Computational Lin- guistics.
Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proceed- ings of the 57th Annual Meeting of the Association for Computational Linguisties. pages 9006, Flo- rence, Italy. Association for Computational Linguis- tics.
Hai Wang, Dian Yu, Kai Sun, Jianshu Chen, and Dong Yu. 2019. Improving pre-trained multilingual model with vocabulary expansion. In Proceedings of the 23rd Conference on Computational Natural Lan- guage Learning (CoNLL), pages 316-327, Hong Kong, China. Association for Computational Lin- guistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. The Journal of Machine Learning Research, 21(1):5485-555 55511
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 483-498, On- line. Association for Computational Linguistics.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Bo Zheng, Li Dong, Shaohan Huang, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, and Furu Wei. 2021. Allocating large vocabulary capacity for cross- lingual language model pre- pre-training. In Proceedings of the 2021 Conference on Empirical Methods in Nat- wral Language Processing, pages 3203-3215, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Sara Rosenthal, Noura Farra, and Preslav Nakov. 2017. SemEval-2017 task 4: Sentiment analysis in Twitter. In Proceedings of the 11th International Workshop on Semantic Evaluation SeemEval-2017 pages 502- 518, Vancouver, Canada. Association for Computa- tional Linguistics.
Teven Le Scao, Angela Fan, Christopher Akiki, EI- lie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b- parameter acen-access multilingual language model. arXiv preprini 221.02221..55000.
Yi Zhou, Masahiro Kaneko, and Danushka Bollegala. 2022. Sense embeddings are also biased evaluating social biases in static and contextualised sense em- beddings. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics

---

Vocah. No-Trim (250K)
5K
10K
15K
30K
60K
Param.
278M
89M
93M
97M
109M
132M
Post- FT (Sentiment)
AR
66.3
64.5
64.5
65.9
65.9
DE
73.2
70.4
72.4
72.1
73.7
73.3
EN
68.4
64.0
66.5
67.4
68.6
68.5
ES
69.0
66.2
67.2
67.8
68.4
69.0
FR
68.6
71.4
71.7
IT
62.9
60.8
61.9
63.5
62.3
62.9
PT
70.7
63.6
65.9
68.2
69.4
70.8
Pre- (Sentiment)
66.3
58.6
61.9
63.1
63.2
DE
73.2
70.6
71.8
73.1
71.5
71.7
EN
68.4
64.6
66.6
67.7
66.0
68.4
69.0
60.4
65.6
66.3
65.9
63.0
FR
71.8
70.3
74.3
73.8
72.2
74.0
IT
62.9
66.2
67.1
68.1
68.2
65.1
65.5
67.5
69.6
Post-FT (NLI)
AR
75.7
75.0
75.8
75.8
75.7
76.6
78.9
79.6
EN
84.6
80.5
83.6
84.3
84.6
84.6
ES
79.8
75.3
78.4
79.4
79.8
79.8
FR
80.1
77.2
80.0
80.1
80.2
80.1
PIe-FT (NLI)
AR
75.7
73.0
75.0
75.4
75.7
DF.
79.9
77.4
79.0
79.0
79.6
79.7
EN
84.6
83.3
84.7
84.4
85.1
ES 79.8
78.9
79.1
79.2
81.0
79.7
FR 80.1
77.3
78.9
78.2
80.1
78.7
Table 4: Results of XLM-R for sentiment analysis (macro FI) and NLI (accuracy) with different top-n vocabulary size VT, where the best results each LM and language are in the bold characters.
(Volume J: Long Papers), pages 1924-1935, Dublin, Ireland. Association for Computational Linguisties.
A Top-n VT of XLM-R
Table 4 shows the results of XLM-R fine-tuned on sentiment and NLI with post/pre-Va VT for different top-n.
B Top-n VT of mT5
Table 5 shows the results of mT5 fine-tuned QA and QG with post/pre-VT VT for different top-n.
Details of Results on Social Bias Evaluation
Table 6 shows the details of social bias evalua- tion (EEC dataset) regarding each emotion type ob- served the LMs fine-tuned on sentiment analysis. Table 7 shows the details of social bias regarding each bias type in both CP and SS datasets observed in the comparison LMs.

---

Vocab.
No- Noimm (250K)
5K
10K
ISK
30K
60K
90K
120K
Param.
300M
49M
54M
59M
74M
105M
136M
166M
70.1 55.5
72.4 59.7
73.2 60.5
73.7 60.9
72.7
58.8
71.0 56.4
70.5/55.x
70.3 55.6
Post-FT (QA)
ES
55.9 34.7
53.5 32.5
54.5/ 54.4/33.1
55.2 / 33.8
56.0 /34.8
55.9 /4.8
34.7
FR
50.0/300
/
29.9
29.6
49.9 30.1
49.8 30.5
50.2/30.9
50.0 30.8
33.2/37.6
/4.6
53.8/38.4
/8.8
54.2 38.5
33.7
53.4/37.8
JA
65.7 / 65.7
54,4
60.3
62.6
60.4 60.4
64.9 / 64.9
65.6/ 63.6
65.7 65.7
KO
77.1 70.6
75.0/
7
76.9 /9.3
77.7
RU
51.4
70.1/48.8
/4.5
70.3 44..6
70.3 /47.9
73.8/51.1
73.9
71.8
EN
23.8/ 90.0
23.6/ 89.8
24.1/89.9
24.2/90.0 99.0
24.2 / 90.0
23.9/90.1
90.0
23.9 90.0
Post- (QG)
ES
84.1
83.6
83.8
83.9
22.4/ 84.0
22.7 84.1
22.7 84.1
22.7 /4.1
17.5 80.7
17.1/80.3
17.1/80.4
17.3/80.5 80.5
17.3 80.5
17.4/ 80.6
17.5 80.77
/ 80.7
IT
17.6 17.6 80.8
80.7
17.5/80.8
17.5/804 80.8
17.6 80.7
17.6 80.8
17.6 80.8
JA
29.0 80.9
77.7/79.2
80.4
28.2 80.5
28.8 / 80,9
29.0 80.9
29.0 /8.9
29.0 80.9
KO
27.5 82.9
27.3/82.8
27.9/82.9
27.9/82.9
27.4/ 82.9
27.5/83..
RU
26.4/ 84.3
26.0/ 84.0
26,2/844
26.3/ 84.2
26.3 / 84.2
26.3 84.3
26.4/ 84.3
26.4/ 84.3
EN
70.1 555.5
61.
60.4
/
56.3/
08.6/54..
/
Pre-FT (QA)
ES
35.9
56.7 36.7
58.1/37.5
42.2/41.4
59.8 /40.
58.0/37.0
58.9/38.1
32.7
FR 30,0/30.9
49.0/32.7
47.6/30.2
23.
44.9/27.4
34.3//0.0
43.1/24.1
43.0/
IT 53.2/37.6
57.6/43.7
61.5/46.8
61.2/45.6
57.8/42.1
56.5/40.5
35.4/
65.7
55.3/55.3
52.2/52.2
61.3
60.4/60.4
64.6
66.9
67.2
KO 70.1
79.5/72.6
83.7/77.77
82.7 /76.4
80.4 73.7
81.6/75.1
RU
51.7
73.5/5177
76.9
77.4/ 56.9
75.9
75.1 /3.00
75.0/53.0
73.3 51.4 51.4
EN
23.8/90.0 90.0
24.8/ 90.0
24.4/ 89.9
24.3 89.9
24.2 / 90.0
24.2 90.0
23.3 89.9
23.6 89.8
ES
22.7 / 84.1
21.91 84.1
83,9
83.7 22.7 /3.77
22.0/ 84.3
22.6/ 84.4
70.4/79.6
22.6/ 84.1
FR
17.5 80.7
76.1
16.9
16.97795.
15.8 78.8
14.6/78.0
15.7
17.0/
IT
17.6 80.8
17.4 80.4
17.8 17.8/8111
18.0 80.8
17.1 80.6
17.4 80.9
17.4/80.8 80.8
Pre-FT
JA
29.0 80.9
27.0 26.6/79.8
79.7
27.3/793
27.7 80.3
28.8
28.2 80.2
KO
27.5
82.9
82.9
23.4/83.4
27.8 83.4
/4.1
/8.8
28.4/ 83.4
26.4/ 84.3
27.7 85.9
30.0 86.6
29.0 86.5
29.2
29.2 86.7
28.9 86.0
Table Results of mT5 for QA (Ans-FI/EM) and QG (MTR/BS) with different top-n vocabulary size VT, where the best results in each LM and language in the bold characters.
Model
AULA-EEC
Anger
Sadness
Fear
Joy
No Emotion Type
FT RoBERTa
64,8
50.0
67.8
71.5
64.3
60.8
Gender Bias
FT XLM-R
44.3
46.9
39.4
51.4
34.4
39.7
Pre-FT XLM-R (EN)
42.5
33.8
55.8
45.1
40.7
42.7
Post-FT XLM-R (EN)
44.3
46.9
39.4
51.4
34.4
39.7
Pre-FT XLM-R (50k)
44.3
46.9
39.4
51.4
34.4
39.7
Post-FT XLM-R (50k)
41.0
41.2
45.8
47.2
36.9
37.2
FT RoBERTa
85.7
50.0
88.0
65.4
50.0
100.0
Race Bias
FT XLM-R
62.0
74.5
32.1
47.3
74.6
17.4
Pre-FT VT XLM-R (EN)
56.9
33.7
58.8
69.2
60.1
50.0
Pre-FT VT XLM-R (EN)
62.0
74.5
32.1
47.3
74.6
17.4
Pre-FT VT XLM-R (50k)
62.0
74.5
32.1
47.3
74.6
17.4
Post-FT VT XLM-R (50k)
62.4
55.7
70.3
70.9
59.1
59.0
Table 6: Social bias scores of LMs fine-tuned on sentiment amalysis on each emotion type w/wo VT on the EEC dataset.

---

RoBERTa
XLM-R
VT XLM-R (EN)
VT XLM-R (50K)
AULA
58.1
49.5
49.5
49.3
Age
59.8
49.4
49.4
49.4
Disability
68.3
66.7
66.7
68.3
CP Dataset
Gender
53.4
48.5
48.5
48.1
Nationality
56.6
44.0
44.0
44.7
Physica--Apeearane
52.4
58.7
58.7
55.6
Race-Color
56.8
43.8
43.8
44.0
Religion
53.3
54.3
54.3
53.3
Sexual-Orientatie
67.9
54.8
54.8
54.8
Socioeconomic
66.3
58.1
58.1
57.6
AULA
58.8
54.9
54.9
55.0
SS Dataset
Gender
62.4
55.3
55.3
54.9
Profession
60.4
54.3
54.3
54.6
Race
57.0
56.0
56.0
56.0
Religion
54.4
46.8
46.8
46.8
Table Social bias scores of VT LMs compared their original ones on both CP and SS datasets for each bias type.


Reviews:
[{"review_id": "qbGBfTcjY7", "reviewer": "Reviewer_XE1g", "summary": "", "questions": "- It's unclear why pre-FT VT reduces finetuning time -- if a language can be modeled with a subset of vocabulary V, V_subset, then finetuning would theoretically have the same computational complexity since we only use a subset of the vocabulary anyways (assuming V_subset contains all the tokens we need for that particular language). Is it just the extra time to perform the embedding lookup?\n- Is there a performane drop for pre-FT VT, since you can't leverage translate-train-all or cross-lingual transfer from high resource languages?\n- Have the authors tried this method for ultra-low-resource languages and on datasets like MasakhaNER?\n- What is the size of the per-language text corpora used to perform vocabulary retention?\n- Do the authors see a marked increase in OOV after applying this technique?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We found the reviews highly useful, and really appreciate the effort you put in writing those comments!\n\n[Reason 1] This method essentially removes the capability of the model to do well in languages other than the intended language. While theoretically sound and would work well on academic datasets, real world text is rife with code-switching and text in multiple languages. This could limit the capabilities of this method.\n[Response] Thank you for the useful feedback. We have not tested our method on a dataset with code-switching, so we will add the note to the limitation section. Indeed, to apply vocabulary-trimming on such a corpus consisting of texts from different languages, is a decent direction for future work.\n\n\n[Reason 2] Much of the benefits of this method could be potentially covered by just loading the vocabulary embeddings into CPU memory and pushing the required vocabulary embeddings per batch into GPU memory on-the-fly. Have the authors compared the inference latency increases using this method?\n[Response] That approach should save the GPU memory at the fine-tuning, but the model is still the same size as the original model. Our approach is not only aiming at increasing the training efficiency but also reducing the cost of hosting the model afterward. Since the vocabulary trimming can reduce the model size largely, hosting the model requires less storage and memory to the server. The CPU core can be less powerful too, as the inference takes less time, mainly because the complexity of the lookup over the vocabulary (embedding to token/token to embedding) is reduced by the smaller size of the vocabulary.\n\n\n[Q1] It's unclear why pre-FT VT reduces finetuning time -- if a language can be modeled with a subset of vocabulary V, V_subset, then finetuning would theoretically have the same computational complexity since we only use a subset of the vocabulary anyways (assuming V_subset contains all the tokens we need for that particular language). Is it just the extra time to perform the embedding lookup?\n[Response] Regarding the fine-tuning, the main benefit is from the reduced time in the embedding lookup, as mentioned.\n\n\n[Q2] Is there a performane drop for pre-FT VT, since you can't leverage translate-train-all or cross-lingual transfer from high resource languages?\n[Response] In general, VT can achieve competitive results as the original model, but related to our response to the [Reason 1], the tasks we considered in our paper are all mono-lingual with less code-switching. If we applied the method to the task with mixed languages, VT would limit the performance due to the lack of cross-lingual understanding caused by VT.\n\n\n[Q3] Have the authors tried this method for ultra-low-resource languages and on datasets like MasakhaNER?\n[Response] As we mention in the limitations section, our study is based on high-resource languages at the moment.\n\n\n[Q4] What is the size of the per-language text corpora used to perform vocabulary retention?\n[Response] We rely on the validation split of the mC4 https://aclanthology.org/2021.naacl-main.41/, the training dataset of mT5. It contains 3,083K (en), 416k (es), 331k (fr), 15.8k (ko), 87.4k (ja), 53k (ar), 186k (it), 399k (de), and 169k (pt).\n\n\n[Q5] Do the authors see a marked increase in OOV after applying this technique?\n[Answer] Within the experiments we conducted in the paper, OOV errors are less prominent. In fact, the tokenizer rolls back unknown vocabularies into smaller chunks, so it should not be an issue as long as the text is in the target language. English words, as an example, would split into a sequence of alphabet at the finest, which are often part of the vocabulary after the trimming. Moreover, we trim the vocabulary based on the frequency of them in the language-specific corpus, so in theory, this already reduces the risk of removing important vocabularies, and avoid raising OOV in practice.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This method can significantly reduce the memory footprint of multilingual models, especially smaller multilingual models where a large portion of the parameters are tied to the vocabulary embedding lookup table.\n- While the capabilities are of this method are mostly tied to memory reduction", "reasons_to_reject": "- This method essentially removes the capability of the model to do well in languages other than the intended language. While theoretically sound and would work well on academic datasets, real world text is rife with code-switching and text in multiple languages. This could limit the capabilities of this method.\n- Much of the benefits of this method could be potentially covered by just loading the vocabulary embeddings into CPU memory and pushing the required vocabulary embeddings per batch into GPU memory on-the-fly. Have the authors compared the inference latency increases using this method?", "questions_to_authors": "- It's unclear why pre-FT VT reduces finetuning time -- if a language can be modeled with a subset of vocabulary V, V_subset, then finetuning would theoretically have the same computational complexity since we only use a subset of the vocabulary anyways (assuming V_subset contains all the tokens we need for that particular language). Is it just the extra time to perform the embedding lookup?\n- Is there a performane drop for pre-FT VT, since you can't leverage translate-train-all or cross-lingual transfer from high resource languages?\n- Have the authors tried this method for ultra-low-resource languages and on datasets like MasakhaNER?\n- What is the size of the per-language text corpora used to perform vocabulary retention?\n- Do the authors see a marked increase in OOV after applying this technique?", "paper_topic_and_main_contributions": "Authors propose a simple vocabulary trimming technique that removes tokens from the vocabulary of multilingual LMs by identifying language-specific tokens and retaining those. Given a large enough text corpus (on which to perform token retention), theoretically there should be no drop in model performance.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4m6NZAgQLb", "reviewer": "Reviewer_9JRE", "summary": "", "questions": "- Why report only accuracy for NLI?\n- Please report effect of VT on low resourced languages.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "[Reason 1] No analysis of tradeoff between extracting monolingual LM and potentially losing helpful cross-lingual transfer.\n[Response] We focus on adapting multilingual LMs to a specific language, in other word, localising multilingual LM to a single target language, to perform well on that language, while reducing the model size largely. We focus on monolingual tasks, rather than multilingual or cross-lingual application. Certainly, the trimmed LMs are no longer useful for cross-lingual/multilingual application, since we drop the vocabularies not used in the target language, but that is not what we care most about in our paper.\n\n\n[Reason 2] Given that it is a long paper, it is missing important analysis such as effect of monolingual corpora domain and size, result on resourced languages and effect of % of target language composition in original multilingual corpora.\n[Response] We focus on relatively high-resource languages in our paper, where the portions of those languages in the pre-training corpus of multilingual LMs are large. Within such high-resource languages, we assume that the difference among the languages in terms of the size for the pre-training corpus, has limited effect on the performance. In fact, our method works well among all the languages we considered in the paper. Perhaps, in a low-resource language, the performance could correlate with the size of that language-specific texts in the pre-training corpus of the multilingual LMs, yet it is out of our scope as mentioned in the limitations section.\n\n\n[Reason 3] Only evaluate on English\n[Response] Our study deals with four multilingual language models (mT5/mBART/XLM-R/XLM-V), and we have 7 languages (en/es/fr/ko/de/ja/ru) for QA/QG task, as well as 6 languages (ar/de/en/es/it/pt) for NLI/sentiment analysis. We would appreciate if the reviewer could elaborate the reason in detail.\n\n\n[Reason 4] Did not evaluate on larger LMs.\n[Response] As we mentioned explicitly in our limitations section, the application of our method on the larger LM such as BLOOM and mT5-XXL is out of the scope for our paper.\n\n\n[Q1]: Why report only accuracy for NLI?\n[Answer] We rely on accuracy, because it is the metric used in the original paper. We do not study NLI but we consider NLI as one of the applications, so it should not be a problem to report a standard metric used in the task setting.\n\n\n[Q2]: Please report effect of VT on low resourced languages.\n[Answer] Thank you for the suggestion. As we mentioned explicitly in our limitations section, the application of our method on the low-resource language is out of the scope for our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Fairly efficient method because you can potentially extract several monolingual LMs from a single multilingual LM.\n-", "reasons_to_reject": "- No analysis of tradeoff between extracting monolingual LM and potentially losing helpful cross-lingual transfer. \n- Given that it is a long paper, it is missing important analysis such as effect of monolingual corpora domain and size, result on resourced languages and effect of % of target language composition in original multilingual corpora. \n- Only evaluate on English\n- Did not evaluate on larger LMs.", "questions_to_authors": "- Why report only accuracy for NLI?\n- Please report effect of VT on low resourced languages.", "paper_topic_and_main_contributions": "This paper introduces vocabulary trimming (VT) which aims to reduce a multilingual LM to a monolingual LM. The method involves trimming the vocabulary of a multilingual LM, using a target language's monolingual corpus, to include only tokens relevant to the target language. They consider applying VT before or after finetuning, and show the tradeoffs between both methods. Furthermore, they compare a trimmed multilingual model to a monolingual model trained from scratch and show that the trimmed model has less bias.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "frrb5tLalt", "reviewer": "Reviewer_dgGG", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "[Reason 1] From reading the introduction and section 4, I was unclear why VT was an improvement over simply using an existing monolingual model. It would be good to have a better justification/explanation in the introduction and/or a comparison with monolingual models in the analysis in section 4\n[Response] As we mentioned in the introduction, the reasons why we need VT are two-fold. First, multilingual LMs are shown to be competitive to its monolingual counterpart (XLM-R, XLM-V, mT5). Second, we do not have mono-lingual LMs in many languages, especially when it comes to encoder-decoder LMs (e.g. T5). Accordingly, we need to fine-tune multilingual LMs in many cases, but they are often larger than its monolingual counterpart due to the multilingual embedding matrix. Here, we propose VT to remove unused vocabulary to make the multilingual LM as small as the monolingual counterpart, when you fine-tune it in a specific language. We unfortunately do not have a result more than English in section 4.\n\n\n[Reason 2] On lines 103-4, the authors say it is a \"natural question\" to ask if VT impacts bias levels. As someone unfamiliar with this field, I found the switch to VT's impact on bias jarring. It would help if the authors could make it clearer how they made this jump e.g. comparison with existing literature\n[Response] As we mentioned in line 97-103, prior works have shown that pretrained language models contain worrying level of social bias (May et al., 2019; Kurita et al., 2019; Kaneko and Bollegala, 2021). In addition, Kaneko et al. (2022) showed that the level of social bias in masked language models increases during fine-tuning. For this reason, we investigate if VT has impacts on social bias in language models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The paper presents extensive empirical work backing up their claims covering multiple high-resource languages.\n2) Their figures are particularly clear and helpful for explaining their method\n3) Their method is well-situated in the current literature and they cover a wide range of related work", "reasons_to_reject": "1) From reading the introduction and section 4, I was unclear why VT was an improvement over simply using an existing monolingual model. It would be good to have a better justification/explanation in the introduction and/or a comparison with monolingual models in the analysis in section 4\n2) On lines 103-4, the authors say it is a \"natural question\" to ask if VT impacts bias levels. As someone unfamiliar with this field, I found the switch to VT's impact on bias jarring. It would help if the authors could make it clearer how they made this jump e.g. comparison with existing literature", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper looks at the problem of compressing the model size of a multilingual language model when its intended downstream use is on a monolingual task. The method the authors present is called vocabulary trimming (VT), which involves deleting tokens from the vocabulary which do not feature in the target language. \n\nThe main contributions of this paper are the following:\n\n- Explanation of VT, a method for removing tokens from the multilingual vocabulary which are not relevant to the target language. \n- Extensive empirical work showing that VT has a minimal impact on downstream monolingual tasks (question answering, question generation, sentiment analysis, natural language inference)\n- Additional empirical work exploring the impact of vocabulary size and when VT is carried out\n- Exploration of VT as a debiasing technique", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- line 1: missing bracket after \"(LMs)\"\n- Figure 4: graphs are too small to interpret easily. Perhaps these results would be easier to understand in a table?\n- Table 1: having two metrics side by side was confusing and overloaded the table. Given they both say roughly the same thing, perhaps just pick one and put the full results for the other in the appendix\n- Tables 1, 2, 3: There is too much information here to parse easily. Given the main point here is that there is not much change over the baseline, a delta might be easier to understand than a raw result."}}]

Author rebuttal:

Meta-review: The reviewers found the paper to be sound, and appreciated the experimental results, such as the result that the vocabulary-trimmed multilingual model shows less bias than a monolingual model trained from scratch.

However, the reviewers ultimately questioned the value of the contribution since 1) it may excessively harm the model in real-world settings, where data is not always cleanly in a single language, and 2) the memory footprint could also be reduced simply by avoiding naively loading the entire embedding table into memory.
Decision: Accept-Findings