Adapters for Altering LLM Vocabularies: What Languages Benefit the Most?
Authors: HyoJung Han
Akiko Eriguchi
Haoran Xu
Hieu Hoang
Marine Carpuat
Huda Khayrallah
Venue: ICLR 2025 Poster (2025)
Keywords: Vocabulary Adaptation
Vocabulary Transfer
Tokenizer Transfer
Initializing Embedding
Adapter
Multilingual
Machine Translation
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We propose VocADT, a novel method for vocabulary adaptation using adapter modules, and explore what language benefits the most among with various scripts, resource availability, and fragmentation.

Abstract: Vocabulary adaptation, which integrates new vocabulary into pre-trained language models, enables expansion to new languages and mitigates token over-fragmentation. However, existing approaches are limited by their reliance on heuristics or external embeddings. We propose VocADT, a novel method for vocabulary adaptation using adapter modules that are trained to learn the optimal linear combination of existing embeddings while keeping the model’s weights fixed. VocADT offers a flexible and scalable solution without depending on external resources or language constraints. Across 11 languages—with diverse scripts, resource availability, and fragmentation—we demonstrate that VocADT outperforms the original Mistral model (Jiang et al., 2023) and other baselines across various multilingual tasks including natural language understanding and machine translation. We find that Latin-script languages and highly fragmented languages
benefit the most from vocabulary adaptation. We further fine-tune the adapted model on the generative task of machine translation and find that vocabulary adaptation is still beneficial after fine-tuning and that VocADT is the most effective.

Published as a conference paper at ICLR 2025
ADAPTERS FOR ALTERING LLM VOCABULARIES: WHAT LANGUAGES BENEFIT THE MOST?
HyoJung Han
Akiko I. Eriguchi
Haoran Xu
University of Maryland hjhanêcs, und edu
Microsoft
Microsoft
ak koe@mi crosoft.
haoranxu@mi crosoft.com
Hieu Hoang
Marine Carpuat
Huda Khayrallah
Microsoft
University of Maryland marine@cs. und. edu
Amazon
sihcan@microsoft.ccm
hudakh amazon, com
ABSTRACT
Vocabulary adaptation, which integrates new vocabulary into pre-trained lan- guage models, enables expansion to new languages and mitigates token over- fragmentation. However, existing approaches are limited by their reliance on heuristics or external embeddings. We propose VocADT, novel method for vo- cabulary adaptation using adapter modules that are trained to learn the optimal linear combination existing embeddings while keeping the model's weights fixed. VocADT offers flexible and scalable solution without depending on exter- nal resources or language constraints. Across 11 languages witth
with diverse scripts, resource availability, and fragmentation- we demonstrate that VocADT outper- forms the original Mistral model (Jiang et al.) (203) and other baselines across various multilingual tasks including natural language understanding and machine translation. We find that Latin-script languages and highly fragmented languages benefit the most from vocabulary adaptation. We further fine-tune the adapted model on the generative task of machine translation and find that vocabulary adap- tation is still beneficial after fine-tuning and that VocADT is the most effective
INTRODUCTION
Vocabulary adaptation (or transfer) process of modifying a pre-trained language model (LM) to use new vocabulary-offee several key advantages. First, enables the introduction of new lan- guages into model, increasing flexibility in handling linguistic diversity and improving downstream
performance in target languages (Wang et al. 2020 Gogoulou et al.] [002]] Downey et al., 2023). Second, it reduces over- vverfrragmenttton. where words are excessively split by the tokenizer. slowing down generation and impairing performance in certain languages (Ahia et al. 2023 Petrov et al, 2023 Yamaguchi et al.) 2024). These benefits have led to the development of numerous vocabulary adaptation approaches that initialize the new embeddings of new vocabulary with various methods
based heuristics (Mosin al. 2023 Gee al., 2022; Downey al., 2023), external resources Liu et 2024), or a separate hypernetwork that generates Minixhofer (024). They generally generate new embeddings using original embeddings, op-
Tran 2020) Dobler & de Melo [203]]
lionally followed by continued training to finalize the adaptation (Minixholer et al. 2022 Ostendorll
Rehm
2023
Dobler & de Melo 2023 Liu et al. 2024).
However, existing vocabulary adaptation approaches face several limitations. Those that rely on heuristics (Gee et al. 2022) Downey et al., (2023), which use predefined rules to initialize new em- beddings from existing ones rather than learning from data, often lack adaptability, where the new embeddings are not fully integrated into the original model and require an additional training phase of full- aill-weight updates to fully adapt to the new vocabulary. Also, those that depend on external em- beddings or networks (Tran 2020
Dobler & de Melo 2023
Liu et al. 2024), increase complexity
Work done Microsoft.
Prroject page: httphhttt////.. / com/h- jar//hjj-haae hathh/n//ooo Models at Hugg ng face Hub Standard transformer decoding quadratic sequence length, length increases can he catastrophic.

---

Published as a conference paper at ICLR 2025
updated weights
fixed weights
Adaptor
Vocab Adapter
Original Emb
Out eimb
New Body
X
0.1
0.5
Body
Body
0.5
0.3
0.1
emb
embb
emb
0.4
Vocab Adapter
the sub #iward
the
Subward
training
Adapted LM with new vocab
Original
new
Original Vocab
New Vocab
(a) Overview of the vocabulary adaptation and training.
(b) Initialization of vocabulary adapter.
Figure 1: Overview of our vocabulary adaptation with adapter (VocADT) and the initialization of
adapter. The vocabulary adapter modules are trained to adapt new vocabulary with existing embed- dings while keeping the original model fixed. We initialize entries of the adapter for overlapping tokens and tokens whose partitions are in the original vocabulary. Once trained, the adapters and original embeddings are merged to form the new embeddings.
and limit scalability. Furthermore, some approaches focus soiely on language-specinn cases may have restrictions on the number of languages in the implementation when configuring the vocabulary (Dobler & de Melo 2023 Minixhofer et al. 2024).
Additionally, we still know little about the impact of vocabulary adaptation across diverse linguistic and task settings. Most prior work investigates few languages. which is insufficient for identify- ing patterss (Tran 2020 Ostendorff & Rehm 2023; Remy et al.] [023] Yamaguchi et al.) 2024),
while studies that consider broader range of languages only report averages without detailed anal-
ysis (Liu et al.] 2024 Mundra et al. 2024). Furthermore, the impact of vocabulary adaptation on cross-lingual and generative tasks like machine translation (MT) is understudied, even though they represent crucial application areas for porting models to new languages. Many adaptation methods (Chung et al. 2020 Gee et al., 2022, Liu et al. 2024) have been evaluated instead on non-cross- lingual and discriminative tasks such as commonsense reasoning, natural language inference (NLI), or question answering (QA) are typically classification tasks.
We propose VocADT, novel solution for vocabulary adaptation using adapters, designed to ad- dress key challenges in existing approaches (Figure We introduce a vocabulary adapter module, a learnable matrix between the new vocabulary and the original embeddings of language model. The module gradually adapts to new vocabularies through training while keeping all weights of the original model fixed, allowing the module to learn the best combination of the original embed- dings without relying on heuristics, external embeddings, or dictionaries. This learned adaptation approach offers better adaptability of new embeddings to the original language model with only its embeddings replaced and more flexibility in the number of languages while removing the necessity of external pre-trained resources. At the end of training, the adapter is merged with the original embeddings to create new embedding matrix.
In addition our novel method, we empirically address the following key questions to understand the effectiveness and behavior of vocabulary adaptation: (1) Which languages benefit most from vo- cabulary adaptation? (2) What are the best strategies for creating new vocabularies? Also, is script consistency necessary? (3) How does vocabulary adaptation impact machine translation? We empha- size this task as it is a critical task for multilingual models that involves cross-lingus and generative capabilities, which are often more complex than classification or monolingual tasks.
We demonstrate the effectiveness of our adaptation method on various NLP tasks spanning Natural Language Understanding and MT. Results show that our approach consistently surpasses the origi- nal Mistral model most cases, both after the adaptation phase and following phase of full-weight training. Additionally, our method outperforms or matches other strong vocabulary adaptation base- lines. Our findings indicate that Latin-script languages and those with severe fragmentation benefit the most from vocabulary adaptation. Finally, while all vocabulary adaptation methods continue to be effective for machine translation after fine-tuning, VocADT shows the best results among them. Our main contributions are summarized as follows:
2

---

Published as a conference paper at ICLR 2025
We propose VocADT, simple and effective solution for vocabulary adaptation using adapters, that addresses key limitations in prior work such as reliance on external embed- ding or language constraints.
We conduct experiments that cover a wide range of languages and scripts, finding that languages with Latin scripts or severe fragmentation benefit the most and that having a consistent grouping of scripts for multilingual vocabulary is helpful.
Our approach consistently outperforms the original language model and is more effee- tive than, or on par with, strong vocabulary adaptation baselines after the adaptation phase across various tasks and after the following full-weight fine-tuning on MT.
BACKGROUND AND MOTIVATION
Approaches Vocabulary Adaptation Prior work focuses on initialization strategies for the new vocabulary embeddings, before continuing training with unlabeled target language text using the original self-supervised pretraining objective. For instance, FOCUS (Dobler & de Melo 2023) initializes embeddings weighted combination of overlapping tokens using external embeddings for non-overlapping ones, while copying the embeddings of overlapping tokens. OFA (Liu et al.) 2024) also relies on external word vectors to initialize embeddings for non-shared new tokens, using a weighted average original tokens based on semantic similarity. This strategy often requires ex- ternal resources such auxiliary embeddings (Dobler & de Melo 2203) Liu et al. [024 Ostendorff & RRhh 2023) or bilingual dictionaries (Mundra et al. 20244 Minixhofer et al. 2022).
After initialization, language adaptive pretraining (LAPT: Chau al.) 2020) usually updates all
model weights (Tran] 2020 Liu et al. 2024] Dobler & de Melo 2023] Ostendorff & Rehm 2023).
except Yamaguchi et al. (2024) which LoRA (Hu et al. [2022] Downey et al. (2023) show full- weight updates outperform embedding-onje training, which is insufficient for multilingual transfer.
Other vocabulary adaptation strategies introduce architecture-speeifi changes to the model, such as MAD-X (Preiffer et al. 2020), which incorporates various adapters into Transformer models, and thus additional computation costs. There are few alternatives these resource- -ittrree- ap- proaches. A notable exception is ZeTT (Minixhofer et al. 2024)) which trains a hypernetwork that generates embeddings for the new vocabulary, allowing immediate zero-shot use by only replacing embeddings without further model training. It can be extended to multilingual hypernetworks by appending a Iearnable language-specific embedding.
Linear Combination of Embeddings Most vocabulary transfer methods combine the existing embeddings to generate new ones. A popular approach is to use weighted average of the orig- inal embeddings (bolded in Appendix Table 99) For example, Gee al. (2022) and MMsiin et al. (023) compute the new embeddings by simply averaging the embeddings of subword tokens, while Tran 2020), Minixhofer et al.] (2022), OFA, and FOCUS utilize external resources to determine the weights initialize the new embeddings with weighted average of the original embeddings. Mundra et al. (2024) established theoretically that initializing within the convex hull of existing embeddings
e.g., using a weighted average of source embeddings is good initialization.
Our motivation stems from the question: rather than deciding how to combine existing embedding vectors heuristically, why not learn this process to create new embedding vectors? Relying on heuris- tics may lack adaptability that typically requires an additional training phase of full-weight updates to fully adapt to the new vocabulary. Building upon prior works, propose learn linear combi- nations with vocabulary adapters.
Empirical Evaluations Many language adaptation experiments have been conducted using new kunuuage-spccffee monolingual vocabularies (Minixhofer al. 2024 Dobler & de Melo 2023 Pfeiffer et al.] 2020; Minixhofer et al.| 20022] Yamaguchi et al. 2024), as well as English-on but
domain-specific vocabularies (Gee et al. [022 Mosin al. 2023). In contrast, [iu et al. (222 and
Mundra et al. (024) use a single unified mu multilingual vocabulary covering at least 369 languages
and four languages, respectively.
Downey et al. (2023) conducted experiments with both monolingual and multilingual vocabular- ies across eight languages and additional vocabulary from the Uralic family. While their findings
3

---

Published as a conference paper at ICLR 2025
indicated that multilingual adaptation in the Uralic family followed overall trends, it remains un- clear whether vocabulary adaptation benefits languages in different script groups. Overall, empirical evidence is still lacking to guide practical decisions for grouping languages in multilingual models.
Furthermore, most studies exclusively evaluate models on non-cross linguul and non-generative tasks, such as binary multi-class classification, sequence labeling (e.g., part-of-speech tagging), or answer span prediction. Mundra et al.] (2024) and Yamaguchi et al. (2024) include monolingual generative tasks like summarization. As a result, the impact of vocabulary adaption on cross-lingua generation tasks such as MT remains understudied, even though this is a crucial application area.
To address these gaps, this work introduces strategy to adapt L.LM new vocabularies, and exper- iments designed to measure its impact across diverse linguistic and task settings. We compare em- pirically against FOCUS and OFA, representing more resource-intensive initialization approaches, and ZeTT, representing a more parsimonious approach that has only been tested on limited tasks so far. Summaries of prior work can be found in Appendix Table
VoCADT: MULTILINGUAL VOCABULARY ADAPTATION WITH ADAPTERS
In this section, we outline our approach to multilingual vocabulary adaptation using adapter modules (VocADT). We detail the architecture of the adapter module ($3.)) and the initialization process (3.2). Additionally, we introduce an additional loss for handling overlapping tokens between the new and original vocabularies ($3.).. Finally, we further fine-tune the vocabulary adapted model for downstream task (3.4).
3.1 VOCABULARY ADAPTER MODULE
We introduce the vocabulary adapter modules to find parameters of new embeddings that can replace the original embedding without changing the non-embedding part of the original model (Figure La For simplicity, we refer to both input and output embeddings (or the LM head) collectively as em- vocabulary, respectively, and let T (I,, (,,...,,4, (x) be a tokenizer associated with a vocabulary V where Ij E V=, Vjj = 1, k. We put vocabulary adapter modules A E g[V"|"[[V") between the new vocabulary V and the original embedding E E V where h is an embedding dimension, a manner similar to bottleneck adapters (Houlsby al. 2019). We train the adapters with the standard language modeling loss C where we freeze the original weights and only update the adapters. This may be analogous to finding the new embedding vector for a token with the weighted combination of original embedding vectors Poblee
beddings. Let Vo and V be the original and
-
(Dwney et al. 2023; Dobler & de Melo 2023 Liu et al. (2024). Unlike similar works, our approach learns the weights for embedding combination. After training the adapters, we get new embeddings ET E |>> by merging the original embeddings and adapters E" AEE" which results in language model with the same architecture as the original one but with a different vocabulary size.
3.2 INITIALIZING ADAPTER
Effective initialization of the new embedding is crucial in adapting to a new vocabulary, as fully random initialization is widely recognized for leading poor performance (Minixhofer al. 2022 Yamaguchi et al. 2024). In our case, random initialization of the adapter A A equivalent to random initialization of E". making proper initialization of A equally important. We suggest simple initialization scheme for the vocabulary adapter, illustrated in Figure [b
First, we follow the common methods of copying the original embeddings of overlapping tokens by setting a one-hot vector in the adapter. Let I V= + Z be the mapping function of a token to an index in a vocabulary / and let = [") be the index of a token V", The row of the adapter A corresponding to the overlapping tokens w E // n V is set as follows:
A (1) = A., =
Vj + (w), where w E V n V
(1)
Mundra et al. (2024) report MT results, but they do not release the models or per-language performance metries, making direct comparisons difficult.
4

---

Published as a conference paper at ICLR 2025
Inspired by Gee et al. (2022). then initialize the row of token in A whose partitioned tokens by the original tokenizer T are subset of the original vocabulary. TT = {ti..... .a.....,tt, .. C Vo, >> 1, with normalized multi-ho hot vector below. This corresponds to directly initializing new emhedding with the average of the original embeddings associated with the tokens produced by To
V (V" n V and 1 jf (I.(t)....,,I (Immm) where w E S = w (T(w) = {tim) C VoJ.
4
F1 0
otherwise
(2)
For token that does not fall into the first two cases above (i.e. a non-overlapping token and its partitions by T are not in Vo), we randomly initialize a row vector of the adapter with the uniform distribution whose sum of each element is one as follows:
A
"j Uniform(O, 1), ,....> |/1| where (M((VNV")) (3)
Vol 2j=1 Uj
3.3 AUXILIARY LOSS
As training progresses, the adapter entries of overlapped tokens tend diverge from their ini- tial states. This divergence can be undesirable because the original embeddings are already well- integrated into the language model, and our goal more focused on adjusting the embeddings of the newly introduced vocabulary items. Following Minixhofer et al. (2024), we experiment with an additional loss term that encourages the adapter entries for overlapping words to remain close to their initial values, formulated as follows:
Laur
Σ
||Arn(w) A ( 12-
(4)
VAUA"
WE VOnvel
The final loss for the adapter training is the combination of the standard language loss and additional loss with the weighing factor of o,, Ltot L + aL
3.4 FURTHER FINE-TUNING FOR DOWNSTREAM TASK
To understand the impact of vocabulary adaptation after task-specific fine-tuning, we follow the full ALMA (Xu et al. 2024) training on all model parameters for the cross-lingual generation task of machine translation after our VocADT on just the embeddings. ALMA training begins with fine- tuning on monolingual data, followed by further weight optimization on small curated parallel data.
4 WHICH LANGUAGES BENEFIT THE MOST FROM VOCABULARY ADAPTATION?
We aim to understand "When and how should we perform vocabulary adaptation?". More specifi- cally, we seek insight into which languages might benefit the most from vocabulary adaptation in terms of improving overall performance or mitigating wer-fraammentttion
To this end, we design experiments to cover non-English languages along with English, listed in [a with variety of scripts and language families. These languages are broadly categorized into three groups: (1) Latin group of Swahili, Indonesian, Estonian, and Haitian, which are low- to mid-resource languages and all use Latin script; (2) Mixed group including Korean, Greek, Russian, and Bulgarian, which utilize mixture of scripts; (3) Cyrillic group for languages with that scrip
We test individual language adaptation with language-specific vocabularies. We also adapt several multilingual vocabularies that include English and four non-English languages in a single shared vocabulary, with each group corresponding to one of the previously mentioned groups. This is to identify a language grouping languages with consistent scripts.
to mix languages with different scripts or grouping
"The non-English languages that we cover are all highly fragmented by common L.M.s, and their fragmen- tation is similarly improved by our method. Therefore, our analysis focuses on performance. and and Appendix F.1 experiment with All group including all languages mentioned here.
IIn Scction 6.4
5

---

Published as a conference paper at ICLR 2025
Table I: Covered Languages and its availability in multilingual benchmarks. We mainly categorize non-English languages by scripts- Latin group (2-5) and the Mixed group (6-7). We additionally experiment with Cyrillic group (8-11). We follow the resource-level of languages from Joshi et al. (2020) and Üstün et al. (2024)
idx
Full Name
Short
Script
Resource
FLORES
XNLI
XCOPA
Belebele
MMMLU
English
en
Latin
High
2
Swahili
Latin
Low
3
Indonesian
id Latin
Mid
Estonian
et
Latin
Mid
5
Haitian Creole ht
Latin Low
6
Korean
ko
Hangul High
7
Greek
Greek
Mid
Russian
ru
Cyrillic
High
9
Bulgarian
og
Cyrillie
Mid
10
Ukrainian
uk
Cyrillic Mid
11
Kazakh kk
Cyrillic Mid
EXPERIMENT DESIGN
5.1 BASELINES AND MODELING
We use Mistral-7B Jiang et al. 2023) our language model, along with its original vocabulary, which consists of 32k tokens ([Vo = 32k). As baselines, we evaluate three state-of-the-ar methods
for vocabulary adaptation, ZeTT (Minixhofer et al. 2024). FOCUS (Dobler & de Melo 2023), and OFA (Liu et al.) 2024 For ZeTT and FOCUS, we experiment with language-specific vocabularies (ZeTT-mono, FOCUS-mono vocabulary. This results in separate adaptations per language, which could be hard to scale with larger language coverage @ For VocADT and OFA methods mYocADT-multi, OFA-multi), periment with multilingual vocabularies of five languages including English and four non-Enge -nnglillshh languages, where we define three distinct language groups such as {en, sw, id, et, ht} ( (Latin group), (en, ko, el, ru, bg) (Mixed group), and (en, ru, bg, uk, kk) (Cyrillic group)
their implementations require specifying the language adapt the
5.2 TRAINING VOCADT
Vocabulary. We train SentencePiece (Kudo & Richardson 2018) tokenizers on either language- specific corpora or combined corpus, with maximum of million tokens per language, and create new vocabularies with a size of 50k for all cases including mono/multilingaal vocabularies (VVl = 50k). Newly created vocabularies for each language group are shared across baselines.
Adapter Training. In the adapter training phase, train only the adapters, while fixing all pa- rameters of the original model. The input and output adapters are separate modules, as preliminary results showed that sharing an adapter for the input and output sides performs worse. We train 0.5B monolingual tokens language, totaling 2.5B mixed by 5 languages (English + 4 non-English from each corresponding group), and report test numbers from it. We use "clean" documents from the corpus of MADLAD-400 (Kudugunta et al. 2023 We set the weighing factor of auxiliary loss with 0.1 for non-Latin groups and 0 for Latin group unless otherwise specified. This based on the empirical results in Appendix Ahttt maintaining the embeddings of overlapping tokens close to the original status during the adaptation is effective only for non Latin script languages and counter-effective for Latin languages. More details regarding the training are in Appendix
5.3
FINE-TUNING
After the adaptation phase, follow the fine-tuning recipe of ALMA (Xu et al. 2024) that consists of full-weight training with monolingual corpus and a small amount highh-quaity parallel corpus
TT' does not support Ukrainian and Kazakh, therefore primarily compare and average the results for 9 languages covered by both methods. See Appendix B for results for Ukrainian and Kazakh.

---

Published as a conference paper at ICLR 2025
Average Performance of multilingua MT/Reasoning/QA tasks
VocADT-multi (Ours)
ZETT-mono
FOCU5-moro
AccurccyNCoHE T.L...L
Mistral
OFA multi
.44
0.6
.53
-15
46
.68
33
25
.28
0.0
Xi-en, shots Nvg pairs
en-ix, shots Avg pairs
xnli. shots Avg 5 langs
xcopo, O shots Avg langs
belebele, 5 shots /ug langs
mmmlu, shots Avg langs
Figure 2: Average scores of original Mistral and its adaptation with new vocabulary, only replacing embeddings and fixing the body of transformer modules. "-multi" indicates models with multi- lingual vocabulary, which includes five languages covering all languages with two separate models, while "-mono" refers to monolingual vocabulary models. xx-en and en-xx indicate MT tasks. See Appendix for individual values.
to enhance MT performance (,34)) We also include Mistral in this phase of training. 1) mono- lingual fine-tuning, we use MADLAD-40K For adapted ZeTT and FOCUS models (prior work), we fine-tune each separate model with non-English language-specific vocabulary except for uk and kk for ZeTT (again, due to unsupported languages in ZeTT) using total of 2B tokens combining English and the corresponding non-Englise, For Mistral and the adapted VocADT and OFA, we fine-tune separate models for all three non-Enn non-English groups (Latin, Mixed, Cyrillic) plus English us- ing a corpus of 5B monolingual tokens containing languages. 2) In the next parallel training, we sample 15k bitext from NLLB dataset (Schwenk al., 20216] Heffernan et al. 2022 NLLB Team et al. 2022 for each English and non-English training pairs with top LASER3 scores (Artetxe & Schwenk 2019). The parallel training is done for one epoch, and we report test set numbers with the best model of the validation set. All the models are fine-tuned and tested with both directions of en-xx and xx-en within single model, meaning there are no separate models for opposite translation directions. We follow the prompting strategy Xf et al. (2024).
5.4 EVALUATION
We evaluate adaptation methods with multilingual benchmarks of various tasks including MT, nat- ural language inference (NL.), common sense reasoning, and multiple choice question answer- ing (QA). For MT of English to non-English (en-x0 xx) and non-English to English (xx-en), use FLORES (Goya et al.) 2022 NLLB Team et al.) 2022) as it supports all the languages that experiment with. We five-shot MT prompting for the model from the adaptation phase, and zero-shot prompting for the model after the ALMA training phase. We assess the transla- tion quality with xCOMET-XL (Guerreiro et al. 2023), which produces score of increasing quality ranging from 0 to 1. For NLI and reasoning, we use XNLI (Conneau et al.) (2018) and XCOPA (Ponti et al. 2020 with zero-shot prompting. For multiple choice QA, bele (Bandarkar et al. 2024 and Multilingual MMLU (Hendrycks et al. 2021 Lai et al. 2023] MMMLU) with five shot prompting. All the tasks except for MT are classification tasks, where we use the lm-eva
Bele-
(Gao et al. 2024 evaluation tool and report accuracy.
VOCABULARY ADAPTATION RESULTS AND ANALYSES
6.1 OVERALL TASK PERFORMANCE
We first present the controlled comparison on diverse tasks of the original Mistral with new vocab- ulary variants obtained by our vocabulary adaptation approach (VocADT) and the ZeTT and OFA baselines. Figure presents ing, and QA tasks. Language-wise results are in Appendix(E Overall, adapting the vocabulary using
presents the average performance across multiple multilingual MT. NLI, reason-
httee://muginnffaccc///assssssss

---

Published as a conference paper at ICLR 2025
VocADT generally leads to better performance compared to the original Mistral model, and either surpasses or performs on par with ZeTT. MMMLU is the only task where Mistral still holds the top spot; however, the performance gap between the new and original embeddings is smaller with Vo- CADT approach than with ZeTT. Remarkably, Voc AoeADtmmultt achieves these results with only two models for the eight languages tested, whereas ZeTT requires separate model for each language.
Tasks increase rate (%6 from Original Mistral in sw,idethht 152.39
Token Counts
Increase Rate (%)
150
Swahili
VocADT-mutti (Ours) ZETT-monD
mono multi mistral
100
Swahhll
65.3%
05%
50
51.5%
26.3%
55318
Increase Rate 1%)
xcopa Indonesian
helebele
25000 50000 75000 108000125000110000
7.5
2.6%
7.5%
Indoresian
24112
-54.18%
5.0
4,1%
26005
-50.59%
52629
(0.4%)
(-14%)--.79
(-22.59)-52.59)
0.0
Incrcase (%)
idien
envd
belebele
mmmlu
25000 50000 75000 108000125000150000
211.%%
Estonian
24963
-52.29%
50
24.%%
5.4%
143%
14.0%%
22.5%
(33,498
Rate (%)
et-en
en-et
xcopa Haitlan Creole
belebele
25008 50000 75000 15000000000220000000000000
27.75
27.7%
Haltian Croole
26704
-43.92%
20
11.3%
32795
-31.13%
10-
47616
(12.5%)
en-ht
xcаpa
25000
75000 108000125000110000
(a) Increase rate of task performance and number of token count of Latin group languages.
Tasks Increase rate (%) from Original Mistral ko,el,ru.bg
Increase hate (Ng)
Korean
VocADT-mutti (Ours) ZE ZETT-meno
meno
100
Korran
multi
-57.41%
mistral
70290
(14.4%)
1-2499%
1-15.0%)
1-102%))
(7.5%)
(-30.0%)
ka-en
en-ke Greek
belebele
23000 50000 75000 1080001250001500000
60
73.0%
51. 71.
30116
-79.84%
Greek
Increase
40
27.5%
36740
-75.41%
20
149418
7.6%
(7.9%)
(9.3%)
25000
75000 1010001250001100000
Russian
Increase Rate
2
Russian
28510
-4.088%
3-4227
36.65%
0.5%
53874
Increase Rate (%)
ni-en
en-ru
Buigarlan,
xnii
belebele
mmmiu
25000 50000 75000 10000015000011000000
5.9%
Bulgortan
5,0
32.988
-40.06%
3.5%
2.5
54948
(3,85)
(-.9%)
(4.4%) (416.99)
6-33.99
0.0
bgien
enibg
xnli
belebele
25000
50000
75000 108000125000150000
(h) Increase of task performance and numher of token count of Mixed group languages.
Figure Effect of vocabulary adaption on mitigating over-fragmentation and task performance. The y-axis for the increase rate the left side limited to the positive range. Languages with Latin scripts or those experiencing severe fragmentation benefit the most. xx-en and en- xx are machine translation tasks. See Appendix for individual task performance values.
8

---

Published as a conference paper at ICLR 2025
Grouping by Mixed len.ko.eL.u.bg and Cyrillk lk (ew.nu.bg.uk.e bg.ukkkk))
Token Counts
1.0
0.84
0.47
Russian
VesAATT-mat--
E-muli-yylliic
VockDTimutti Cyrilli-
0.6
0.08
2.46
misital
53874
1.0
0.4
nuven
#n-m
Xhi Bulgarian
beiebele
mmmmla
10000
20000
AccOCCONET
0.92
192
0.85
VoCADT multi Mixed
32308
40.20%
0.6
bulgurian
0.70
0,69
DocADTimubi-Cyilli
32938
-60, 01%
0.8-
mictral
54948
bg.cn
bg
Xhi
belobole
10000 20000
30000
40000
50000
Figure 4: Comparison of task performance between two grouping strategies of Mixed-script and
Cyrillic-script on two shared languages. Consistent script within group provides minor benefits.
6.2 WHICH LANGUAGES BENEFIT THE MOST FROM VOCABULARY ADAPTATION?
The previous section presented a macro view of performance across all languages. This section drills down to look the impact of vocabulary adaptation in language-wis wwie manner. Figure shows the increased rate of task performance and tokenization statistics for various languages after applying different vocabulary adaptation methods. The results shown for Latin group languages (Swahili, Indonesian, Estonian, and Haitian Creole) and Mixed group (Korean, Greek, Russian, and Bulgarian). We compare VocADT and ZeTT against the original Mistral model. The vertical axis of the increase rate is fixed to the positive range to see the benefit trends more easily and all the numbers including the negative range of the increase rate are Appendix We use the FLORES development set for counting the tokens by various tokenizers where the semantic contents of every language are the same.
Languages with Latin Scripts Severe Fragmentation Benefit the Most In Figures 3a observe that Latin script languages consistently benefit from vocabulary adaptation, regardless of the task, adaptation method employed. However, when they suffer from severe frammmragmentttion, as seen the case of Greek in Figures 3b) Among the eight languages, Greek is the most fragmented by the Mistral tokenizer, and it demonstrates significant improvement after the adaptation to less fragmented vocabulary, particularly in MT tasks, while other non-Latin languages in Mixed group show zero, modest, even negative gains.
non-Latin languages show improvements
In Appendix B we further discuss the pronounced performance declines observed in Korean com- pared to Russian or Bulgarian within the same Mixed group. Despite improving fragmentation for Korean, we suspect that the linear combination assumption is insufficient given the lack of represen- tation of Korean characters the original vocabulary.
6.3 DOES SCRIPT MATTER FOR LANGUAGE GROUPING?
Multilingual vocabularies for language groups strike balance between the extensive coverage of the original Mistral and the limited scope of language-specific monolingual models. We investi- gate strategies for grouping, in particular the effect of script.
Figure O compares the performance and token count reduction between two non-English grouping strategies for Russian and Bulgarian: Mixed-script (ko, el, ru, bg) and Cyrillic-crript (ru, bg, uk, kk) languages. For Russian, the consistent script language group performs slightly better, especially in the MT task. For Bulgarian, both grouping strategies deliver nearly identical results. Overall, the results suggest that maintaining a consistent script within a group enhances performance, though outcomes tend to be influenced more by the language type itself than by the grouping strategy.
6.4 SCALABILITY AND GENERALIZABILITY OF VOCADT
We further explore the language scalability of the method with All language groups including 11 languages (FF.I), and the generalizability of our VocADT findings to other language models ($F2). Both scalability and generalizability experiments show that the All group follows trends similar

---

Published as a conference paper at ICLR 2025
to the Latin, Mixed, and Cyrillic setups while the performance trends observed with LLaMA consistent with those seen in Mistral.
IMPACT OF VOCABULARY ADAPTATION ON DOWNSTREAM FINE-TUNING
Do the effects of vocabulary adaptation hold up after fine-tuning the adapted language model? After completing the VocADT process, which keeps non-embedding model weights fixed, we update the full weights of the adapted model to enhance MT performance following ALMA (Xu et al. 2024).
As can be seen in Table 2 all vocabulary adaptation approaches are effective compared to Mistral except for en-sw, and among those, our approach (VocADT) achieves the highest average score in both en-xx and xx-en directions. In the xx-en direction, the performance of VocADT matches that of ZeTT, despite using smaller number of individual models of the same size (2 VocADT vs 8 ZeTT). Interestingly. language-specific models (ZeTT, FOCUS) tend to excel in Latin languages, whereas multilingual models (Mistral, VocADT, OFA) generally outperform language-specifg mod- els in non-Latin cases.
In sum, with full parameter fine-tuning after the vocabulary adaptation, our VocADT model offers a competitive edge across both xx-en and en-xx tasks, further validating the effectiveness of our approach. VocADT demonstrates that a multilingual model can achieve or surpass aaguage--ppectnn models like ZeTT, offering more flexible and scalable solution for handling multiple languages.
Table 2: MT performance after full-weight fine-tuning the new vocabulary-adaptee model. The sym- bol "#" indicates the number of separate models for this experiment table. All vocabulary adaptation approaches after fine-tuning are effective compared to Mistral except for en-sw. VocADT -ulti shows the best average performance both directions while matching the score of ZeTT in xx-en.
FLORES
Xx-an
en-Xx
Lang (group) J
VocADT
Mistral
Z:TT
OFA
FOCUS
VocADT
Mistral
ZeTT
OFA
FOCUS
(Lari))
0.893
0.891
0.889
0.893
0.753
0.762
0.763
0.762
1d (Tarin)
0.951
0.950
0,953
0.951
0.945
0.874
0.874
0.879
0.871
0.876
et (Latin)
0.939
0.925
0.941
0.937
0.939
0.852
0.868
0.856
0,845
0,870
(Latin)
0.706
0.696
0.699
0.699
0.685
0.336
0.336
0.334
0.332
0.339
(Mixed)
0.886
0.892
0.898
0.897
0.906
0.755
0.715
0.772
0.742
0.754
el (Mixed)
0.922
0.845
0.910
0.894
0.902
0.876
0.817
0.857
0,862
0.861
zu (Mixed)
0.945
0.894
0.944
0.946
0.945
0.889
0.828
0.872
0,882
0.868
(Mixed)
0.953
0.904
0.952
0.950
0.949
0.895
0.844
0.895
0.899
0.895
Avg pains)
0,899
0.875
0.895
0.895
0.779
0.757
0.778
0.774
0.778
uk (Cynlllee
0.941
0.941
0.936
0.928
0.878
0.876
0.868
0.840
kk (Cyrillle)
0.881
0.875
0.875
0,880
0.807
0.790
0.785
0.796
Avg (10 pains)
0,881
0.897
0.897
0.792
0.772
0.785
0.786
# of Models +
3
3
3
3
8
3
8 CONCLUSION
We propose simple and effective vocabulary adaptation method using vocabulary adapter. Our approach consistently outperforms the original Mistral model after the adaptation phase across var- ious tasks and after the following full-weight finetuning on machine translation. Furthermore, our method is on par with or more effective than strong vocabulary adaptation baselines, without relying on external embeddings or language constraints, offering a flexible and scalable solution for han- dling multiple languages. Our experiments cover a wide range of languages and scripts, revealing that languages with Latin scripts or severe fragmentation benefit the most. We also explored differ- ent grouping strategies, finding that maintaining consistent scripts within group offers relatively minor benefits. Lastly, with a focus on machine translation, we confirm that vocabulary adaptation remains effective even after full-weight fine-tuning, and VocADT is the most effective approach.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
We thank Anthony Aue for early discussions, and Marcin Junczys-Dowmunt and the anonymous reviewers for their insightful and helpful feedback.
REFERENCES
Roee Aharoni, Melvin Johnson, and Orhan Firat. Massively multilingual neural machine translation. In Jill Burstein, Christy Doran, and Thamar Solorio ), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language
Technologies. Volume I (Long and Short Papers), Pp. 1874-3884, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 18653/v1/N19- 1 1388. URL https:
Taclantholln wtt.chhoooo....g///... 1388
Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David Mortensen, Noah Smith, and Yulia Tsvetkov. Do all languages cost the same? tokenization in the era commercial language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedingy of the 2023 Confer- ence on Empirical Methods in Natural Language Processing, pp. 9904-9923, Singapore, Decem- ber 2023. Association for Computational Linguistics. doi: 1...86633v112023.mmmmpp URL httpps://alaantologgee .rg. 2023.022... www.eennnaaeeeeeeee 614
Mikel Artetxe and Holger Schwenk. Massively Multilingual Sentence Embeddings for Zero-Shot Cross-L.ingual Transfer and Beyond. Transactions of the Association for Computational Lin- guistics, 7:597-610, 09 2019. ISSN 2307-387X. doi: 10. 1. 1..1162ttall
URL https:
//dbi. 100 1162/............. a 00288
Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. The belebele benchmark: a parallel reading comprehension dataset in 122 language variants. In Lun-Wot Ku, Andre Martins, and Vivek Srikumar Proceedigss of the 62nd Annual Meeting of the Associ- ation for Computational Linguistics (Volume 1: Long Papers), pp. 749-775, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 10.5///5//1//2 URL https: https::///alaathhoooggoo org/ 2024 acl-log .4
Sheila Castilho and Rebecca Knowles. A survey of context in neural machine translation and its evaluation. Natural Language Processing, pp. 1-31, 2024. doi: 2..1017/nl..222..7..
Isaac Caswell, Theresa Breiner, Daan van Esch, and Ankur Bapna. Language ID in the wild: Unexpected challenges the path a thousand-language web text corpus. In Donia Scott, Nuria Bel, and Chengqing Zong (eds.), Proceedings of the 28th International Conference on Computational Linguistics, pp. 6588-660S, Barcelona, Spain (Online), December 2020. Interna- URL
tional Committee on Computational Linguistics. doi: http:///ww//2222.oooggommoaa org/ 2020. www.ecoooiiiciiooeo 579/
TLEEs::/aclaattie
Ethan C. Chau, Lucy H. Lin, and Noah A. Smith. Parsing with multilingual BERT, a small corpus, and small treebank. In Trevor Cohn, Yulan He, and Yang Liu (eds.). Findings of the Asso- ciation for Computational Linguistics: EMNLP 2020, pp. 1324-1334, Online, November 2020. Association for Computational Linguistics. doi: 10. 1/2000. http:///w/200...nnggeeeem//a findings-emmll..11 URL aattps://aclantholog wtt.aaattoooooyy.oo//0.......... wtddigggs.....i.eee.eeee..e
Vishrav Chaudhary, Yuging Tang, Francisco Guzmán, Holger Schwenk, and Philipp Koehn. Low- resource corpus filtering using multilingual sentence embeddings. In Ondřej Bojar, Rajen Chat- terjee, Christian Federmann. Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, An- tonio Jimeno Yepes, Philipp Koehn, André Martins, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Matt Post, Marco Turchi, and Karin Verspoor (eds.), Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2). pp. 261-266, Florence, Italy, August 2019. Association for Computational Linguistics. doi: URL
10.18653/v1/W19-5533
wrg.y.orrrgoooo. www./oggg/-00 5435/
Hyung Won Chung, Dan Garrette, Kiat Chuan Tan, and Jason Riesa. Improving multilingual models with language-clustered vocabularies. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu
11

---

Published as a conference paper at ICLR 2025
(eds.). Proceedings of the 2020 Conference Empirical Methods in Natural Language Process- ing (EMNLP), pp. 1536-4546, Online, November 2020. Association for Computational Linguis- tics. doi: 10.18153/1 1ttp33//11222.mmm////. 100p.000000mmmmmmmimm..6.... main.367.
emn p-ma 1...mnn..66 367
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. XNLI: Evaluating cross-lingual sentence representations. In Ellen Riloff, David Chiang. Julia Hockenmaier, and Jun'ichi Tsujü (eds.), Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp. 2475-2485, Brussels, Belgium, October-Noventee 2018. Association for Computational Linguistics. doi: 1.L.8653//1DDD1-L1
URL
Daniel Deutsch, Rotem Dror, and Dan Roth. A statistical analysis of summarization evaluation met- rics using resampling methods. Transactions of the Association for Computational Linguistics, 9: 1132- 1146, 2021. doi: 10.1 166/tall 00417. Utthttps: / /aalaaaclcatooocc org/2211 677
Daniel Deutsch, Juraj Juraska, Mara Finkelstein, and Markus Freitag. Training and meta-evaluating machine translation evaluation metrics at the paragraph level. In Philipp Koehn, Barry Haddow, Tom Kocmi, and Christof Monz (eds.), Proceedings of the Eighth Conference on Machine Trans- lation, pp. 996-9996211199 1013, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023 wmn- 1.96. URL https: battps://aclanee http:///aaaantttoooceeeo wwt.oog//2222 2023. wmt-1.
96/
Konstantin Dobler and Gerard de Melo. FOCUS: Effective embedding initialization for mono- lingual specialization of multilingual models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Pro- cessing, pp. 13440-13454, Singapore, December 2023, Association for Computational Linguis- tics. doi: 111188653/1/22223 111.:///////22..mmmmmmmm..m main.829, URL
emn p- ma in..n... 829
Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Mar- garet Mitchell, and Matt Gardner. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 1286-1305, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. (0.18663/112221.mmmmll main.98. URL /etps://aclanien
org//o2111222 wwnlp-ma -app--mannn. 987
C.m. Downey, Terra Blevins, Nora Goldfine, and Shane Steinert-Threlkeldd Embedding structure matters: Comparing methods to adapt multilingual vocabularics to new languages. In Duygu Ataman (ed.), Proceedings of the 3rd Workshop Multi-lingual Representation Learning (MRL), pp. 268-281, Singapore, December 2023, Association for Computational Linguistics. doi: 10. URL httpt//ss//cllaathooooooo/////// mm11223...111...
Markus Freitag, Nitika Mathur, Chi-kiu Lo. Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Tom Kocmi, Frederic Blain, Daniel Deutsch, Craig Stewart, Chrysoula Zerva, Sheila Castilho, Alon Lavie, and George Foster. Results of WMT23 metrics shared task: Metrics might be guilty but references are not innocent. In Philipp Koehn, Barry Haddow, Tom Koemi, and Christof Monz (eds.), Proceedings of the Eighth Conference on Machine Translation, pp. December 2023. Association for Computational Linguistics. doi: 00.18653/v120223 30t0223..tt111. 1.51. URL https:
Singapore,
www.antoooooooooooooooooooooo 201/. wmt-1.517
Markus Freitag, Nitika Mathur, Daniel Deutsch, Chi-Kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Frederic Blain, Tom Kocmi, Jiayi Wang, David Ifeoluwa Adelani, Mari- anna Buchicchio, Chrysoula Zerva, and Alon Lavie. Are LLMs breaking MT metrics? re- sults of the WMT24 metrics shared task. In Barry Haddow, Tom Kocmi, Philipp Koehn, and Christof Monz (eds.), Proceedings of the Ninth Conference on Machine Translation, pp. 47- 81, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. N0.18653/v1/200 wmt- 1.2. URL htRtst/t::/ :/asc://clla anthhonttollooo ogy wwwp//gg/22222 wmt 11.t-11.2 .2/
12

---

Published as a conference paper at ICLR 2025
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa. Jason Phang. Laria Reynolds. Hailey Schoelkopf, Aviya Skowron, Lin- tang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024. Uttttttt:://eet 12608602
org/ hatgg//ecrds/////
Leonidas Gee, Andrea Zugarini, Leonardo Rigutini, and Paolo Torroni. Fast vocabulary trans- fer for language model compression. In Yunyao Li and Angeliki Lazaridou (eds.), Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track, pp.
Abu Dhabi, UAE, December 2022. Association for Computational Lin- guistics. doi: 10. (6653////22222mmm--nnddnrrr 2022. emn 1p- mndssrr.rr...mm 41
URL pettss//accanttaee watahoooooggy... org/
Evangelia Gogoulou, Ariel Ekgren, Tim lshister, and Magnus Sahlgren. Cross-lingual transfer of monolingual models. In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi. Hitoshi Isahara, Bente Maegaard, Joseph Mari- ani, Hélène Mazo, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Thirteenth Language Resources and Evaluation Conference, pp. 948 955, Marseille, France, June 2022. European Lan- guage Resources Association. URL https: /aai://alllaas haw.thhhccacaccccaeaa logy. wtg/ rrec- 100
Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, San- jana Krishnan, Marc' Aurelio Ranzato, Francisco Guzmán, and Angela Fan. The Flores-101 Eval- uation Benchmark for Low-Resource and Multilingual Machine Translation. Transactions of the Association for Computational Linguisties, 10:522-532 05 2022. ISSN 2307-387X. doi: a_474
10.1 162/tacl a 00474. WtRlthttps:////o /doi .org
Nuno M. Guerreiro, Ricardo Rei, Daan van Stigt, Luisa Coheur, Pierre Colombo, and André T. Martins. xcomet: Transparent machine translation evaluation through fine-grained error detection, 2023. httpttts:///ww..///////aaa//////
Kevin Heffernan, Onur Çelebi, and Holger Schwenk. Bitext mining using distilled sentence rep- resentations for low-resource languages. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.). Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 2101-2112, 12, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022. http://///2222..nnnnnn-eeee findinggs-emlll.5444 URL hhattss://aclaanteellg/./ org/2022.
f Findings-emn pp 154
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Ja- cob Steinhardt. Measuring massive multitask language understanding. In International Confer- ence on Learning Representations, 2021. URL
d7KB 3Cm@
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th In- lernational Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research. pp. 2790-2799. PMLR, 09-15 Jun 2019. URL pttpss::/ppoccecingg...
pzess/v97/boulsby/
html
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con- ference on Learning Representations, 2022. URL ptttps:///peerreveee net net/fof forum?
nd=nZevKeeFFFs
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anm Lachaux. Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL hhtps:://////... ://axiii///xi...
org/ 2310 06825
13

---

Published as a conference paper at ICLR 2025
Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. The state and fate of linguistic diversity and inclusion in the NLP world. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.). Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 6282-6293, Online, July 2020. Association for Computational URL https://acl
Linguistics. doi: 10118633/112022.cc-mmm
ogy .tggy..rrr///..
2020. acl main. 560.
Marzena Karpinska and Mohit lyyer. Large language models effectively leverage document-level context for literary translation, but critical errors persist. In Philipp Koehn, Barry Haddow, Tom Koemi, and Christof Monz (eds.), Proceedings of the Eighth Conference on Machine Translation, pp. 419-451, Singapore, December 2023, Association for Computational Linguistics. doi: 10. 18653/v 1/2023.222mmmmmmm 1.41. URL httt:::: httes//aclatttt ataatpss//a..annboooooooooo org /2033, wmt -14t-1141/
Huda Khayrallah and Philipp Koehn. On the impact of various types of noise on neural machine translation. In Alexandra Birch, Andrew Finch, Thang Luong, Graham Neubig, and Yusuke Oda (eds.). Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, pp. 74- 83, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/ #1/W18-2707 2--99. URL htt.thttt//ccl.t...e pttes//aclaattti www.oaannnooooo...o.oooooooooooe wwt.ooogo.o.///00/000000
Huda Khayrallah, Brian Thompson, Kevin Duh, and Philipp Koehn. Regularized training objective for continued training for domain adaptation in neural machine translation. In Alexandra Birch, Andrew Finch, Thang Luong, Graham Neubig, and Yusuke Oda (eds.), Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, pp. 36-44, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. 10.18653/11/111-220 2705. URL https: 7/a //cclanthllogyy... org. wttpoo/gg/////0o001 -7057
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Trabska-Barwinska. Demis Has- sabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic for- getting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521- 3526, 2017. doi: 10. 10.33///p 161 1835114. URL https: http://wwww.p... wrt.////.or////////oo//// 10. 10.1073/pnass 18118311611111111
Philipp Koehn. Statistical significance tests for machine translation evaluation. In Dekang Lin and Dekai Wu (eds.), Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pp. 388-395, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL httpps://aaaatthhloooy// org/ 404 32500/
Philipp Koehn, Vishrav Chaudhary, Ahmed Ei-Kishky, Naman Goyal, Peng-Jen Chen, and Fran- cisco Guzmán. Findings of the WMT 2020 shared task on parallel corpus filtering and align- ment. In Loic Barrault, Ondrej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-jusså, Christian Federmann, Mark Fishel, Alexander Fraser, Yvette Graham, Paco Guzman, Barry Had- dow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, André Martins, Makoto Morishita, Christof Monz, Masaaki Nagata, Toshiaki Nakazawa, and Mattoo Negri (eds.). Proceedings of the Fifth Conference on Machine Translation, pp. 742, Online, November 2020. Association for Computational Linguistics. URL
www.y.orr//0000 wmn 1. 78/
Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-rshikkh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytoně Sikasote, Monang Setyawan, Supheakmungkol Sarin, Sokhar Samb, Benoît Sagot, Clara Rivera, Annette Rios, Isabel Pa- padimitriou. Salomey Osei, Pedro Ortiz Suarez. Iroro Orife, Kelechi Ogueji, Andre Niyongabo Rubungo, Toan Q. Nguyen, Mathias Müller, André Müller, Shamsuddeen Hassan Muhammad, Nanda Muhammad, Ayanda Mnyakeni, Jamshidbek Mirzakhalov, Tapiwanashe Matangirā, Colin Leong, Nze Lawson, Sneha Kudugunta, Yacine Jernite, Mathias Jenny, Orhan Firat, Bonaven- ture F. P. Dossou, Sakhile Dlamini, Nisansa de Silva, Sakine Çabuk Ball Stella Biderman, Alessia Battisti, Ahmed Baruwa, Ankur Bapna, Pallavi Baljekar, Israel Abebe Azime, Ayodele Awokoya, Duygu Ataman, Orevaoghene Ahia, Oghenefegó Ahia, Sweta Agrawal, and Mofe- toluwa Adeyemi. Quality glance: An audit of web-crawled multilingual datasets. Transac- tions of the Association for Computational Linguistics, 10:50-72, 01 2022. ISSN 2307-387X. 00447
doi: 10.1 162/tacl 00447. J /ooi wttpiiiorg/111110
14

---

Published as a conference paper at ICLR 2025
Taku Kudo and John Richardson. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Eduardo Blanco and Wei Lu (eds.), Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pp. 66-71, Brussels, Belgium, November 2018. Association for Com- putational Linguistics. doi: 10. (0.1663/11/D182000 URL ftteps::/accaanthol.c
D18 2012
Sneha Kudugunta, Isaac Caswell, Biao Zhang. Xavier Garcia, Christopher A. Choquette-Choo, Katherine Lee, Derrick Xin, Aditya Kusupati, Romi Stella, Ankur Bapna, and Orhan Firat. Madlad-400; A multilingual and document-level large audited dataset, 2023. URL https:
//arxiv. wwt. abs/ 2309 04662
Viet Lai, Chien Nguyen, Nghia Ngo, Thuat Nguyen, Franck Demoncouut, Ryan Rossi, and Thien Nguyen. Okapi: Instruction-tume large language models in multiple languages with reinforce- ment learning from human feedback. In Yansong Feng and Els Lefever (eds.). Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demon- strations, pp. 318-327 Singapore, December 2023, Association for Computational Linguis- tics. doi: 10..1853// 1/2033 ce0.0023..mmlm-mmm20.0 URL htttts:://claatttlleee/// wrg.:/23.rg/000..
emn1p-demo 28
Yihong Liu, Peiqin Lin, Mingyang Wang, and Hinrich Schuetze. OFA: A framework of ini- tializing unseen subword embeddings for efficient large-scale multilingual continued pretrain- ing. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Findings of the Association for Computational Linguistics: NAACL 2024, pp. 1067-1097, Mexico City, Mexico, June 2024. Association for Computational Linguisties. doi: 10. 11111666111222444. www.://44.innninnnnaa....m URL
2024. findings naacl. 68
Chi-kiu Lo. YiSi unified semantic MT quality evaluation and estimation metric for languages with different levels of available resources. In Ondiej Bojar. Rajen Chatterjee, Christian Fed- ermann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, André Martins, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Matt Post, Marco Turchi, and Karin Verspoor (eds.), Proceedings of the Fourth Conference Machine Trunslation (Volume 2: Shared Task Pupers, Day 1), pp. 507-513, Florence, Italy, August 2019. Association for Computational Linguistics. doi: 10. 119653/11/W19-55555 https: httpss//acaattthhooc wwwp:/looooo.ogg//go.o org/N19--3333
URL
Chi-kiu Lo and Samuel Larkin. Machine translation reference-less evaluation using YiSi-2 with bilingual mappings of massive multilingual language model. In Loic Barrault, Ondřej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-jussă, Christian Federmann, Mark Fishel, Alexander Fraser, Yvette Graham, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, André Martins, Makoto Morishita, Christof Monz, Masaaki Nagata, Toshiaki Nakazawa, and Matteo Negri (eds.), Proceedings of the Fifth Conference on Machine Transla- tion, pp. 903-910, Online, November 2020. Association for Computational Linguistics. URL https: http:://ccaaathccoo..oorr//o///
WMz -m.-1.10 1.00//
Chi-kiu Lo, Rebecca Knowles, and Cyril Goutte. Beyond correlation: Making sense of the score differences of new MT evaluation metrics. In Masao Utiyama and Rui Wang (eds.), Pro- ceedings of Machine Translation Summit XIX, Vol. Research Track, pp. 186-199, Macau SAR, China, September 2023. Asia-Pacific Association for Machine Translation. URL https: //aclanthology. 2023. mt mumuumm t - 167
Antonio Valerio Miceli Barone, Barry Haddow, Ulrich Germann, and Rico Sennrich. Regularization techniques for fine-tuning in neural machine translation. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel (eds.), Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 1489-1494, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10. 1017-13//1/111-11556
URL https:://claattholsss
prg/D17-1155
Benjamin Minixhofer, Fabian Paischer, and Navid Rekabsaz. WECHSEL: Effective initializa- tion of subword embeddings for cross-lingual transfer of monolingual language models. In
15

---

Published as a conference paper at ICLR 2025
Marine Carpuat, Marie-Cate Catherine de Mamnefe, and Ivan Vladimir Meza Ruiz (eds.), Proceed- ings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pp. 3992 3992-4006, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10. 10.18v33/vl 1005533/1/222.nee/ m22022.aacl-mainn22900 URI.
naacl -ma i 293
Benjamin Minixhofer, Edoardo Maria Ponti, and Ivan Vulić. Zero-shot tokenizer transfer, 2024.
URL
07883
Vladislav Mosin, Igor Samenko, Borislav Kozlovskii, Alexey Tikhonov, and Ivan Yamshchikov. Fine-tuning transformers: Vocabulary transfer. Artificial Intelligence, 317:103860, 2023. ISSN 0004-3702. doi: https://bb..og//0.1006...aaaiii a023 103360 URL https: //www.
sciencedirect com http:://oo.oc/t/ttttotttaaaa////
Nandini Mundra, Aditya Nanda Kishore, Raj Dabre, Ratish Puduppully, Anoop Kunchukuttan, and Mitesh M. Khapra. An empirical comparison of vocabulary expansion and initialization ap- proaches for language models, 2024. /ttntttss//axxxi...tot http://wwww.rrg/bb//////////// abs/ 248070584711
NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia-Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shan- non Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. No language left be- hind: Scaling human-centered machine translation. 2022. URL 2207 04672]
Malte Ostendorff and Georg Rehm. Efficient language model training through cross-lingual and progressive transfer learning, 2023. URL BtRppstss://axxiit www.rxii...rr.b....eeeeeeeeeeeee httgg//bb/230111 /2001 09626
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Pierre Isabelle, Eugene Charniak, and Dekang Lin (eds.), Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguis- tics. doi: 01.11151177308.10733
Aleksandar Petrov, Emanuele La Malfa, Philip Torr, and Adel Bibi. Language model tokenizers introduce unfairness between languages. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL wwtp:httttt/seeeeeeeettt
net./ htt//forumm? dd d 78DDLKi 95p
Jonas Pfeiffer, Ivan Vulié, Iryna Gurevych, and Sebastian Ruder. MAD-X: An Adapter- Adapter-Based Framework for Multi-Task Cross-Lingual Transfer. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.). Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pp. 7674-7673, Online, November 2020. Association for Computa- tional Linguisties. doi: 10. 10.18653/w1/2220.cmm
main.617. URL httpp:://ccaathhaaaaa
org/ emnlp ma maimann. mii.n.... 617
Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulié, and Anna Korho- nen. XCOPA: A multilingual dataset for causal commonsense reasoning. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empir- ical Methods in Natural Language Processing (EMNLP), pp. 2376, Online, November URL
2020. Association for Computational Linguistics. doi: 10. ht httts::/aclanthoi..o wwtpo/////22/ 2020. emnlp ma in.i 185
Maja Popović. chriF: character n-gram F-score for automatic MT evaluation. In Ondřej Bojar, Rajan Chatterjee, Christian Federmann, Barry Haddow, Chris Hokamp, Matthias Huck, Varvara Logacheva, and Pavel Pecina (eds.), Proceedings of the Tenth Workshop Statistical Machine Translation, pp. 392-395, Lisbon, Portugal, September 2015. Association for Computational Lin- 5
guistics. doi: 10.
URL ht //acl
16

---

Published as a conference paper at ICLR 2025
Maja Popović. chrF++:: words helping character n-grams. In Ondiej Bojar, Christian Buck, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Ji- meno Yepes, Philipp Koehn, and Julia Kreutzer (eds.), Proceedings of the Second Conference on Machine Translation, pp. 612-618, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10. 18653/v1/W17-47700 URL https:
org/ MSS/W17-477777
Vikas Raunak, Tom Koemi, and Matt Post. SLIDE: Reference-free evaluation for machine transla- tion using a sliding document window. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies ( (Volume 2: Short Papers), pp. 205-211, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10. 18.63/1/v URL TRRLhttpe https///aaaaaahhooooooooo wwg. http:///222.aaaanshhheeeee
18/
Ricardo Rei, Craig Stewart, Ana Farinha, and Alon Lavie. COMET: A neural framework for MT evaluation. In Bonnie Webher, Trevor Cohn, Yulan He, and Yang Liu (eds.), Pro- ceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2685-2702, Online, November 2020. Association for Computational Linguis- tics. doi: 10. 10.118655/11/0022emm.m/... wew.00000mmmmmmmmm.......... URL
emnlp-mainn 213/
François Remy, Pieter Delobelle, Bettina Berendt, Kris Demuynck, and Thomas Demeester. Tik- to-tok: Translating language models one token at a time: An embedding initialization strategy for efficient language adaptation, 2023. URL htRpssst:///axtti wttp://gg..//.........
org/ abt/
Holger Schwenk, Vishrav Chaudhary, Shuo Sun, Hongyu Gong, and Francisco Guzmán. WikiMa- trix: Mining 135M parallel sentences in 1620 language pairs from Wikipedia. In Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty (eds.), Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 1351-1361, Online, April 2021a. Association for Computational Linguistics. doi: 10. 10.118633v1/2222 1ae.1e..lllmall mael-main.1 115.
URL http:s://aclaathollooooo org/ 2021 wwc.-main.. 115/]
Holger Schwenk, Guillaume Wenzek, Sergey Edunov, Edouard Grave, Armand Joulin, and Angela Fan. CCMatrix: Mining billions of high-quality parallel sentences on the web. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 6490-6500, Online, August 2021b. Association for Computational Linguistics. doi: 10. N0.11865//11/20 wtwp://.0001211ooo...oo.oo www.-lllogg50.... URL https:
//aclanthology, orq/ / 2021 acl- 507
Thibault Sellam, Dipanjan Das, and Ankur Parikh. BLEURT:
Learning robust metrics for text generation. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7881-7892, Online, July 2020. Association for Computational Linguistics. doi: 10. 10.1865330 1/2000//2202 704. URL https:tt/aaaact http::////caaatooooo...ooo org/ 220
Steve Sloto, Brian Thompson, Huda Khayrallah, Tobias Domhan, Thamme Gowda, and Philipp Koehn. Findings of the WMT 2023 shared task on parallel data curation. In Philipp Koehn, Barry Haddow, Tom Kocmi, and Christof Monz (eds.). Proceedings of the Eighth Conference on Machine Translation, pp. 95-102, Singapore, December 2023, Association for Computational Linguistics. doi: 10.18653/v1/20222 wmt- 1.5. httpttts::/// //ac /acl antho l logy .ag.yorg org/2033. wmt-1.. wm7
Brian Thompson and Philipp Koehn. Exploiting sentence order in document alignment. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Online, Novem- ber 2020. Association for Computational Linguistics. doi: 10. 1/2600011/2220..oooo/ www.:mmmmlll..mm..m. URL https: http//aaaaattaaa ogy. org 2020 emn p- n. 483/
Empirical Methods in Natural Language Processing (EMNLP), pp.
Brian Thompson and Matt Post. Automatic machine translation evaluation in many languages via zero-shot paraphrasing. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP),
17

---

Published as a conference paper at ICLR 2025
pp. 90-121, Online, November 2020a. Association for Computational Linguittse doi: 10.18653/ httph::tt////ct
wwt.200.emmlppmainn http:ttt///cclaaooo..oooo// wttp://ttthoooooyooo//222222 2020..200.... ea0.00mmmlllmamma. 8/
Brian Thompson and Matt Post. Paraphrase generation as zero-shot multilingual translation: Dis- entangling semantic similarity from lexical and syntactic diversity. In Loic Barrault, Ondřej Bo- jar. Fethi Bougares, Rajen Chatterjee, Marta R. Costa-jussă, Christian Federmann, Mark Fishel, Alexander Fraser, Yvette Graham, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Ji- meno Yepes, Philipp Koehn, André Martins, Makoto Morishita, Christof Monz, Masaaki Nagata, Toshiaki Nakazawa, and Matteo Negri (eds.), Proceedings of the Fifth Conference on Machine Translation, pp. 561-570, Online, November 2020b. Association for Computational Linguistics. URL https: a http///caaanntooo...rrro/
wmt 1. 677
Brian Thompson, Huda Khayrallah, Antonios Anastasopoulos, Arya D. McCarthy, Kevin Duh, Re- becca Marvin, Paul McNamee, Jeremy Gwinnup. Tim Anderson, and Philipp Koehn. Freezing subnetworks to analyze domain adaptation in neural machine translation. In Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Matt Post, Lucia Specia, Marco Turchi, and Karin Verspoor (eds.), Proceedings of the Third Conference Machine Translation: Reseurch Papers, pp. 124-132, Brussels, Belgium, October 2018. Association for Computational Linguistics. doi: N0.18653/v1/M 1/W18-6313. URL https: pttp::/aalantthliie org. 322g//186311//
Brian Thompson, Jeremy Gwinnup, Huda Khayrallah, Kevin Duh, and Philipp Koehn. Over- coming catastrophic forgetting during domain adaptation of neural machine translation. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, Volume (Long and Short Pupers), pp. 2062-2068, Minneapolis, Minnesota, June 2019a. Association for Computational Linguistics. doi: 10. 10.18653/v1/N19/ 1209. URL
1209/
Brian Thompson, Rebecca Knowles, Xuan Zhang, Huda Khayrallah, Kevin Duh, and Philipp Koehn. HABLex: Human annotated bilingual lexicons for experiments in machine translation. In Ken- taro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.). Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Confer- ence on Natural Language Processing (EMNLP-IJCNLP), pp. 1382 1387, Hong Kong, China, November 2019b. Association for Computational Linguistics. doi: 10. 10.18653/v1/D19 1142. URL https: antho Tochholoool orw..90/-/191111222
Brian Thompson, Mehak Dhaliwal, Peter Frisch, Tobias Domhan, and Marcello Federico. A shock- ing amount of the web is machine translated: Insights from multi-way parallelism. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Compu- tational Linguistics: ACL 2024, pp. 1763-1775, Bangkok, Thailand, August 2024a. Associa- tion for Computational Linguisties. doi: 10. 18151/611/22244 fin.224.inning-aa....00 URL https: 7/ //aclanthology.coom org/ 2tolgg//222222 2024. mandingssacl.222
Brian Thompson, Nitika Mathur, Daniel Deutsch, and Huda Khayrallah. Improving statistical sig- nificance human evaluation of automatic metrics via soft pairwise accuracy. In Barry Had- dow, Tom Kocmi. Philipp Kochn, and Christof Monz (eds.), Proceedings of the Ninth Con- ference on Machine Translation, pp. 1222-1234, Miami, Florida, USA, November 2024b. As- sociation for Computational Linguistics. doi: 10. 111558/112002.. wwt-1.21..mmm1111.. URL https: h/aclanthology.crr org/ wmt-1.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao. Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog. Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta. Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh
18

---

Published as a conference paper at ICLR 2025
Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
Ke Tran. From english to foreign languages: Transferring pre-trained language models, 2020. URL https://arxiv.om
btlgg/abs/20e wttp:xivvvrr///////////// /ttps://axxv.orr//aa////0000
07306
Ahmet Üstün, Viraat Aryabumi, Zheng Yong. Wei-Yin Ko, Daniel D' souua Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadaee, Julia Kreutzer, and Sara Hooker. Aya model: An instruction finetuned open-access multilingual language model. In Lun-Wat Wei Ku, Andre Mar- tins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 15894-15939, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL org/2024 acl
long. 845
Giorgos Vernikos, Brian Thompson, Prashant Mathur, and Marcello Federico. Embarrassingly easy document-level MT' metrics: How to convert any pretrained metric into a document-level metric. In Proceedings of the Seventh Conference on Machine Translation (WMT), pp. 118-128, Abu Dhabi, United Arab Emirates (Hybrid), December 2022. Association for Computational Linguis- tics. URL httt:s/s//alarthooooyyooooo
wmn 1. 67
Zihan Wang, Karthikeyan K, Stephen Mayhew, and Dan Roth. Extending multilingual BERT to low-resource languages. In Trevor Cohn, Yulan He, and Yang Liu (eds.), Findings of the Asso- ciation for Computational Linguistics: EMNLP 2020, pp. 2649-26555 Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v 1/2020. 1/t02..innnnnneeemn.i...i findings-emnlp.240
URL
emn. 1p.240 240
Joern Wuebker, Patrick Simianer, and John DeNero. Compact personalized models for neural ma- chine translation. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun' un'ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 881-886, Brussels, Belgium, October-November 2018. Association for Computational Linguis- tics. doi: 10.18653/w1/D18- 1104. URL/tttss://aclatthhi wttp://owooo...oo/o--111111
Haoran Xu, Young Jin Kim, Amr Sharaf, and Hany Hassan Awadalla. A paradigm shift in machine translation: Boosting translation performance of large language models. In The Twelfth Interna- tional Conference Learning Representations, 2024. URL /ettts::/cpenrreeete
net/
Eorum??d=farr 6XXntP.
Atsuki Yamaguchi, Aline Villavicencio, and Nikolaos Aletras. An empirical study on lingual vocabulary adaptation for efficient language model inference, 2024. URL
org/ 2402. 10712
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evalu- ating text generation with BERT. CoRR, abs/1904.09675, 2019. URL http:://axxii..///
abs/ /994. 09675
Vilém Zouhar, Shuoyang Ding, Anna Currey, Tatyana Badcka, Jenyuan Wang, and Brian Thomp- son. Fine-tuned machine translation metrics struggle in unseen domains. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 588-500, Bangkok, Thailand, August 2024. Association for Computational Lin- hta.lh/ppp///ellaatttoooooo.ooo/
guistics, doi: 10066/3////22244acssshoor
acl sel--short.o 457
19

---

Published as a conference paper at ICLR 2025
A IS THE AUXILIARY Loss HELPFUL?
We examine the effects of the auxiliary loss that aims to mitigate the divergence of original status for overlapping words described in Section 3.3 Figure 5 illustrates the impact task performance of vocabulary adaptation with and without auxiliary loss on Latin and Mixed group vocabulary. We report the average of four non-English languages in each group along with English.
For Latin languages (left plot of Figure 5 omittting the auxiliary loss (a 0) performs slightly better or comparably using non-zero For the Mixed group plus English vocabulary (right plot of Figure 5). maintaining the embedding values of overlapping words shows slight effectiveness in both non-English and English. We hypothesize that non-Latin languages are less prone to have word collisions with the original vocabulary compared to the Latin group, as the Mistral model is largely English (Latin) centric. As a result, retaining the established embeddings for overlapped words in Latin group vocabulary and Mistral vocabulary may disrupt effective learning due to the possible similarity in scripts with English. On the other hand, keeping the original embeddings during adaptation for overlapping tokens in Mixed may be helpful to maintain the already established em- beddings for overlapped tokens while adjusting the embeddings for new non- non-Latin script tokens.
VocADT www Aux Loss in Latin group English
VocADT w/wo Aux Loss in Mixed group
Average sw,id,et,hi
AAerrage ko, rl.r.g...bg....
English.
0.8
VocADT (e=0)
0.8
0.7
VarADT-mabi (a=00
0.7
0.5
0.5
0.4
VocADT-mulo
0.3
0.3
Voc AD A-llm
(m10
0.2
0.2
ADT-mull 0.21
on-Oo pairs
belebelo langs
boleblee
Xo:en
belebele
mmmlu
pairs
pairs
langs
Figure 5: Effects of the auxiliary loss various settings.
This auxiliary loss can be thought of as a form of regularization. There is a long history of applying regularization during adaptation of MT models control the adaptation process by limiting the amount that the output distribution or the weights of the fine-tuned model can vary from the original model weights. Prior work has explored using dropout and L2 regularization (Miceli Barone al. 2017]. cross entropy (Khayrallah et al. 2018), freezing parts of the network (Wuebker et al. [2018] Thompson et al., (2018). and Elastic Weight Consolidation (Kirkpatrick al. 2017 Thomppsn al. 2019a b). As our auxiliary loss has mixed effectiveness depending on language characteristics, future work could consider other methods.
B DISCUSSIONS ON ALA-BLTHA SCRIPTS AND POSSIBLE LIMITATIONS OF LINEAR COMBINATION ASSUMPTION
As shown in Figure 3b| the performance decline for Korean in VocADT vocabulary transfer (-14% to -15% in MT) is more pronounced than for Russian Bulgarian (-1% -0-% within the same Mixed group, even though the fragmentation improvement for Korean is greater. This deviates from the expected performance improvements seen in Greek, where mitigating extreme over-fragmentation (150k tokens) led to gains. Although Korean has significant over- over-fragmen fragfrrrmmtttiionn (70k tokens), its severity is less than half that of Greek and closer to Russian and Bulgarian (54k tokens).
One possible explanation is the limitation of our assumption that new embeddings can be solely represented as linear combinations of old embeddings. This assumption may not hold well for Ko- rean, which uses the non-alphabetic Hangul script. In Hangul, tokens represent entire syllables or consonant-vowel combinations rather than individual phonemes, making them difficult to decom- pose into subwords using the original tokenizer. For instance, the new token "2| (meaning "like" in English) appears frequently in Korean. However, the original Mistral vocabulary lacks a dedicated token for "ø"". preventing proper decomposition of "X] ş]" without resorting to byte-level tokens. These byte-level fallbacks may not effectively capture the linguistic structure of the character, poten- tially degrading performance. This issue is less prevalent in alphabetic scripts such Latin, Cyrillic,
20

---

Published as a conference paper at ICLR 2025
or Greek, where words can be easily broken down into individual characters. This limitation may account for the observed performance discrepancy.
C DETAILS OF TRAINING, BASELINE & EVALUATION
Here we describe training in detail. We use four Nvidia A 100 GPUs for adapter training and 16 AMD MI200 GPUs for full-weight fine-tuning. For all monolingual training including adaptation phase and fine-tuning phase, we follow (Xu et al. 2024) for setting the sampling ratio of monolingual data to mitigate language balance in the monohigual data and avoid prioritizing English. The method fixes the sampling ratio for English with a certain probability e.g. 1/n if there are n languages to mix and allocate the remaining ratio (e.g. n=1) by employing temperature sampling suggested by Aharoni et al. (2019). We mix the monolingual data for Latin group {en, sw, id, et. ht} with (17%, 16%, 32%, 23%, 12%) ratio, for Mixed group (en, ko, el, ru, bg} with (17%, 17% 19%, 30%, 17% ). and for Cyrillic (en, ru, bg. uk, kk} with {17%, 32%, 18%. 20%, ]3%]. }. For Add group in $6.4 and $F.1 fen, sw. id, et, ht, ko, el, ru, bg, uk, kk) with {10% 7%, 10%, 9%, 6%, 9%, 12% 10%, 9%, 10%, 8%) ratio.
For parallel training data for the MT task. we use bitext from the NLLB dataset (Schwenk al., 2021b, Heffernan et al. 2022 NLLB Team et al. 2022 8 This includes web-scraped data, which has the potential to include nosise such as text being automatically identified as the wrong language, mis-aligned or mis-translated segments, and low-quality machine translated segments (Khayrallah)
& Koehn 2018 Caswell et al. 2020 Dodge et al. 2021 Kreutzer et al., 2022; Thompson et al., 2024a). We use LASER3 (Artetxe & Schwenk| 2019) to select higher quality segments for fine- tuning. LASER has been used extensively to both locate parallel segments on the web (Schwenk al. 2021a b) well for filtering noisy sentence and document pairs (Chaudhary et al.) 2019
Koehn et al. 2020 Thompson & Koehn, 2020) Sloto et al. 2023).
In adapter training for VocADT, we (peak) learning rate of 2e-6 with cosine scheduler, maximum sequence length of 512 tokens, a warm-up ratio of 0.01, and a weight decay of 0.01. In full-weight fine-tuning phase, mostly follow the training setting from ALMA.
Details of Baseline. For ZeTT, we use multilingual hypernetwork for Mistral-7B We use the code of OF and FOCUS create new embeddings for Mistral-7B.
Machine Translation Metrics. We assess translation quality using XCOMET-XL. (uerreiro et al, 2023). as recent WMT metric shared tasks (Freitag et al. [003]] 2024) have found neural metrics like Yisi (Lo 2019 Lo & Larkin 2020), Bert-score (Zhang et al. 2019), Prism (Thompson & Post 2020a b). Comet (Rei et 20000, BLEURT (Sellam et al. 2020) correlate much better with human judgements, than surface- form metrics like BLEU (Papineni et al. 2002) or chrF (Popović) 2015] 2017) which consider only surface form. Trained metrics like the comet and BLEURT, which train on prior human annotations of translation quality, achieve the highest correlation with human judgments. While these correlations are less strong out domain (relative to the domains used in WMT, c.g. FLORES) the trained metries still outperform surface level ones (Zouhar et al. 2024).
We also caveat that xCOMET-XL does not consider context when judging translation quality, and context has been shown to be important aspect of translation quality evaluation (for thorough overview see (Castilho& & Knowles 2024). especially for LLMs Karpinska & Tyyer (2023). While there have been several efforts to incorporate context in MT evaluation (e.g. Vernikos et al. 2022 Deutsch et al. (2023); Raunak et al. (2024), there is no consensus in the community as to which method, we stick the established xCOMET-XL the sentence level. Finally, metric differ- ences, especially small ones, may not correspond to statistically significant differences (Koehn
2004 Deutsch et al. 2021 Lo ct al. 2023 Thompson et al. 2024b).
httess///aithbb.ccm/////////////
21

---

Published as a conference paper at ICLR 2025
D COMPUTATIONAL COST VOCADT
We report the computational cost of our approach. We use batch size of 128 (four A100 GPUs 8 batch size 4 gradient accumulation) and a sequence length of 512. The FLOPs per token for VocADT is 7.7GFLOPPstoken resulting in 1160T "FLOPs per batch" (128 512 17.7G). Our training requires 38k update steps (2.49B, roughly 0.5B per langs). Therefore, the computational cost of a VocADT model (1 160T "FLOPs per batch" x 38k step). We use profile () method of accelerator Python library and our estimation of FLOPs per token for Mistral-7B 14.2 GFLOPs/token.
E LANGUAGE-WISE RESULTS OF VOCABULARY ADAPTATIONS
Table 3: xx-en MT results with xCOMET-XL score and the increase rate from the original Mistral after the vocabulary adaptation-oni. replacing embeddings while fixing the rest.
MT (xx-en)
Mistral
VocADT-multi (Ours)
ZETT-mono
OFA-multi
FOCUS-mono
8w-en 0.485
0.801
65.32%
0.734
51.50%
0.215
-55.75%
0.216
-55.53%
id-en
0.946
0.942
-0.44%
0,880
-7.01 %
0.246
-73.97%
0.186
-80.36%
et-en
0.722
0.899
24.46%
0.769
6.40%
0.196
-72.93%
0.248
-65.70%
ht-en
0.554
0.669
20.72%
0.484
-12.55%
0.249
-54.66%
0.212
-61.72%
ko-en
0.882
0.755
-14.39%
0.662
-24.87%
0.189
-78.56%
0.318
-63.99%
el-en
0.438
0.760
73.59%
0.664
51.69%
0.182
-88.54%
0.250
-42.87%
ru=en
0.959
0.927
-3.33%
0.882
-8.06%
0.249
-74.06%
0.264
-72.43%
bg-en
0.952
0.918
-3.56%
0.896
-5.93%
0.228
-76.00%
0.271
-71.55%
Avg (8 pairs)
0,742
0.834
12.35%
0.746
0.56%
0.219
-70.47%
0.246
-66.92%
uk-en
0.944
0.915
-3.07%
0.201
-78. 78.70%
0.288
-69 669.4%%
kk-en
0.411
0.763
85.82%
0.190
-53.72%
0.308
-24.98%
Avg pairs)
0.729
0.835
14.49%
0.215
-70.59%
0.256
-64.89%
Total # of Models
3
8
10
Table 4: en-xx MT results with xCOMET-XL score and the increase rate from the original Mistral after the vocabulary adaptation-only replacing embeddings while fixing the rest.
MT (en-xx)
Mistral
VocADT-multi (Ours)
ZETT-mono
OFA-multi
FOCUS-mon mono
en-sw
0.238
0.562
135.88%
0.673
182.29%
0.342
43.54%
0.209
-12.23%
en-id
0.778
0.837
7.65%
0.836
7.47%
0.436
-43.99%
0.224
-71.26%
en-et
0.309
0.643
108.37%
0.652
111.23%
0.405
31.12%
0.247
-19.86%
en-ht
0.308
0.329
7.03%
0.339
10.11%
0.227
-26.88%
0.235
-23.61%
en-ko
0.703
0.598
-14,99%
0.631
-10.24%
0.309
-6.06%
0.258
-63.33%
en-el
0.384
0.413
7.56%
0.524
36.71%
0.232
-39,58%
0.215
-43.90%
0.900
0.854
-5.17%
0.719
-20.10%
0.388
-6.887%
0.371
-58.80%
en-bg
0.899
0.859
4.43%
0.755
-16.02%
0.332
-63.03%
0.289
-67.80%
Avg pairs)
0.637
12.77%
0.641
13.53%
0.334
40.89%
-54.66%
en-uk
0.865
0.851
-1.59%
0.318
-63.23%
0.310
-64.14%
en-kk
0.222
0.522
135. 11%
0.294
32.25%
0.223
0.65%
Avg (10 pairs)
0.560
0.647
15.40%
0.328
-41.44%
0.258
-53.93%
Total # Models
22

---

Published as a conference paper at ICLR 2025
Table 5: XNLI results with Accuracy score and the increase rate from the original Mistral after the vocabulary adaptation-one replacing embeddings while fixing the rest.
XNLI
Mistral
VocADT-multi (Ours)
ZETT-mono
OFA-multi
FOCUS-mono
en 0.550
0.47%
0.554
0.73%
0.547
-0.47%
0.537
-2.30%
S%
0.353
0.398
12.63%
0.453
28.33%
0.345
-2.16%
0.325
-7.96%
el
0.419
0.387
-7.60%
0,380
-9.31%
0.330
-21.21%
0.337
-19,58%
0.488
0.490
0.48%
0.474
-2.87%
0.347
-28.98%
0.331
-32.27%
bg
0.425
7.63%
5.88%
0.344
-19.02%
0.371
-12.59%
Avg (5 langs)
0.447
0.457
2.24%
0.462
3.40%
0.383
-14.38%
0,380
-14.93%
Total of Models
Table 6: XCOPA results with Accuracy score and the increase rate from the original Mistral after the vocabulary adaptation- only replacing embeddings while fixing the rest.
XCOPA
Mistral
VocADT-multt (Ours)
ZETT-mono
OFA-multi
FOCUS-mono
SW
0.510
0.574
12.55%
0.576
12.94%
0.564
10.59%
0.544
6.67%
1d
0.584
0.608
4.11%
0.598
2.40%
0.508
-13.01%
0.512
-12.33%
0.470
0.538
14.47%
0.536
14.04%
0.516
9.79%
0.520
10.64%
0.514
0.548
6.61%
0.572
11.28%
0.526
2.33%
0.534
3.89%
Avg (4 langs)
0.520
0.567
9.14%
0.571
9.82%
0.529
1.73%
0.528
1.54%
Total of Models
Table 7: Belebele results with Accuracy score and the increase rate from the original Mistral after the vocabulary adaptation-only replacing embeddings while fixing the rest.
Belchele
Mistral
VocADT-multi (Ours)
ZETT-mono
OFA-multi
FOCUS-mono
0.843
0.833
-1.18%
0.780
-7.51%
0.546
-35.11%
0.367
-56.52%
0.391
0.440
12.50%
0,476
21.61%
0.248
33.655%
0.252
-35.51%
id
0.647
0.638
-1.38%
0.610
-5.67%
0.289
-55.33%
0.230
-64.43%
et
0.439
0.538
22.53%
0.380
-13.42%
0.250
-43.04%
0.213
-51.39%
ht
0.397
0.507
27.72%
0.507
27.73%
0.248
-37.54%
0.240
-39.50%
ko
0,666
0.616
-7.52%
0,466
-30.05%
0.278
-58.27%
0.274
-58.77%
0.442
0.566
27.90%
0.468
5.79%
0.287
-35.17%
0.284
-35.68%
ru
0.727
0.696
-4.29%
0.459
-36.85%
0.248
90%
0.239
-67.13%
bg
0.674
0.698
3.47%
0.446
-33.93%
0.276
-59.14%
0.233
-65.40%
Avg (9 langs)
0,581
0.614
5.83%
0.510
-12.16%
0.296
-8.95%
0.259
-55.35%
0.728
0.693
-4.76%
0.254
-65.05%
0.231
-68.25%
kk
0.364
0.442
21.36%
0.256
29.877%
0.220
-39.63%
Avg (11 langs)
0.574
0.606
5.50%
0.289
-49.7%%
0.253
-55.93%
Total # of Models
9
11
Table 8: Multilingual MMLU results with Accuracy score and the increase rate from the original Mistral after the vocabulary adaptation-only replacing embeddings while fixing the rest.
MMMLU
Mistral
VocADT-multi (Ours)
ZETT-mono
OFA-multi
FOCUS-mon
0.607
0.577
-488%
0.537
-11.50%
0.464
-23.48%
0.288
-52,61%
0.468
0.410
-12.49%
0.316
-32.53%
0.256
45.34
0.269
-42.60%
0.500
0.468
-6.39%
0.348
-30.34%
0.259
-48.25%
0.272
-45.55%
Avg (3 langs)
0.525
0.485
-7.62%
0.400
-23.73%
0.326
-37.84%
0.276
-47.9%%
uk
0.489
0.462
-5.57%
0.269
45.06%
0.253
-48.19%
Avg (4 langs)
0.516
0.479
-7.14%
0.312
-39.55%
0.270
-47,8%%
Total Models
23

---

Published as a conference paper at ICLR 2025
Table 9: Tables of various vocabulary adaptation methods. The works in bold linearly combine original embeddings to generate new embeddings.
Vocabulary
External Resources
Base Model
Generative
Grouping
Aduptation
Langs
Task
multilingual
VocADT (Ours)
(Latin, Mixed,
11
Mistral
MT
Cyrillic group)
ZeTT
Mistral.
lang-specific
26
[Minixhofer et al.
2024
XLM-R
RAMEN
FastAlign fastText
BERT.
Tran 2020
lang-specific
RoBERTa
X
FVT
English,
(en)
BERT
Gee et al.
2022
domain-specific English,
VIPI
(Mosin et al.
2023
domain-speciin
(en)
BERT
X
OFA
multilingual (all in 401k)
XLM-R,
(Liu et al.
2024
min 369
ColexNett
RoBERTa
X
FOCUS
lang-specific
10
fastText
XLM-R
(Dobler & de Melo
2023
MAD-X
lang-specific
16
X-M-R
PPeiifee al.
2020
fastText,
WECHSEL
RoBERTa,
Minixhofer al.
2022
lang-specific
8
bilingual
GPT-2
X
dictionaries
CW2V
multilingual
bilingual
LLaMA2.
MT.
4
MMnnra et
2024
(all 4)
dictionarics
RoBERTa
summarization
CLP
GPT2-hase w
GPT2,
Ostendorff & Rehm
2023
lang-specific
(de)
WECHSEL
BLOOM
x
BLOOM-1/7B,
CLP+
GPT2-basc w WECHSEL
lang-specific
TigerBot- 7B,
summarization
Yamaguchi al.
2024
Mistral-7B
lang-specift.
Downey al. 2023)
multilingual
10
XLM-R
X
(Mixed, Uralic family)
F ADDITIONAL EXPERIMENT FOR SCALABILITY AND GENERALIZABILITY OF VOCADT
F.1 COMBINING LANGUAGES OF Latin, Mixed, AND Cyrillic INTO All GROUP
In Section 6.3 we observed that while grouping languages for the new vocabulary with a consistent script improves performance, script-based grouping strategies had little overall impact. This suggests that we can enhance the method's s scalability for greater practicality with minimal performance tradeoffs. In this section, we explore a multilingual group with shared vocabulary at larger scales to provide better insights into scalability for multilingual setups.
We combine languages from the Mulri-Latin, Mixed, and Cyrillic into one unified set into All. This set comprises 11 languages-Englisee and non-English languages (Swahili, Indonesian, Estonian, Haitian, Korean, Greek, Russian, Bulgarian, Ukrainian, and Kazakh) listed in Table [ Following our experimental setup of 0.5B tokens per language, we train on a combined corpus of 5.5B monolingual tokens, covering all 11 languages We set a 0.
Tables 10 and 11 show that the All group follows trends similar to the initial Latin, Mixed, and Cyrillic setups. Figure 6 further illustrates that while the token count for the All group is slightly higher than that of the Multi group setup, it remains significantly lower than that of the original Mistral model.
Available in htttees/hwngggggge...../////////
7 50
24

---

Published as a conference paper at ICLR 2025
F.2 GENERALIZATION TO LLAMA
We primarily conducted our experiments using the Mistral model. To validate the generalizability of VocADT findings to other language models, we also test our approach additional candidate LM, LLaMA (Tovrron et al. 2023).
We conducted an additional adaptation experiment using LLaMA2-7B, following the same exper- imental setup described in the main section. Figure 6 shows that the severity of fragmentation in LLaMA is similar to that in Mistral, with Greek being the most severely fragmented language fol- lowed by Korean. Tables 10 and 11 confirm that the performance trends observed with LLaMA are consistent with those seen Mistral. Overall, Latin group languages benefit largely from vocabu- lary adaptation, while non-Latin languages in the Mixed group show minus modest gains, except for Greek, which benefits due its severe fragmentation. These findings validate that our method generalizes effectively another language model.
Table 10: xx-en and en-xx MT results with XCOMET-XL score and the increase rate from the original Mistral after the vocabulary adaptation- only replacing embeddings while fixing the rest. The tables compare the All 11-language group versus the Multi groups Latin, Mixed, and Cyrillic (each comprising languages). We also compare the experiments using Mistral versus LLaMA as the base model.
Mistral
Llama
MT xx-en
Orig
VocADT-multi
VocADT-all
Orig
VocADT-multi
Total # Models
3
3
0.485
0.801
65.32%
0.775
59.89%
0.359
0.698
94.43%
id-en
0.946
0.942
-0.44%
0.919
-2.89%
0.954
0.933
-2.20%
0.722
0.899
24.46%
0.851
17.79%
0.496
0.858
72.98%
ht atn
0.554
0.669
20.72%
0.63
13.74%
0.392
0.645
64.54%
ko-en
0.882
0.755
-14.39%
0.834
-5.41%
0.872
0.776
-11.01%
el-en
0.438
0.760
73.59%
0.856
95.44%
0.439
0.777
76.99%
ru-en
0.927
-3.33%
0.929
-3.14%
-2.21%
bg-en
0.952
0.918
-3.56%
0,92
-3,38%
0,941
0.916
-2.66%
Avg (8 pairs)
0.742
0.834
12.35%
0.839
13.03%
0.675
0.817
21.04%
uk-en
0.944
0.915
-3.07%
0.909
-3.74%
0.947
0.897
-5.28%
kk-en
0.411
0.763
85.82%
0.751
82.92%
0.286
0.611
113.64%
Avg (10 pairs)
0.729
0.835
14.49%
0.837
14.76%
0.664
0.804
21.08%
Mistral
Liama
MT en-xx
Orig
VocADT-multi
VocADT-all
Orig
VocADT
en-sw
0.238
0.562
135.88%
0.466
95.48%
0.291
0.367
26.12%
en-id
0.837
7.65%
0.763
-1.89%
0.872
0.46%
en-et
0.309
0.643
108.37%
0.587
90.12%
0.279
0,581
108.24%%
en-ht
0.308
0.329
7.03%
0.312
1.40%
0.286
0.315
10.14%
en-ko
0.703
0.598
-14.99%
0.631
-10.23%
0.669
0.566
-15.40%
en-el
0.384
0.413
7.56%
0.635
65.56%
0.297
0,511
72.05%
0.900
0.854
-5.17%
0.855
-5.02%
0.877
0.824
-6.04%
en-bg
0.899
0.859
-4.43%
0.854
4.96%
0.826
0.825
-0.12%
Avg pairs)
0.565
0.637
12.77%
0.638
12.98%
0.549
0.608
10.75%
en-uk
0.865
0.851
-1.59%
0.83
-4.05%
0.83
0.814
-1.93%
en-kk
0.222
0.522
135.11% %6
0.555
150.05%
0.188
0.354
88.30%
Avg (10 pairs)
0.560
0.647
15.40%
0.649
15.79%
0.541
0.603
11.46%
25

---

Published as a conference paper at ICLR 2025
Table 11: XNLI, XCOPA, Belebele, MMMLU results with Accuracy score the increase rate from the original Mistral after the vocabulary adaptation-only replacing embeddings while fixing the rest. The tables compare the All I1-language group versus the Multi groups groups-Latin, Latin, Mixed, and Cyrillic (each comprising languages). We also compare the experiments using Mistral versus LLaMA as the base model.
Mistral
Llama
XNLI
Orig
VocADT-multi
VocADT-all
Orig
VocADT-multi
Total of Models
3
3
0.550
0.553
0.47%
0.53
-3.64%
0.554
0.568
2.53%
SW
0.353
0.398
12.63%
0.397
12.46%
0.348
0.378
8.62%
el
0.419
0.387
-7.60%
0.396
-5.49%
0.370
0.382
3.24%
0.488
0.490
0.48%
0.494
1.23%
0.425
0.47
10.59%
bgg
0.425
0.457
7.63%
0.435
2.35%
0.424
0.388
-8.49%
Avg (5 langs)
0.447
0.457
2.24%
0.451
0.89%
0.424
0.437
3.00%
Mistral
Llama
XCOPA
Orig
VocADT-multi
VocADT-all
Orig
VocADT-mule
0.510
0.574
12.55%
0.54
5.88%
0.522
0.546
4.60%
0.584
0.608
4.11%
0.592
1.37%
0.628
0.604
-3.82%
0.470
14.47%
6.38%
0.538
10.25%
ht
0.514
0.548
6.61%
0.538
4.67%
0.506
0.526
3.95%
Avg (4 langs)
0.520
0.567
9.14%
0.542
4.33%
0.536
0.5535
3.26%
Mistral
Llama
Belebele
Orig
VocADT-multi
VocADT-all
Orig
VocADT-muli
en
0.843
-1.18%
0.824
-2.29%
0.482
0.456
-5.39%
Sw
0.391
0,440
12.50%
0.454
16.08%
0.262
0.289
10.31%
id
0.647
0.638
-1.38%
0.636
-1.65%
0.380
0.346
-8.95%
0.439
0.538
22.53%
0.54
23.03%
0.312
0.319
2.24%
0.397
0.507
27.72%
0.522
31.59%
0.287
0.322
12.20%
0.666
0.616
-7.52%
0.644
-3.25%
0.336
0.354
5.36%
el
0.442
0.566
27. 27.90%
0.631
42,70%
0.301
0.357
18.60%
0.727
0.696
-4.29%
0.71
-2.30%
0.428
0.378
-1 11.8%%
bg
0.674
0.698
3.47%
0.694
2.91%
0.398
0.392
-1.51%
Avg (9 langs)
0.581
0.614
5.83%
0.629
8.33%
0.354
0.357
0.86%
uk
0.728
0.693
4.76%
0.682
-6.32%
0.398
0.352
-11.50%
kk
0.364
0.442
21.36%
0.427
17.18%
0.261
0.277
6.00%
Avg(11 langs)
0.574
0.606
5.50%
0.615
7.08%
0.349
0.349
0.05%
Mistral
Llama
MMMLU
Orig
VocADT-multi
VocADT-all
Orig
AocADT-mull
0.607
0.577
-4,88
0.561
-7.53%
0.452
0.415
-8.19%
0.468
0.410
-12.49%
0.444
-5.17%
0.367
0.296
-19.35%
0.500
0.468
-6.39%
0.468
-6.33%
0.355
0.34
-4.23%
Avg (3 langs)
0.525
0,485
-7.62%
0.491
-6.45%
0.391
0.351
-10.23%
0.489
0.462
-5.57%
0.463
-5,34%
0.346
0.328
-5.20%
Avg (4 langs)
0.516
0.479
-7.14%
0.484
-6.18%
0.38
0.345
-9.21%
26

---

Published as a conference paper at ICLR 2025
Token Counts
Token Counts
25833
53.30%
24734
Swahili
28736
45%
-47 58%
multi
Korean
29938
-57.41%
168 166% 26%
muti
34440
34%
37.18%
all
33810
51.90%
64.16%
53319
mistral
mistral
54823
94332
25000 50000
75000 100000123000150000
25000 50000
75000 100000125000150000
Indoressian
24112
-54.18%
-53 69%
30116
-79.84%
-79.56%
-50.59%
-50.05%%
Graek
30907
-41.28%
-40.65%
41948
-71.93%
-71.54%
32629
149418
52067
147371
250000 50000
75000 100000000010100001000000000
25000 50000
75000 10000012500115000
24963
-52.29%
-51 98%
28510 -47 -8%
541
Eskonian
27677
-47.10%
-46.76%
Aussian
34127
-36653% -30.05%
34 83%
34.40%
94%
-22.64%
52322
53874
25000
75000 100000000000100000000000
25000 50000
75000 100000150001150000
Creale
-43.
-42.64%
-47.07%
32795
33..13% 29.55%
Bulgarian aaaa
32938
40
06%
37.39%
-17.00% -15.111%
38%
330.41%
47618
$4948
46554
52608
25000 50000 75000 100000125000150000
25000
50000 75000 100001150001500000
Figure 6: Token count reduction with new vocabulary. Each bar displays two percentage reduction values: the first (e.g., -53.30% in Swahili) indicates the reduction relative to the original Mistral model, while the second (e.g., -52.88%) represents the reduction relative to the original LLaMA model. We the FLORRS development set for counting the tokens by various tokenizers where the semantic contents of every language are the same. While All group with all 11 languages is slightly higher than that of the Multi group with five languages, it remains significantly lower than that of the original models. The severity of fragmentation in LLaMA is similar to that in Mistral, with Greek being the most severely fragmented language followed by Korean.
27


Reviews:
[{"review_id": "l39EQnn2z2", "reviewer": "Reviewer_XRrF", "summary": "The paper introduces VocADT, an adapter-based method for vocabulary adaptation in pre-trained language models. New embeddings for an expanded vocabulary are learned by placing adapter modules between the new vocabulary and the original language model embeddings, enabling the model to use new tokens without modifying its main weights. The weights of the adapter module are initialized by (i) copying original embeddings of overlapping tokens, (ii) averaging embeddings where partitioned tokens in the original vocabulary subset exist, and (iii) randomly initializing the rest. An auxiliary loss is introduced to keep the embeddings of overlapping tokens close to their original values, preserving established representations for these tokens in the language model. The robustness of VocADT's impact on LLM performance for downstream tasks is tested by fine-tuning all model parameters after vocabulary adaptation. Detailed results are provided, evaluating 11 languages across 3 language groups — Latin, Mixed, and Cyrillic. VocADT is compared with three baseline methods: ZeTT, FOCUS, and OFA. Results show that, across languages and tasks, VocADT outperforms the original LM (Mistral). Latin languages benefit most from vocabulary adaptation, with language grouping strategies playing a minor role. Additionally, the auxiliary loss improves performance for Mixed-group languages but not for Latin languages. Finally, following ALMA, the authors fine-tune the full model to evaluate MT performance, showing that VocADT improves performance compared to the original Mistral baseline.", "questions": "1. Adaptation improves performance for fragmented Greek but less so for similarly fragmented Korean. Does this suggest vocabulary adaptation may not be effective for highly fragmented languages, and that other properties of Greek contribute to the improvements?\n2. Was any ablation conducted comparing initialization methods (e.g., one-hot vs. multi-hot) for the adapter module and their impact on downstream task performance?\n3. How effective would VocADT be for mixed-script or multilingual groups with shared vocabulary at different scales? Exploring grouping strategies in larger, mixed-language models could provide valuable insights into scalability for multilingual setups.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The introduction of VocADT, an efficient, adapter-based approach to vocabulary adaptation without relying on external embeddings, is novel.\n2. The comprehensive experimental evaluation covers a wide range of multilingual tasks and scripts across 11 languages. It compares the proposed method against three baselines to assess the benefits of vocabulary adaptation and the robustness of VocADT.\n3. The paper is well-structured and clearly presented.\n4. VocADT’s method for efficient adaptation without altering main model weights is well-motivated, especially for cross-lingual transfer and adaptation to low-resource languages.", "weaknesses": "1. Given that grouping Mixed-scripts and Cyrillic scripts had little impact on performance, it would have been more interesting to see if grouping all languages could have achieved identical results, demonstrating the language scalability of the method.\n2. The auxiliary loss, intended to retain original embeddings for overlapping tokens, lacks analysis across alpha values; further exploration in non-Latin languages might have improved performance.\n3. The paper evaluates the proposed method on Mistral alongside several baselines. However, including at least one additional candidate LM would have strengthened the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AKoZEK2Op9", "reviewer": "Reviewer_CsRD", "summary": "The authors propose a new vocabulary adaptation method that enables new words to be added to a pretrained LLM's vocabulary. The proposed approach uses adapters to learn embeddings for new words. The new embeddings are linear combinations of the existing embeddings. The approach is tested on 11 languages, showing that the best results are achieved for highly fragmented and Latin script based languages.", "questions": "1. Can you give an example where the case 3 mentioned in section 3.2 happen?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The methodology is clearly explained with reference to the prior work.\n2. The experiment design of this paper is well-crafted and supports the claims of the paper.", "weaknesses": "1. The motivation behind restricting the new embedding to a linear combination of original embeddings has not been explained.\n2. Since the approach requires some amount of training, the authors should report the computational cost of their approach compared to the baselines.\n3. The results in Appendix B shows significant difference among performance gains of different languages. The authors should perform analysis to determine why this happens. Does the amount of new pretraining data used for vocabulary adaptation affect the results?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6BMzDxjQGZ", "reviewer": "Reviewer_qgGK", "summary": "The article introduces VocADT, a method to conduct vocabulary adaptation with adapters. This method enables the training of new vocabularies while keeping all parameters of the original model fixed. Experimental results demonstrate that VocADT outperforms existing baselines. The study finds that languages with Latin scripts and those with severe fragmentation benefit the most from vocabulary adaptation. Additionally, the article validates that incorporating new languages through vocabulary adaptation improves performance in machine translation tasks.", "questions": "1. Can you further explain the limitations of heuristics-based methods and methods that rely on external embeddings? For example, what does it mean when the introduction states that \"heuristics-based methods often lack adaptability\"?\n\n2. Is your proposed method, VocADT, effective for low-resource languages?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The method proposed in this article allows for vocabulary adaptation by keeping the original model completely frozen and training only the adapter, which is highly efficient.\n\n2. Experimental results demonstrate that the method presented in this paper can effectively improve the performance.\n\n3. The article provides a detailed analysis of the MT task, offering a more accurate assessment of the model on cross-lingual tasks.", "weaknesses": "1. The article does not clearly explain or analyze the advantages of VocADT compared to heuristic-based methods and those relying on external embeddings or networks. This causes a disconnect between the claims about existing methods' shortcomings in the introduction and the proposed method.\n\n2. In VocADT, the initialization of embeddings and the use of auxiliary loss are very similar to existing work, raising concerns about the novelty of the paper.\n\n3. The design of the adapter in this paper assumes that the new embeddings are solely linear combinations of the old embeddings. This assumption could negatively impact performance. For example, languages with Latin scripts may perform better under this assumption, while languages with greater differences, such as Korean, might not fit this assumption effectively.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CA2LHZ12K5", "reviewer": "Reviewer_NnMw", "summary": "This paper proposes a method to address the vocabulary adaptation problem in multilingual NLP, and investigates the impacts of language scripts after vocabulary adaptation. After experiments using Mistral-7b on 11 languages, they find that the method proposed achieves better or on par with the vanilla model and the baseline methods. And the Latin-script and severe fragmented languages by Mistral-7b tokenizer benefit the most after vocabulary adaptation.", "questions": "1) The number of tokens to train baseline methods is unknown. Is it same with the VocADT?\n2) It can be found that the language Korean also suffers from severe fragment in Figure 3, while the improvement on Korean is little. How to interpret this phenomenon?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The findings about the impact of language scripts are interesting, which are the Latin-script and severe fragmented languages by Mistral-7b tokenizer benefit the most after vocabulary adaptation.", "weaknesses": "- [addressed]**Lack of novelty**: The method proposed is an incremental work, which combines the heuristic methods like OFA in the initialization and language adapters like MAD-X. It incorporates additional parameters and computation for the adapter and 8.5B(3.5B+5B) multilingual corpora in adaptation. Moreover, the incorporated auxiliary loss is not effective in Figure 5. The improvement seems little other than the \"en-el\" when using this auxiliary loss ($\\alpha=0.1$).\n\n- **Missing details and additional experiments to support the findings**: only Mistral-7b is used in this work. It is better to conduct experiments on other base models like llama or gemma to improve the generalization of the findings. The detail settings for baseline methods are unknown in this paper.\n\n- **Writing is not clear**: \n  1) Line 067: \";\" --> \"?\"\n  2) Figure 1:  \"Vocab Adaptor\" --> \"Vocab Adapter\"\n  3) Line 148: \"lightweight\": I cannot agree with this for the additional parameters and computation from the adapter training.\n  4) The Figure 3 is confusing and hard to follow: a) The y-axis of 3(a) left part are increase rate(%), thus the negative results are missing like \"id-en\", \"belebele\" on Indonesian? b). The mismatched legends for left and right part, which are \"VocADT-multi(ours), ZETT-mono\"(left) and \"mono, multi, mistral\"(right). These results will be better to report on Tables.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)