Adapters for Altering LLM Vocabularies: What Languages Benefit the Most?

## Abstract

Abstract Vocabulary adaptation, which integrates new vocabulary into pre-trained language models (LMs), enables expansion to new languages and mitigates token over-fragmentation.
However, existing approaches are limited by their reliance on heuristic or external embeddings.
We propose VocADT, a novel method for vocabulary adaptation using adapter modules that are trained to learn the optimal linear combination of existing embeddings while keeping the model’s weights fixed. VocADT offers a flexible and scalable solution without requiring external resources or language constraints.
Across 11 languages—with various scripts, resource availability, and fragmentation—we demonstrate that VocADT outperforms the original Mistral model (Jiang et al., 2023 ) and other baselines across various multilingual tasks.
We find that Latin-script languages and highly fragmented languages benefit the most from vocabulary adaptation.
We further fine-tune the adapted model on the generative task of machine translation and find that vocabulary adaptation is still beneficial after fine-tuning and that VocADT is the most effective method. 1 1 1 Project page: https://github.com/h-j-han/VocADT . Models at Huggingface Hub

## 1 Introduction

Vocabulary adaptation (or transfer)—a process of modifying a pre-trained language model (LM) to use a new vocabulary—offers several key advantages.
First, it enables the introduction of new languages into a model, allowing for greater flexibility in handling linguistic diversity and improving downstream performance in target languages (Wang et al., ; Gogoulou et al., ; Downey et al., ). Second, it addresses the problem of over-fragmentation, where tokens are excessively split, potentially slowing down generation and impairing performance in certain languages (Ahia et al., ; Petrov et al., ; Yamaguchi et al., ).
These benefits have led to the development of numerous vocabulary adaptation approaches
that initialize the new embeddings of new vocabulary with various methods based on heuristics (Mosin et al., ; Gee et al., ; Downey et al., ), external resources (Tran, ; Dobler & de Melo, ; Liu et al., ), or a separate hypernetwork that generates it (Minixhofer et al., ), optionally followed by continued training to finalize the adaptation (Minixhofer et al., ; Ostendorff & Rehm, ; Dobler & de Melo, ; Liu et al., ).

However, existing vocabulary adaptation approaches face several limitations.
Those that rely on heuristics (Gee et al., ; Downey et al., ), often lack adaptability and require an additional training phase of full-weight updates to fully adapt to the new vocabulary.
Those that depend on external embeddings or networks (Tran, ; Dobler & de Melo, ; Liu et al., ), increase complexity and limit scalability.
Furthermore, some approaches focus solely on language-specific cases or may have restrictions on the number of languages in the implementation when configuring the vocabulary (Dobler & de Melo, ; Minixhofer et al., ).

Additionally, we still know little about the impact of vocabulary adaptation across diverse linguistic and task settings.
Most prior work investigates a small number of languages, which is insufficient for identifying which languages benefit most from adaptation
(Tran, ; Ostendorff & Rehm, ; Remy et al., ; Yamaguchi et al., ), while studies that consider a broader range of languages only report averaged results without detailed analysis
(Liu et al., ; Mundra et al., ).
Furthermore, the impact of vocabulary adaptation on cross-lingual and generative tasks like machine translation (MT) is understudied, even though they represent crucial application areas for porting models to new languages. Many adaptation methods (Chung et al., ; Gee et al., ; Liu et al., ) have been evaluated instead on non-cross-lingual and discriminative tasks such as commonsense reasoning, natural language inference (NLI), or question answering (QA)—which are typically classification tasks.

We propose VocADT, a novel solution for vocabulary adaptation using adapters, designed to address key challenges in existing approaches.
We introduce a vocabulary adapter module between the new vocabulary and the original embeddings of a language model.
The module gradually adapts to new vocabularies through training while keeping all weights of the original model fixed, allowing the module to learn the best combination of the original embeddings.
This learned adaptation approach makes it more adaptable to new languages and more flexible in the number of languages while removing the necessity of external pre-trained resources.

In addition to our novel method, we empirically address the following key questions to understand the effectiveness and behavior of vocabulary adaptation:
(1) Which languages benefit most from vocabulary adaptation?
(2) What are the best strategies for creating new vocabularies;
is script consistency necessary?
(3) How does vocabulary adaptation impact machine translation?
We emphasize this task as it is a critical task for multilingual models that involves cross-lingual and generative capabilities, which are often more complex than classification or monolingual tasks.

We demonstrate the effectiveness of our adaptation method on various NLP tasks spanning Natural Language Understanding and MT.
Results show that our approach consistently surpasses the original Mistral model in most cases, both after the adaptation phase and following phase of full-weight training.
Additionally, our method outperforms or matches other strong vocabulary adaptation baselines. Our findings indicate that Latin-script languages and those with severe fragmentation benefit the most from vocabulary adaptation, while preserving embeddings of overlapped tokens proves useful for non-Latin languages, but not for Latin languages.
Finally, while all vocabulary adaptation methods continue to be effective for machine translation after fine-tuning, VocADT shows the best results among them.
Our main contributions are summarized as follows:

- •
We propose VocADT, a simple and effective solution for vocabulary adaptation using adapters, that addresses key challenges such as reliance on external embedding or language constraints.
- •
We design experiments that cover a wide range of languages and scripts, finding that languages with Latin scripts or severe fragmentation benefit the most and that having a consistent grouping of scripts for multilingual vocabulary is helpful.
- •
Our approach consistently outperforms the original language model and is more effective than, or at least on par with, strong vocabulary adaptation baselines after the adaptation phase across various tasks and after the following full-weight fine-tuning on MT.

Figure: (a) Overview of the vocabulary adaptation and training.
Refer to caption: x1.png

## 2 Background

#### Approaches

Many vocabulary adaptation methods focus on initialization strategies for the new vocabulary embeddings, before continuing training with unlabeled target language text using the original self-supervised pretraining objective. For instance, FOCUS (Dobler & de Melo, ) initializes embeddings as a weighted combination of overlapping tokens using external embeddings for non-overlapping ones, while copying the embeddings of overlapping tokens. OFA (Liu et al., ) also relies on external word vectors to initialize embeddings for non-shared new tokens, using a weighted average of original tokens based on semantic similarity. This strategy often requires external resources such as auxiliary embeddings (Dobler & de Melo, ; Liu et al., ; Ostendorff & Rehm, ) or bilingual dictionaries (Mundra et al., ; Minixhofer et al., ).

After initialization, language adaptive pretraining (LAPT; Chau et al., ) typically updates all model weights (Tran, ; Liu et al., ; Dobler & de Melo, ; Ostendorff & Rehm, ), except Yamaguchi et al. () which use LoRA (Hu et al., ). Downey et al. () show that full-weight updates outperform embedding-only training, which is insufficient for multilingual model transfer.

Other vocabulary adaptation strategies introduce architecture-specific changes to the model, such as MAD-X (Pfeiffer et al., ), which incorporates various adapters into Transformer models, and thus additional computation costs. There are few alternatives to these resource-intensive approaches. A notable exception is ZeTT (Minixhofer et al., ), which trains a hypernetwork that generates embeddings for the new vocabulary, allowing immediate zero-shot use by only replacing embeddings without further model training. It can be extended to multilingual hypernetworks by appending a learnable language-specific embedding.

#### Empirical Evaluations

Many language adaptation experiments have been conducted using new language-specific monolingual vocabularies (Minixhofer et al., ; Dobler & de Melo, ; Pfeiffer et al., ; Minixhofer et al., ; Yamaguchi et al., ), as well as English-only but domain-specific vocabularies (Gee et al., ; Mosin et al., ).
In contrast, Liu et al. () and Mundra et al. () use a single unified multilingual vocabulary covering at least 369 languages and four languages, respectively.

Downey et al. () conducted experiments with both monolingual and multilingual vocabularies across eight languages and additional vocabulary from the Uralic family. While their findings indicated that multilingual adaptation in the Uralic family followed overall trends, it remains unclear whether vocabulary adaptation benefits languages in different script groups. Overall, empirical evidence is still lacking to guide practical decisions for grouping languages in multilingual models.

Furthermore, most studies exclusively evaluate models on non-cross-lingual and non-generative tasks, such as binary or multi-class classification, sequence labeling (e.g., part-of-speech tagging), or answer span prediction. Mundra et al. () and Yamaguchi et al. () include monolingual generative tasks like summarization. As a result, the impact of vocabulary adaption on cross-lingual generation tasks such as MT remains understudied, even though this is a crucial application area. (^2^22Mundra et al. () report MT results, but do not release model, code, and number of individual languages to enable comparisons.)

To address these gaps, this work introduces a lightweight strategy to adapt LLM to new vocabularies, and experiments designed to measure its impact across diverse linguistic and task settings. We compare empirically against FOCUS and OFA, representing more resource-intensive initialization approaches, and ZeTT, representing a more parsimonious approach that has only been tested on limited tasks so far. Summaries can be found in Appendix Table 9.

## 3 VocADT: Multilingual Vocabulary Adaptation with Adapters

In this section, we outline our approach to multilingual vocabulary adaptation using adapter modules (VocADT).
We detail the architecture of the adapter module (§3.1) and the initialization process (§3.2).
Additionally, we introduce an additional loss for handling overlapping tokens between the new and original vocabularies (§3.3).
Finally, we further fine-tune the vocabulary adapted model for downstream task (§3.4).

### 3.1 Vocabulary Adapter Module

We introduce the vocabulary adapter modules to find parameters of new embeddings that can replace the original embedding without changing the non-embedding part of the original model (Figure 1(a)).
For simplicity, we refer to both input and output embeddings (or the LM head) collectively as embeddings.
Let $V^{o}$ and $V^{n}$ be the $\bm{o}$riginal and $\bm{n}$ew vocabulary, respectively, and let $\mathcal{T}^{x}:w\rightarrow(t_{1},t_{2},\dots,t_{k})$ be a tokenizer associated with a vocabulary $V^{x}$ where $t_{j}\in V^{x},\forall j=1,\dots,k$.
We put vocabulary adapter modules ${\bm{A}}\in\mathbb{R}^{|V^{n}|\times|V^{o}|}$ between the new vocabulary $V^{n}$ and the original embedding ${\bm{E}}^{o}\in\mathbb{R}^{|V^{o}|\times h}$ where $h$ is an embedding dimension, in a manner similar to bottleneck adapters (Houlsby et al., ).
We train the adapters with the standard language modeling loss $\mathcal{L}^{lm}$, where we freeze the original weights and only update the adapters.
This may be analogous to finding the new embedding vector for a token with the weighted combination of original embedding vectors (Downey et al., ; Dobler & de Melo, ; Liu et al., ). Unlike similar works, our approach learns the combination with adapter training.
After training the adapters, we get new embeddings ${\bm{E}}^{n}\in\mathbb{R}^{|V^{n}|\times h}$ by merging the original embeddings and adapters to ${\bm{E}}^{n}={\bm{A}}{\bm{E}}^{o}$, which result in a language model with the same architecture as the original one but with a different vocabulary size.

### 3.2 Initializing Adapter

Effective initialization of the new embedding is crucial in adapting to a new vocabulary, as fully random initialization is widely recognized for leading to poor performance (Minixhofer et al., ; Yamaguchi et al., ).
In our case, random initialization of the adapter ${\bm{A}}^{0}$ is equivalent to random initialization of ${\bm{E}}^{n}$, making proper initialization of ${\bm{A}}^{0}$ equally important.
We suggest a simple initialization scheme for the vocabulary adapter, illustrated in Figure 1(b).

First, we follow the common methods of copying the original embeddings of overlapping tokens by setting a one-hot vector in the adapter. Let $\mathcal{I}^{x}:V^{x}\rightarrow\mathbb{Z}$ be the mapping function of a token to an index in a vocabulary $V^{x}$ and let $i=\mathcal{I}^{n}(w)$ be the index of a token $w$ in $V^{n}$. The row of the adapter ${\bm{A}}^{0}_{i}$ corresponding to the overlapping tokens $w\in V^{o}\cap V^{n}$ is set as follows:

$$ ${\bm{A}}^{0}_{i,\mathcal{I}^{o}(w)}=1,\quad{\bm{A}}_{i,j}=0\quad\forall j\neq \mathcal{I}^{o}(w),\quad\text{where}\ w\in V^{o}\cap V^{n}.$ (1) $$

Inspired by Gee et al. (), we then initialize the row of a token $w$ in ${\bm{A}}^{0}$, whose partitioned tokens by the original tokenizer $\mathcal{T}^{o}$ are subset of the original vocabulary, $\mathcal{T}^{o}(w)=\{t_{1},\dots,t_{m}\}\subset V^{o},m>1$, with normalized multi-hot vector as below. This corresponds to directly initializing new embedding with the average of the original embeddings associated with the tokens produced by $\mathcal{T}^{o}$.

$$ ${\bm{A}}^{0}_{i,j}=\begin{cases}\frac{1}{m}&\text{if }j\in\{\mathcal{I}^{o}(t_ {1}),\dots,\mathcal{I}^{o}(t_{m})\}\\ 0&\text{otherwise}\end{cases}\text{where}\begin{array}[]{l}w\in V^{n} \backslash(V^{o}\cap V^{n})\quad\text{and}\\ w\in S=\{w\mid\mathcal{T}^{o}(w)=\{t_{1:m}\}\subset V^{o}\}.\end{array}$ (2) $$

For a token that does not fall into the first two cases above (i.e. a non-overlapping token and its partitions by $\mathcal{T}^{o}$ are not in $V^{o}$), we randomly initialize a row vector of the adapter with the uniform distribution whose sum of each element is one as follows:

$$ ${\bm{A}}^{0}_{i}=\frac{\mathbf{u}}{\sum_{j=1}^{|V^{o}|}u_{j}},\quad u_{j}\sim \text{Uniform}(0,1),\ j=1,\dots,|V^{o}|\quad\text{where}\ w\in V^{n}\backslash (V^{o}\cap V^{n})\backslash S.$ (3) $$

### 3.3 Auxiliary Loss

As training progresses, the adapter entries of overlapped tokens tend to diverge from their initial states.
This divergence can be undesirable because the original embeddings are already well-integrated into the language model, and our goal is more focused on adjusting the embeddings of the newly introduced vocabulary items.
Following Minixhofer et al. (), we experiment with an additional loss term that encourages the adapter entries for overlapping words to remain close to their initial values, formulated as follows:

$$ $\mathcal{L}^{aux}=\frac{1}{|V^{o}\cap V^{n}|}\sum_{w\in|V^{o}\cap V^{n}|}||{ \bm{A}}_{\mathcal{I}^{n}(w)}-{\bm{A}}^{0}_{\mathcal{I}^{n}(w)}||_{2}.$ (4) $$

The final loss for the adapter training is the combination of the standard language loss and additional loss with the weighing factor of $\alpha$, $\mathcal{L}^{tot}=\mathcal{L}^{lm}+\alpha\mathcal{L}^{aux}$.

### 3.4 Further Fine-tuning For Downstream Task

Does the impact of vocabulary adaptation persist after fine-tuning the vocabulary adapted language model for strong performance in a target task?
We address this question for the cross-lingual generation task of machine translation, following the training recipe of ALMA (Xu et al., ) after the vocabulary adaptation process.
ALMA training begins with fine-tuning on monolingual data, followed by further weight optimization on small high-quality parallel data.
This two-step fine-tuning process ensures strong translation performance.
During the VocADT adaptation phase, we only update the new embeddings. Subsequently, and then update all model parameters in the task-oriented fine-tuning phases.

## 4 Which Languages Benefit the Most from Vocabulary Adaptation?

We aim to understand “When and how should we perform vocabulary adaptation?”.
More specifically, we seek insight into which languages might benefit the most from vocabulary adaptation in terms of improving overall performance or mitigating over-fragmentation.(^3^33 All non-English that we cover tend to be highly fragmented by a vocabulary of popular large language models such as Mistral (Figure 4), resulting in similar effects of mitigating over-fragmentation. Consequently, we emphasize the analysis of the performance aspect.)
To this end, we design experiments to cover 10 non-English languages along with English, listed in Table 1, with a variety of scripts and language families.
These languages are broadly categorized into three groups.
The first one is the Latin group of Swahili, Indonesian, Estonian, and Haitian, which are low- to mid-resource languages and all use Latin script.
The second is a Mixed group including Korean, Greek, Russian, and Bulgarian, which utilize a mixture of scripts.
We also experiment with grouping the Cyrillic script (Cyrillic group) languages.

We test individual language adaptation with language-specific vocabularies.
We also adapt several multilingual vocabularies that include English and four non-English languages in a single shared vocabulary, with each group corresponding to one of the previously mentioned groups.
This is to identify a language grouping strategy—whether to mix languages with different scripts or grouping languages with consistent scripts.

**Table 1: Covered Languages and its availability in multilingual benchmarks. We mainly categorize non-English languages by scripts—Latin group (2-5) and the Mixed group (6-7). We additionally experiment with Cyrillic group (8-11). We follow the resource-level of languages from Joshi et al. () and Üstün et al. ()**
| idx | Full Name | Short | Script | Resource | FLORES | XNLI | XCOPA | Belebele | MMMLU |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 1 | English | en | Latin | High | ✓ | ✓ |  | ✓ | ✓ |
| 2 | Swahili | sw | Latin | Low | ✓ | ✓ | ✓ | ✓ |  |
| 3 | Indonesian | id | Latin | Mid | ✓ |  | ✓ | ✓ | ✓ |
| 4 | Estonian | et | Latin | Mid | ✓ |  | ✓ | ✓ |  |
| 5 | Haitian Creole | ht | Latin | Low | ✓ |  | ✓ | ✓ |  |
| 6 | Korean | ko | Hangul | High | ✓ |  |  | ✓ |  |
| 7 | Greek | el | Greek | Mid | ✓ | ✓ |  | ✓ |  |
| 8 | Russian | ru | Cyrillic | High | ✓ | ✓ |  | ✓ | ✓ |
| 9 | Bulgarian | bg | Cyrillic | Mid | ✓ | ✓ |  | ✓ |  |
| 10 | Ukrainian | uk | Cyrillic | Mid | ✓ |  |  | ✓ | ✓ |
| 11 | Kazakh | kk | Cyrillic | Mid | ✓ |  |  | ✓ |  |

## 5 Experiment Design

### 5.1 Baselines and Modeling

We use Mistral-7B (Jiang et al., ) as our language model, along with its original vocabulary, which consists of 32k tokens ($|V^{o}|=32k$).
As baselines, we evaluate three state-of-the-art methods for vocabulary adaptation, ZeTT (Minixhofer et al., ), FOCUS (Dobler & de Melo, ), and OFA (Liu et al., ). For ZeTT and FOCUS, we experiment with language-specific vocabularies (ZeTT-mono, FOCUS-mono) as their implementations require specifying the language to adapt the vocabulary.
This results in separate adaptations per language, which could be hard to scale with larger language coverage.(^4^44 ZeTT does not support Ukrainian and Kazakh, therefore we primarily compare and average the results for 9 languages covered by both methods. See Appendix B for results for Ukrainian and Kazakh.)
For VocADT and OFA methods (VocADT-multi, OFA-multi), we experiment with multilingual vocabularies of five languages including English and four non-English languages, where we define three distinct language groups such as $\{$en, sw, id, et, ht$\}$ ( Latin group), $\{$en, ko, el, ru, bg$\}$ (Mixed group), and $\{$en, ru, bg, uk, kk$\}$ (Cyrillic group).

### 5.2 Training

#### Vocabulary.

We train SentencePiece (Kudo & Richardson, ) tokenizers on either language-specific corpora or a combined corpus, with a maximum of 2 million tokens per language, and create new vocabularies with a size of 50k for all cases including mono/multilingual vocabularies ($|V^{n}|=50k$).
Newly created vocabularies for each language group are shared across baselines.

#### Adapter Training.

In the adapter training phase, we train only the adapters, while fixing all parameters of the original model. The input and output adapters are separate modules, as preliminary results showed that sharing an adapter for the input and output sides performs worse.
We train 3.5B monolingual tokens mixed by 5 languages (English + 4 non-English from each corresponding group) and report test numbers from it.
We use “clean” documents from the corpus of MADLAD-400 (Kudugunta et al., ). We set the weighing factor of auxiliary loss $\alpha$ with 0.1 for non-Latin groups and 0 for the Latin group unless otherwise specified.
This is based on the empirical results in Figure 6 that maintaining the embeddings of overlapping tokens close to the original status during the adaptation is effective only for non-Latin script languages and counter-effective for Latin languages.
More details regarding the training are in Appendix A.

#### Full-Weight Fine-tuning.

After the adaptation phase, we follow the fine-tuning recipe of ALMA (Xu et al., ) that consists of full-weight training with monolingual corpus and a small amount of high-quality parallel corpus to enhance MT performance (§3.4).
We also include Mistral in this phase of training.
1) In monolingual fine-tuning, we use MADLAD-400. For adapted ZeTT and FOCUS models, we fine-tune each separate model with non-English language-specific vocabulary except for uk and kk for ZeTT (again, due to unsupported languages in ZeTT) using a total of 2B tokens combining English and the corresponding non-English. For Mistral and the adapted VocADT and OFA, we fine-tune separate models for all three non-English groups (Latin, Mixed, Cyrillic) plus English using a corpus of 5B monolingual tokens containing 5 languages.
2) In the next parallel training, we sample 15k bitext from NLLB dataset(^5^55[https://huggingface.co/datasets/allenai/nllb](https://huggingface.co/datasets/allenai/nllb)) for each English and non-English training pairs with top LASER3 scores (Artetxe & Schwenk, ; Schwenk et al., ; Heffernan et al., ; NLLB Team et al., ).
The parallel training is done for one epoch, and we report test set numbers with the best model of the validation set.
All the models are fine-tuned and tested with both directions of en-xx and xx-en within a single model, meaning there are no separate models for opposite translation directions.
We follow the prompting strategy of Xu et al. ().

### 5.3 Evaluation

We evaluate adaptation methods with multilingual benchmarks of various tasks including MT, natural language inference (NLI), common sense reasoning, and multiple choice question answering (QA).
For MT of English to non-English (en-xx) and non-English to English (xx-en), we use FLORES (Goyal et al., ; NLLB Team et al., ) as it supports all the languages that we experiment with.
We use five-shot MT prompting for the model from the adaptation phase, and zero-shot prompting for the model after the ALMA training phase.
We assess the translation quality with xCOMET-XL (Guerreiro et al., ), which produces a score of increasing quality ranging from 0 to 1.
For NLI and reasoning, we use XNLI (Conneau et al., ) and XCOPA (Ponti et al., ) with zero-shot prompting. For multiple choice QA, we use Belebele (Bandarkar et al., ) and Multilingual MMLU (Hendrycks et al., ; Lai et al., , MMMLU) with five shot prompting. All the tasks except for MT are classification tasks, where
we use the lm-evaluation-harness (Gao et al., ) evaluation tool and report accuracy.

Figure: Figure 2: Average scores of original Mistral and its adaptation with new vocabulary, only replacing embeddings and fixing the body of transformer modules. “-multi” indicates models with a multilingual vocabulary, which includes five languages covering all languages with two separate models, while “-mono” refers to models with monolingual vocabulary. Our VocADT adaptation outperforms the original Mistral and matches or exceeds ZeTT. xx-en and en-xx indicate MT tasks.
Refer to caption: x3.png

## 6 Vocabulary Adaptation Results and Analyses

### 6.1 Overall Task Performance

We first present the controlled comparison on diverse tasks of the original Mistral with new vocabulary variants obtained by our vocabulary adaptation approach (VocADT) and the ZeTT and OFA baselines.
Figure 2 presents the average performance across multiple multilingual MT, NLI, reasoning, and QA tasks.
Language-wise results are in Appendix B.
Overall, adapting the vocabulary using VocADT generally leads to better performance compared to the original Mistral model, and either surpasses or performs on par with ZeTT.
MMMLU is the only task where Mistral still holds the top spot; however, the performance gap between the new and original embeddings is smaller with VocADT approach than with ZeTT. Remarkably, VocADT-multi achieves these results with only two models for the eight languages tested, whereas ZeTT requires a separate model for each language.

### 6.2 Which Languages Benefit the Most from Vocabulary Adaptation?

Instead of “averaging out” the languages, we get closer look on the impact of vocabulary adaptation in a language-wise manner.
Figure 4 shows the increased rate of task performance and tokenization statistics for various languages after applying different vocabulary adaptation methods.
The results are shown for Latin group languages (Swahili, Indonesian, Estonian, and Haitian Creole) and Mixed group (Korean, Greek, Russian, and Bulgarian).
We compare VocADT and ZeTT against the original Mistral model.
The vertical axis of the increase rate is fixed to the positive range to see the benefit trends more easily and all the numbers including the negative range of the increase rate are in Appendix B.
We use the FLORES development set for counting the tokens by various tokenizers where the semantic contents of every language are the same.

#### Languages with Latin Scripts or Severe Fragmentation Benefit the Most

In Figures 4(a), we observe that Latin script languages consistently benefit from vocabulary adaptation, regardless of the task, adaptation method employed. However, even non-Latin languages show improvements when they suffer from severe over-fragmentation, as seen in the case of Greek in Figures 4(b).
Among the eight languages, Greek is the most fragmented by the Mistral tokenizer, and it demonstrates significant improvement after the adaptation to less fragmented vocabulary, particularly in MT tasks, while other non-Latin languages in Mixed group show zero or modest gains.

Figure: (a) Increase rate of task performance and number of token count of Latin group languages.
Refer to caption: x4.png

Figure: Figure 5: Comparison of task performance between two grouping strategies of Mixed-script and Cyrillic-script on two shared languages. Consistent script within a group provides minor benefits.
Refer to caption: x8.png

### 6.3 Does Script Matter for Language grouping?

Multilingual vocabularies for language groups can strike a balance between the extensive coverage of the original Mistral and the limited scope of language-specific monolingual models.
We investigate strategies for grouping, in particular the effect of script.

Figure 5 compares the performance and token count reduction between two non-English grouping strategies: Mixed-script (ko, el, ru, bg) and Cyrillic-script (ru, bg, uk, kk) languages.
The results are shown for the common languages of Russian and Bulgarian across several tasks.
For Russian, the consistent script language group performs better, especially in the MT task, though the improvement rate is not substantial.
For Bulgarian, both grouping strategies deliver nearly identical results.
In summary, the results suggest that maintaining a consistent script within a group enhances performance, though overall outcomes seem to be influenced more by the language type itself than by the grouping strategy.

### 6.4 Is the Auxiliary Loss Helpful?

We examine the effects of the auxiliary loss that aims to mitigate the divergence of original status for overlapping words as described in Section 3.3.
Figure 6 illustrates the impact on task performance of vocabulary adaptation with and without auxiliary loss on Latin and Mixed group vocabulary.
We report one language in each group along with English.

For the Mixed group plus English vocabulary (right plot of Figure 6), maintaining the embedding values of overlapping words shows slight effectiveness.
However, for Latin languages (left plot of Figure 6), applying auxiliary loss ($\alpha=0.1$) leads to a significant decline in overall task performance.
We hypothesize that non-Latin languages are less prone to have word collisions with the original vocabulary compared to the Latin group, as the Mistral model is largely English (Latin) centric.
As a result, retaining the established embeddings for overlapped words in Latin group vocabulary and Mistral vocabulary may disrupt effective learning due to the possible similarity in scripts with English.
On the other hand, keeping the original embeddings during the adaptation for overlapping tokens in Mixed may be helpful to maintain the already established embeddings for overlapped tokens while adjusting the embeddings for new non-Latin script tokens.

Figure: Figure 6: Effects of the auxiliary loss on various settings. Keeping the embedding of overlapping words close to the original is ineffective for Latin groups but is effective in Mixed vocabulary.
Refer to caption: x10.png

**Table 2: MT performance after full-weight fine-tuning the new vocabulary-adapted model. The symbol “#” indicates the number of separate models for this experiment table. All vocabulary adaptation approaches after fine-tuning are effective compared to Mistral except for en-sw. VocADT-multi shows the best average performance in both directions while matching the score of ZeTT in xx-en.**
| FLORES | xx-en | en-xx |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Lang (group) $\downarrow$ | VocADT | Mistral | ZeTT | OFA | FOCUS | VocADT | Mistral | ZeTT | OFA | FOCUS |
| sw (Latin) | 0.893 | 0.891 | 0.897 | 0.889 | 0.893 | 0.753 | 0.770 | 0.762 | 0.763 | 0.762 |
| id (Latin) | 0.951 | 0.950 | 0.953 | 0.951 | 0.945 | 0.874 | 0.874 | 0.879 | 0.871 | 0.876 |
| et (Latin) | 0.939 | 0.925 | 0.941 | 0.937 | 0.939 | 0.852 | 0.868 | 0.856 | 0.845 | 0.870 |
| ht (Latin) | 0.706 | 0.696 | 0.699 | 0.699 | 0.685 | 0.336 | 0.336 | 0.334 | 0.332 | 0.339 |
| ko (Mixed) | 0.886 | 0.892 | 0.898 | 0.897 | 0.906 | 0.755 | 0.715 | 0.772 | 0.742 | 0.754 |
| el (Mixed) | 0.922 | 0.845 | 0.910 | 0.894 | 0.902 | 0.876 | 0.817 | 0.857 | 0.862 | 0.861 |
| ru (Mixed) | 0.945 | 0.894 | 0.944 | 0.946 | 0.945 | 0.889 | 0.828 | 0.872 | 0.882 | 0.868 |
| bg (Mixed) | 0.953 | 0.904 | 0.952 | 0.950 | 0.949 | 0.895 | 0.844 | 0.895 | 0.899 | 0.895 |
| Avg (8 pairs) | 0.899 | 0.875 | 0.899 | 0.895 | 0.895 | 0.779 | 0.757 | 0.778 | 0.774 | 0.778 |
| uk (Cyrillic) | 0.941 | 0.941 | _ | 0.936 | 0.928 | 0.878 | 0.876 | _ | 0.868 | 0.840 |
| kk (Cyrillic) | 0.881 | 0.875 | _ | 0.875 | 0.880 | 0.807 | 0.790 | _ | 0.785 | 0.796 |
| Avg (10 pairs) | 0.902 | 0.881 | _ | 0.897 | 0.897 | 0.792 | 0.772 | _ | 0.785 | 0.786 |
| # of Models $\rightarrow$ | 3 | 3 | 8 | 3 | 10 | 3 | 3 | 8 | 3 | 10 |

## 7 Impact of Vocabulary Adaptation on Downstream Fine-tuning

Do the effects of vocabulary adaptation hold up after fine-tuning the adapted language model? After completing the VocADT process, which keeps non-embedding model weights fixed, we update the full weights of the adapted model to enhance MT performance following ALMA (Xu et al., ).

As can be seen in Table 2, all vocabulary adaptation approaches are effective compared to Mistral except for en-sw, and among those, our approach (VocADT) achieves the highest average score in both en-xx and xx-en directions.
In the xx-en direction, the performance of VocADT matches that of ZeTT, despite using a smaller number of individual models of the same size (2 VocADT vs 8 ZeTT).
Interestingly, language-specific models (ZeTT, FOCUS) tend to excel in Latin languages, whereas multilingual models (Mistral, VocADT, OFA) generally outperform language-specific models in non-Latin cases.

In sum, with full parameter fine-tuning after the vocabulary adaptation, our VocADT model offers a competitive edge across both xx-en and en-xx tasks, further validating the effectiveness of our approach.
VocADT demonstrates that a multilingual model can achieve or surpass language-specific models like ZeTT, offering a more flexible and scalable solution for handling multiple languages.

## 8 Conclusion

We propose a simple and effective vocabulary adaptation method using a vocabulary adapter that addresses key challenges in existing approaches.
Our approach consistently outperforms the original Mistral model after the adaptation phase across various tasks and after the following full-weight finetuning on machine translation.
Furthermore, our methods prove more effective than, or at least on par with, strong vocabulary adaptation baselines, without relying on external embeddings or language constraints, offering a flexible and scalable solution for handling multiple languages.
Our experiments cover a wide range of languages and scripts, revealing that languages with Latin scripts or severe fragmentation benefit the most, and that maintaining the embedding of overlapping words proves helpful for non-Latin languages but is less effective for Latin groups.
We also explored different grouping strategies, finding that while maintaining consistent scripts within a group offers some benefits, the impact is relatively minor.
Lastly, with a focus on machine translation, we confirm that vocabulary adaptation remains effective even after fine-tuning, with VocADT emerging as the most effective approach.

## Appendix A Details of Training

This is a section for additional training details.
We use four Nvidia A100 GPUs for adapter training and 16 AMD MI200 GPUs for full-weight fine-tuning.
For all monolingual training including adaptation phase and fine-tuning phase, we follow (Xu et al., ) for setting the sampling ratio of monolingual data to mitigate language balance in the monolingual data and avoid prioritizing English.
The method fixes the sampling ratio for English with a certain probability e.g. $1/n$ if there are $n$ languages to mix and allocate the remaining ratio (e.g. $\frac{n-1}{n}$) by employing temperature sampling suggested by Aharoni et al. ().
We mix the monolingual data for Latin group $\{$en, sw, id, et, ht$\}$ with $\{$17%, 16%, 32%, 23%, 12%$\}$ ratio, for Mixed group $\{$en, ko, el, ru, bg$\}$ with $\{$17%, 17%, 19%, 30%, 17%$\}$, and for Cyrillic $\{$en, ru, bg, uk, kk$\}$ with $\{$17%, 32%, 18%, 20%, 13%$\}$.
In adapter training for VocADT, we use a (peak) learning rate of 2e-6 with a cosine scheduler, a maximum sequence length of 512 tokens, a warm-up ratio of 0.01, and a weight decay of 0.01.
In full-weight fine-tuning phase, we mostly follow the training setting from ALMA.

## Appendix B Language-wise Results of Vocabulary Adaptations

**Table 3: xx-en MT results with xCOMET-XL score and the increase rate from the original Mistral after the vocabulary adaptation—only replacing embeddings while fixing the rest.**
| MT (xx-en) | Mistral | VocADT-multi (Ours) | ZETT-mono | OFA-multi | FOCUS-mono |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| sw-en | 0.485 | 0.801 | 65.32% | 0.734 | 51.50% | 0.215 | -55.75% | 0.216 | -55.53% |
| id-en | 0.946 | 0.942 | -0.44% | 0.880 | -7.01% | 0.246 | -73.97% | 0.186 | -80.36% |
| et-en | 0.722 | 0.899 | 24.46% | 0.769 | 6.40% | 0.196 | -72.93% | 0.248 | -65.70% |
| ht-en | 0.554 | 0.669 | 20.72% | 0.484 | -12.55% | 0.249 | -54.96% | 0.212 | -61.72% |
| ko-en | 0.882 | 0.755 | -14.39% | 0.662 | -24.87% | 0.189 | -78.56% | 0.318 | -63.99% |
| el-en | 0.438 | 0.760 | 73.59% | 0.664 | 51.69% | 0.182 | -58.54% | 0.250 | -42.87% |
| ru-en | 0.959 | 0.927 | -3.33% | 0.882 | -8.06% | 0.249 | -74.06% | 0.264 | -72.43% |
| bg-en | 0.952 | 0.918 | -3.56% | 0.896 | -5.93% | 0.228 | -76.00% | 0.271 | -71.55% |
| Avg (8 pairs) | 0.742 | 0.834 | 12.35% | 0.746 | 0.56% | 0.219 | -70.47% | 0.246 | -66.92% |
| uk-en | 0.944 | 0.915 | -3.07% | _ | _ | 0.201 | -78.70% | 0.288 | -69.49% |
| kk-en | 0.411 | 0.763 | 85.82% | _ | _ | 0.190 | -53.72% | 0.308 | -24.98% |
| Avg (10 pairs) | 0.729 | 0.835 | 14.49% | _ | _ | 0.215 | -70.59% | 0.256 | -64.89% |
| Total # of Models | 1 | 3 | 8 | 3 | 10 |  |  |  |  |

**Table 4: en-xx MT results with xCOMET-XL score and the increase rate from the original Mistral after the vocabulary adaptation—only replacing embeddings while fixing the rest.**
| MT (en-xx) | Mistral | VocADT-multi (Ours) | ZETT-mono | OFA-multi | FOCUS-mono |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| en-sw | 0.238 | 0.562 | 135.88% | 0.673 | 182.29% | 0.342 | 43.54% | 0.209 | -12.23% |
| en-id | 0.778 | 0.837 | 7.65% | 0.836 | 7.47% | 0.436 | -43.99% | 0.224 | -71.26% |
| en-et | 0.309 | 0.643 | 108.37% | 0.652 | 111.23% | 0.405 | 31.12% | 0.247 | -19.86% |
| en-ht | 0.308 | 0.329 | 7.03% | 0.339 | 10.11% | 0.227 | -26.38% | 0.235 | -23.61% |
| en-ko | 0.703 | 0.598 | -14.99% | 0.631 | -10.24% | 0.309 | -56.06% | 0.258 | -63.33% |
| en-el | 0.384 | 0.413 | 7.56% | 0.524 | 36.71% | 0.232 | -39.58% | 0.215 | -43.90% |
| en-ru | 0.900 | 0.854 | -5.17% | 0.719 | -20.10% | 0.388 | -56.87% | 0.371 | -58.80% |
| en-bg | 0.899 | 0.859 | -4.43% | 0.755 | -16.02% | 0.332 | -63.03% | 0.289 | -67.80% |
| Avg (8 pairs) | 0.565 | 0.637 | 12.77% | 0.641 | 13.53% | 0.334 | -40.89% | 0.256 | -54.66% |
| en-uk | 0.865 | 0.851 | -1.59% | _ | _ | 0.318 | -63.23% | 0.310 | -64.14% |
| en-kk | 0.222 | 0.522 | 135.11% | _ | _ | 0.294 | 32.25% | 0.223 | 0.65% |
| Avg (10 pairs) | 0.560 | 0.647 | 15.40% | _ | _ | 0.328 | -41.44% | 0.258 | -53.93% |
| Total # of Models | 1 | 3 | 8 | 3 | 10 |  |  |  |  |

**Table 5: XNLI results with Accuracy score and the increase rate from the original Mistral after the vocabulary adaptation—only replacing embeddings while fixing the rest.**
| XNLI | Mistral | VocADT-multi (Ours) | ZETT-mono | OFA-multi | FOCUS-mono |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| en | 0.550 | 0.553 | 0.47% | 0.554 | 0.73% | 0.547 | -0.47% | 0.537 | -2.30% |
| sw | 0.353 | 0.398 | 12.63% | 0.453 | 28.33% | 0.345 | -2.16% | 0.325 | -7.96% |
| el | 0.419 | 0.387 | -7.60% | 0.380 | -9.31% | 0.330 | -21.21% | 0.337 | -19.58% |
| ru | 0.488 | 0.490 | 0.48% | 0.474 | -2.87% | 0.347 | -28.98% | 0.331 | -32.27% |
| bg | 0.425 | 0.457 | 7.63% | 0.450 | 5.88% | 0.344 | -19.02% | 0.371 | -12.59% |
| Avg (5 langs) | 0.447 | 0.457 | 2.24% | 0.462 | 3.40% | 0.383 | -14.38% | 0.380 | -14.93% |
| Total # of Models | 1 | 2 | 5 | 2 | 5 |  |  |  |  |

**Table 6: XCOPA results with Accuracy score and the increase rate from the original Mistral after the vocabulary adaptation—only replacing embeddings while fixing the rest.**
| XCOPA | Mistral | VocADT-multi (Ours) | ZETT-mono | OFA-multi | FOCUS-mono |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| sw | 0.510 | 0.574 | 12.55% | 0.576 | 12.94% | 0.564 | 10.59% | 0.544 | 6.67% |
| id | 0.584 | 0.608 | 4.11% | 0.598 | 2.40% | 0.508 | -13.01% | 0.512 | -12.33% |
| et | 0.470 | 0.538 | 14.47% | 0.536 | 14.04% | 0.516 | 9.79% | 0.520 | 10.64% |
| ht | 0.514 | 0.548 | 6.61% | 0.572 | 11.28% | 0.526 | 2.33% | 0.534 | 3.89% |
| Avg (4 langs) | 0.520 | 0.567 | 9.14% | 0.571 | 9.82% | 0.529 | 1.73% | 0.528 | 1.54% |
| Total # of Models | 1 | 1 | 4 | 1 | 5 |  |  |  |  |

**Table 7: Belebele results with Accuracy score and the increase rate from the original Mistral after the vocabulary adaptation—only replacing embeddings while fixing the rest.**
| Belebele | Mistral | VocADT-multi (Ours) | ZETT-mono | OFA-multi | FOCUS-mono |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| en | 0.843 | 0.833 | -1.18% | 0.780 | -7.51% | 0.546 | -35.31% | 0.367 | -56.52% |
| sw | 0.391 | 0.440 | 12.50% | 0.476 | 21.61% | 0.248 | -36.65% | 0.252 | -35.51% |
| id | 0.647 | 0.638 | -1.38% | 0.610 | -5.67% | 0.289 | -55.33% | 0.230 | -64.43% |
| et | 0.439 | 0.538 | 22.53% | 0.380 | -13.42% | 0.250 | -43.04% | 0.213 | -51.39% |
| ht | 0.397 | 0.507 | 27.72% | 0.507 | 27.73% | 0.248 | -37.54% | 0.240 | -39.50% |
| ko | 0.666 | 0.616 | -7.52% | 0.466 | -30.05% | 0.278 | -58.27% | 0.274 | -58.77% |
| el | 0.442 | 0.566 | 27.90% | 0.468 | 5.79% | 0.287 | -35.17% | 0.284 | -35.68% |
| ru | 0.727 | 0.696 | -4.29% | 0.459 | -36.85% | 0.248 | -65.90% | 0.239 | -67.13% |
| bg | 0.674 | 0.698 | 3.47% | 0.446 | -33.93% | 0.276 | -59.14% | 0.233 | -65.40% |
| Avg (9 langs) | 0.581 | 0.614 | 5.83% | 0.510 | -12.16% | 0.296 | -48.95% | 0.259 | -55.35% |
| uk | 0.728 | 0.693 | -4.76% | _ | _ | 0.254 | -65.05% | 0.231 | -68.25% |
| kk | 0.364 | 0.442 | 21.36% | _ | _ | 0.256 | -29.87% | 0.220 | -39.63% |
| Avg (11 langs) | 0.574 | 0.606 | 5.50% | _ | _ | 0.289 | -49.70% | 0.253 | -55.93% |
| Total # of Models | 1 | 3 | 9 | 3 | 11 |  |  |  |  |

**Table 8: Multilingual MMLU results with Accuracy score and the increase rate from the original Mistral after the vocabulary adaptation—only replacing embeddings while fixing the rest.**
| MMMLU | Mistral | VocADT-multi (Ours) | ZETT-mono | OFA-multi | FOCUS-mono |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| en | 0.607 | 0.577 | -4.88% | 0.537 | -11.50% | 0.464 | -23.48% | 0.288 | -52.61% |
| id | 0.468 | 0.410 | -12.49% | 0.316 | -32.53% | 0.256 | -45.34% | 0.269 | -42.60% |
| ru | 0.500 | 0.468 | -6.39% | 0.348 | -30.34% | 0.259 | -48.25% | 0.272 | -45.55% |
| Avg (3 langs) | 0.525 | 0.485 | -7.62% | 0.400 | -23.73% | 0.326 | -37.84% | 0.276 | -47.39% |
| uk | 0.489 | 0.462 | -5.57% | _ | _ | 0.269 | -45.06% | 0.253 | -48.19% |
| Avg (4 langs) | 0.516 | 0.479 | -7.14% | _ | _ | 0.312 | -39.55% | 0.270 | -47.58% |
| Total # of Models | 1 | 3 | 3 | 3 | 4 |  |  |  |  |

**Table 9: Tables of various vocabulary adaptation methods.**
| Vocabulary<br>Adaptation | Grouping | #<br>Langs | External<br>Resources | Base<br>Model | Generative<br>Task |
| --- | --- | --- | --- | --- | --- |
| vADT (Ours) | multilingual<br>(Latin, Mixed,<br>Cyrillic group) | 11 | x | Mistral | MT |
| ZeTT<br>(Minixhofer et al., ) | lang-specific | 26 | x | Mistral,<br>XLM-R | x |
| RAMEN<br>(Tran, ) | lang-specific | 6 | FastAlign,<br>fastText | BERT,<br>RoBERTa | x |
| FVT<br>Gee et al. () | English,<br>domain-specific | 1 (en) | x | BERT | x |
| VIPI<br>Mosin et al. () | English,<br>domain-specific | 1 (en) | x | BERT | x |
| OFA<br>(Liu et al., ) | multilingual<br>(all in 401k) | min 369 | ColexNet+ | XLM-R,<br>RoBERTa | x |
| FOCUS<br>(Dobler & de Melo, ) | lang-specific | 10 | fastText | XLM-R | x |
| MAD-X<br>(Pfeiffer et al., ) | lang-specific | 16 | x | XLM-R | x |
| WECHSEL<br>(Minixhofer et al., ) | lang-specific | 8 | fastText,<br>bilingual<br>dictionaries | RoBERTa,<br>GPT-2 | x |
| CW2V<br>Mundra et al. () | multilingual<br>(all 4) | 4 | bilingual<br>dictionaries | LLaMA2,<br>RoBERTa | MT,<br>summarization |
| CLP<br>(Ostendorff & Rehm, ) | lang-specific | 1 (de) | GPT2-base w<br>WECHSEL | GPT2,<br>BLOOM | x |
| CLP+<br>(Yamaguchi et al., ) | lang-specific | 4 | GPT2-base w<br>WECHSEL | BLOOM-1/7B,<br>TigerBot- 7B,<br>Mistral-7B | summarization |
| Downey et al. () | lang-specific<br>& multilingual<br>(Mixed, Uralic family) | 10 | x | XLM-R | x |

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 6
presentation: 4
questions: 1. Adaptation improves performance for fragmented Greek but less so for similarly fragmented Korean. Does this suggest vocabulary adaptation may not be effective for highly fragmented languages, and that other properties of Greek contribute to the improvements?
2. Was any ablation conducted comparing initialization methods (e.g., one-hot vs. multi-hot) for the adapter module and their impact on downstream task performance?
3. How effective would VocADT be for mixed-script or multilingual groups with shared vocabulary at different scales? Exploring grouping strategies in larger, mixed-language models could provide valuable insights into scalability for multilingual setups.
soundness: 3
strengths: 1. The introduction of VocADT, an efficient, adapter-based approach to vocabulary adaptation without relying on external embeddings, is novel.
2. The comprehensive experimental evaluation covers a wide range of multilingual tasks and scripts across 11 languages. It compares the proposed method against three baselines to assess the benefits of vocabulary adaptation and the robustness of VocADT.
3. The paper is well-structured and clearly presented.
4. VocADT’s method for efficient adaptation without altering main model weights is well-motivated, especially for cross-lingual transfer and adaptation to low-resource languages.
summary: The paper introduces VocADT, an adapter-based method for vocabulary adaptation in pre-trained language models. New embeddings for an expanded vocabulary are learned by placing adapter modules between the new vocabulary and the original language model embeddings, enabling the model to use new tokens without modifying its main weights. The weights of the adapter module are initialized by (i) copying original embeddings of overlapping tokens, (ii) averaging embeddings where partitioned tokens in the original vocabulary subset exist, and (iii) randomly initializing the rest. An auxiliary loss is introduced to keep the embeddings of overlapping tokens close to their original values, preserving established representations for these tokens in the language model. The robustness of VocADT's impact on LLM performance for downstream tasks is tested by fine-tuning all model parameters after vocabulary adaptation. Detailed results are provided, evaluating 11 languages across 3 language groups — Latin, Mixed, and Cyrillic. VocADT is compared with three baseline methods: ZeTT, FOCUS, and OFA. Results show that, across languages and tasks, VocADT outperforms the original LM (Mistral). Latin languages benefit most from vocabulary adaptation, with language grouping strategies playing a minor role. Additionally, the auxiliary loss improves performance for Mixed-group languages but not for Latin languages. Finally, following ALMA, the authors fine-tune the full model to evaluate MT performance, showing that VocADT improves performance compared to the original Mistral baseline.
weaknesses: 1. Given that grouping Mixed-scripts and Cyrillic scripts had little impact on performance, it would have been more interesting to see if grouping all languages could have achieved identical results, demonstrating the language scalability of the method.
2. The auxiliary loss, intended to retain original embeddings for overlapping tokens, lacks analysis across alpha values; further exploration in non-Latin languages might have improved performance.
3. The paper evaluates the proposed method on Mistral alongside several baselines. However, including at least one additional candidate LM would have strengthened the results.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Can you give an example where the case 3 mentioned in section 3.2 happen?
soundness: 3
strengths: 1. The methodology is clearly explained with reference to the prior work.
2. The experiment design of this paper is well-crafted and supports the claims of the paper.
summary: The authors propose a new vocabulary adaptation method that enables new words to be added to a pretrained LLM's vocabulary. The proposed approach uses adapters to learn embeddings for new words. The new embeddings are linear combinations of the existing embeddings. The approach is tested on 11 languages, showing that the best results are achieved for highly fragmented and Latin script based languages.
weaknesses: 1. The motivation behind restricting the new embedding to a linear combination of original embeddings has not been explained.
2. Since the approach requires some amount of training, the authors should report the computational cost of their approach compared to the baselines.
3. The results in Appendix B shows significant difference among performance gains of different languages. The authors should perform analysis to determine why this happens. Does the amount of new pretraining data used for vocabulary adaptation affect the results?

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Can you further explain the limitations of heuristics-based methods and methods that rely on external embeddings? For example, what does it mean when the introduction states that "heuristics-based methods often lack adaptability"?

2. Is your proposed method, VocADT, effective for low-resource languages?
soundness: 3
strengths: 1. The method proposed in this article allows for vocabulary adaptation by keeping the original model completely frozen and training only the adapter, which is highly efficient.

2. Experimental results demonstrate that the method presented in this paper can effectively improve the performance.

3. The article provides a detailed analysis of the MT task, offering a more accurate assessment of the model on cross-lingual tasks.
summary: The article introduces VocADT, a method to conduct vocabulary adaptation with adapters. This method enables the training of new vocabularies while keeping all parameters of the original model fixed. Experimental results demonstrate that VocADT outperforms existing baselines. The study finds that languages with Latin scripts and those with severe fragmentation benefit the most from vocabulary adaptation. Additionally, the article validates that incorporating new languages through vocabulary adaptation improves performance in machine translation tasks.
weaknesses: 1. The article does not clearly explain or analyze the advantages of VocADT compared to heuristic-based methods and those relying on external embeddings or networks. This causes a disconnect between the claims about existing methods' shortcomings in the introduction and the proposed method.

2. In VocADT, the initialization of embeddings and the use of auxiliary loss are very similar to existing work, raising concerns about the novelty of the paper.

3. The design of the adapter in this paper assumes that the new embeddings are solely linear combinations of the old embeddings. This assumption could negatively impact performance. For example, languages with Latin scripts may perform better under this assumption, while languages with greater differences, such as Korean, might not fit this assumption effectively.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: 1) The number of tokens to train baseline methods is unknown. Is it same with the VocADT?
2) It can be found that the language Korean also suffers from severe fragment in Figure 3, while the improvement on Korean is little. How to interpret this phenomenon?
soundness: 2
strengths: The findings about the impact of language scripts are interesting, which are the Latin-script and severe fragmented languages by Mistral-7b tokenizer benefit the most after vocabulary adaptation.
summary: This paper proposes a method to address the vocabulary adaptation problem in multilingual NLP, and investigates the impacts of language scripts after vocabulary adaptation. After experiments using Mistral-7b on 11 languages, they find that the method proposed achieves better or on par with the vanilla model and the baseline methods. And the Latin-script and severe fragmented languages by Mistral-7b tokenizer benefit the most after vocabulary adaptation.
weaknesses: - [addressed]**Lack of novelty**: The method proposed is an incremental work, which combines the heuristic methods like OFA in the initialization and language adapters like MAD-X. It incorporates additional parameters and computation for the adapter and 8.5B(3.5B+5B) multilingual corpora in adaptation. Moreover, the incorporated auxiliary loss is not effective in Figure 5. The improvement seems little other than the "en-el" when using this auxiliary loss ($\alpha=0.1$).

- **Missing details and additional experiments to support the findings**: only Mistral-7b is used in this work. It is better to conduct experiments on other base models like llama or gemma to improve the generalization of the findings. The detail settings for baseline methods are unknown in this paper.

- **Writing is not clear**: 
  1) Line 067: ";" --> "?"
  2) Figure 1:  "Vocab Adaptor" --> "Vocab Adapter"
  3) Line 148: "lightweight": I cannot agree with this for the additional parameters and computation from the adapter training.
  4) The Figure 3 is confusing and hard to follow: a) The y-axis of 3(a) left part are increase rate(%), thus the negative results are missing like "id-en", "belebele" on Indonesian? b). The mismatched legends for left and right part, which are "VocADT-multi(ours), ZETT-mono"(left) and "mono, multi, mistral"(right). These results will be better to report on Tables.

Meta-review: This paper presents VocADT, a method for multilingual vocabulary adaptation using adapters. Specifically, rather than altering the embedding layer of the multilingual model to create a new multilingual vocabulary, the work proposes using adapters to learn linear combinations of the existing embeddings for the new vocabulary. The comprehensive experiments show that VocADT improves performance over multiple baselines across different language types and resourcefulnesses. 

Strengths:
- Well-motivated, novel approach to the issue of multilingual tokenization, and is particularly useful given that it does not modify the original model weights and introduces a limited number of new parameters in the adapter models.
- Comprehensive experimental setting with many different tasks across 11 languages, including cross-lingual tasks such as translation. 
The paper is well-written, and the experiments are well-designed and clearly explained (Reviewer XRrF, CSrD).

In the rebuttal, the authors provided additional results with a second language model -- the next version of the paper should also include the additional results with Llama.

Weaknesses: 
- Some of the parameters and experimental settings considered are not ablated, such as the auxiliary loss alpha and the choice to divide representations across scripts versus grouping them all together (Reviewer XRrF).
- The paper presents a limited analysis of why some languages benefit more from the approach than others (e.g., the differences seen for Greek and Korean).

The authors should also include the details raised by reviewers and clarified in the rebuttal in the next version; these include the choice to limit embedding combinations to linear ones (Reviewer CSrD, qgGK) and computational costs of the experiments (Reviewer CSrD), among others.

Decision: Accept (Poster)