TokAlign: Efficient Vocabulary Adaptation via Token Alignment
Authors: Chong Li, Jiajun Zhang, Chengqing Zong
Accepted at: acl25

TokAlign: Efficient Vocabulary Adaptation via Token Alignment

Chong Li, Jiajun Zhang*, Chengqing Zong
State Key Laboratory of Multimodal Artificial Intelligence Systems,
Institute of Automation, CAS, Beijing, China
School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
lichong2021@ia.ac.cn, {jjzhang, cqzong}@nlpr.ia.ac.cn

Abstract

Tokenization serves as a foundational step for Large Language Models (LLMs) to process text. In new domains or languages, the inefficiency of the tokenizer will slow down the training and generation of LLM. The mismatch in vocabulary also hinders deep knowledge transfer between LLMs like token-level distillation. To mitigate this gap, we propose an efficient method named TokAlign to replace the vocabulary of LLM from the token co-occurrences view, and further transfer the token-level knowledge between models. It first aligns the source vocabulary to the target one by learning a one-to-one mapping matrix for token IDs. Model parameters, including embeddings, are rearranged and progressively fine-tuned for the new vocabulary. Our method significantly improves multilingual text compression rates and vocabulary initialization for LLMs, decreasing the perplexity from \(3.4e^2\) of strong baseline methods to \(1.2e^2\) after initialization. Experimental results on models across multiple parameter scales demonstrate the effectiveness and generalization of TokAlign, which costs as few as 5k steps to restore the performance of the vanilla model. After unifying vocabularies between LLMs, token-level distillation can remarkably boost (+4.4% than sentence-level distillation) the base model, costing only 235M tokens.¹

1 Introduction

Large language models (Touvron et al., 2023a; OpenAI, 2023; Yang et al., 2024) first tokenize text input into several tokens during inference and training, which compresses text and addresses the out-of-vocabulary problem (Sennrich et al., 2016; Wu et al., 2016; Kudo, 2018). However, the low compression rate of vanilla tokenizers on new languages or domains decelerates the training and inference process. As shown in Figure 1, the compression rate of capable large language models like LLaMA3 (Meta, 2024) on low-resource languages still largely lags behind the others. For example, Armenian text is 3.95x longer in tokens than English text under the same byte size with the LLaMA3 tokenizer. On the other hand, each LLM has specific strengths and weaknesses, which arise from its pre-training corpus and method. The mismatch in the vocabulary impedes the deep knowledge transfer between them like token-level distillation and ensemble (Xu et al., 2024; Lu et al., 2024). Considering the huge cost of re-training LLM for a new tokenizer, it is important to investigate efficient vocabulary adaptation methods.

To address the problems above, we introduce a novel method called TokAlign for large language models from a view of token-token co-occurrences. It is motivated by the general process of training an LLM: the pre-training corpus is first tokenized into tokens, and then input into the model. Given the same pre-training corpus, different tokenizers result in various sequences of token IDs, while the semantic and syntactic information is

![The compression rates of tokenizers across different domains and languages, which are still low in the code domain and low-resource languages for most of tokenizers. Refer to Table 6 in Appendix B.1 for more details.](page_1012_370_388_282.png)

Figure 1: The compression rates of tokenizers across different domains and languages, which are still low in the code domain and low-resource languages for most of tokenizers. Refer to Table 6 in Appendix B.1 for more details.

¹Corresponding author.
²Our codes and model are available at https://github.com/ZNLP/TokAlign
preserved in the token-token co-occurrence. Therefore, TokAlign strives to align token IDs from the original vocabulary and the target ones based on the global token-token co-occurrence matrix (Pennington et al., 2014) and learns a token-token alignment matrix. We further propose two metrics to evaluate the performance of the token-token alignment matrix based on text matching and semantic similarity. Given the learned alignment matrix, the new target embedding and language modeling head of LLM ("lm_head" in the Transformers (Wolf, 2019)) are initialized from the parameters of the most similar source token. Further vocabulary adaptation process is divided into a progressive two-stage procedure to improve the stability of convergence.

Given a target multilingual vocabulary for substitution, the model trained on the English corpus obtains a good initialization, decreasing the perplexity from 3.4e^2 to 1.2e^2, and improves 29.2% compression rates across 13 languages on average. The training process of TokAlign is 1.92x faster than strong baseline methods, and does not require additional hundreds of GPU hours to train a hyper-network for embedding initialization (Minixhofer et al., 2024). Experimental results on models across different scales show that as few as 5k steps are needed for our method to recover the performance of vanilla models on the general domain. Moreover, unifying vocabulary between models further facilitates the token-level distillation, which is 4.4% better than the sentence-level distillation on the same corpus. The performance of the 1B model is comparable with the vanilla 7B model after token-level distillation from a capable LLM. In summary, our contributions are as follows:

• We propose an unsupervised method to align token IDs between two vocabularies and replace the vocabulary of LLMs from the token-token co-occurrence view.

• We introduce two metrics to evaluate the performance of the token-level alignment matrix learned, which are proportional to the initial loss of pre-training.

• Experimental results on ten datasets show that our method promotes the cross-lingual knowledge transfer among multiple languages and deep knowledge transfer between models like token-level distillation.

2 Related Works

Our work is related to word representation, large language models, and vocabulary adaption, which will be briefly introduced below.

Word Representation Based on the distributional semantic hypothesis, Bengio et al. (2003) introduced the neural probabilistic language model to learn word representation. Researchers mainly focus on improving the effectiveness during learning word representations (Mikolov et al., 2013a,b; Bojanowski et al., 2017; Li et al., 2017; Wang et al., 2018), which provide a good initialization for neural networks like LSTM and GRU (Hochreiter, 1997; Chung et al., 2014). GloVe (Pennington et al., 2014) provides a method to train word representations from a view of global word-word co-occurrence matrix decomposition. It motivates us to train a word representation for each token and align tokens from statistical co-occurrence information in the pre-training corpus.

Large Language Model Through scaling in the parameters and pre-training corpus (Kaplan et al., 2020; Hoffmann et al., 2022), large language models like GPT-4 and LLaMA3 (Radford et al., 2018, 2019; Brown et al., 2020; OpenAI, 2023; Touvron et al., 2023a,b; Meta, 2024; GLM et al., 2024) demonstrate impressive performance across multiple tasks. However, the mismatch in the vocabulary greatly hinders the deep knowledge transfer between different models. We aim to mitigate this problem by introducing an efficient method to replace the tokenizer of a large language model.

Vocabulary Adaption is investigated mainly in the multilingual domain, especially the cross-lingual knowledge transfer problem (Scao et al., 2023; Muenninghoff et al., 2023; Yang et al., 2023; Zhu et al., 2023; Üstün et al., 2024; Li et al., 2024; Liu et al., 2024; Minixhofer et al., 2024; Yamaguchi et al., 2024; Mundra et al., 2024; Balde et al., 2024). It aims to improve the encoding effectiveness of tokenizer on corpora from new languages or domains, and is often implemented by extending the original vocabulary (Tran, 2020; Chau et al., 2020; Minixhofer et al., 2022; Dobler and de Melo, 2023; Downey et al., 2023). Most methods, like Focus (Dobler and de Melo, 2023), rely on the tokens belonging to both source vocabulary and target vocabulary to initialize the other new tokens in the target vocabulary. Our method differs from
these studies for the whole replacement of vocabulary and does not rely on the tokens in both source vocabulary and target vocabulary.

The pipeline of TokAlign to adapt vocabulary is similar to WECHSEL (Minixhofer et al., 2022), while the main difference lies in the representation and alignment of tokens. WECHSEL requires a bilingual dictionary and word representation to align tokens and calculates the similarity between tokens by tokenizing all words in the dictionary and linearly composing word representations. In contrast, TokAlign conducts token representation learning and alignment in an unsupervised way, which can apply to languages without bilingual dictionaries.

3 Method: TokAlign

3.1 Vocabulary Alignment

As shown in Figure 2, there are three steps for TokAlign to align two vocabularies from the token-token co-occurrence information. We denote the source tokenizer as Tokenizer_s, which has \( \mathcal{V}_s \) tokens, and the target tokenizer as Tokenizer_t with \( \mathcal{V}_t \) tokens, correspondingly.

Step 1: Tokenization The comprehensiveness of the pre-training corpus is important to obtain a well-trained token representation. An unbalanced corpus makes it hard to learn the representation of tokens in the tail of vocabulary. Thus, the corpus used in this work is empirically composed of multilingual corpus “CulturaX” [40%] (Nguyen et al., 2024), code corpus “The Stack” [30%] (Kocetkov et al., 2023), and math corpus “Proof-Pile-2” [30%] (Azerbayev et al., 2024). We tokenize the mixed corpus using various tokenizers and obtain multiple sequences of token IDs for the same corpus. The default amount of tokens used in this step is 1B, which is investigated in Appendix B.2.

Step 2: Token Representation Learning We adopt GloVe (Pennington et al., 2014) to learn the representation of tokens from the first step. The main reason is that GloVe considers more global statistical information than those slide window methods like CBOW and FastText (Mikolov et al., 2013a,b; Bojanowski et al., 2017). The details of training settings for GloVe vectors refer to Appendix A.

Step 3: Token Alignment Based on the assumption that token representations capture the semantic information in the token, we align token IDs using the pair-wise cosine similarity of learned token representations. It should be noted that the IDs of tokens belonging to both vocabularies are directly replaced without the need to align. \( M_{s \rightarrow t} \) denotes the learned token-token alignment matrix, which records the pair-wise similarity of each source token and target token. It can serve as the one-to-one mapping function for each source/target token to find the most similar token from the target/source vocabulary.

3.2 Alignment Evaluation

Figure 3(a) illustrates our metrics to evaluate the performance of alignment matrix \( M_{s \rightarrow t} \). We first

![Illustration of TokAlign to align token IDs from different vocabularies. We train token representations on the tokenized corpus, and align token IDs by the cosine similarity. It is noted that the IDs of tokens belonging to both vocabularies are directly replaced without alignment.](page_186_120_1077_393.png)

Figure 2: Illustration of TokAlign to align token IDs from different vocabularies. We train token representations on the tokenized corpus, and align token IDs by the cosine similarity. It is noted that the IDs of tokens belonging to both vocabularies are directly replaced without alignment.
Figure 3: (a) We choose BLEU-1 and BERTScore to evaluate the performance of alignment matrix \( M_{s \rightarrow t} \) (b) Embedding and lm_head are tuned at the first half part of the process, followed by full parameter tuning. * indicates the parameter of each target token is first initialized from the most similar source token by alignment matrix \( M_{s \rightarrow t} \).

tokenize the test corpus \( C \) using different tokenizers, which results in \( C_s \) and \( C_t \). The token ID corpus \( C_s \) from the source tokenizer is converted to its most similar target token ID by alignment matrix \( M_{s \rightarrow t} \), and comes to the corpus \( C'_t \). From the view of token ID matching, the higher BLEU-1 score between \( C'_t \) and the corpus \( C_t \) from the Tokenizer_t, the better alignment matrix \( M_{s \rightarrow t} \) is.

We further propose a semantic evaluation metric: It de-tokenizes the target token ID corpus \( C'_t \) using Tokenizer_t into the recovered text corpus \( C' \), and evaluates the semantic similarity between \( C' \) and original corpus \( C \) using BERTScore. The better alignment matrix \( M_{s \rightarrow t} \) learned preserves more semantics in the test corpus \( C \), bringing higher BERTScore of the recovered \( C' \) and \( C \).

3.3 Progressive Adaptation
Given the alignment matrix \( M_{s \rightarrow t} \), the parameters of each token in the target vocabulary are initialized from the ones of the most similar source token. We find that these re-arranged embeddings and lm_head provide a good initialization for the new model (Section 4.2.1). Figure 3(b) illustrates the two-stage tuning for an LLM to adapt to the new vocabulary. The re-arranged embedding and lm_head are tuned first to avoid loss spike and improve the training stability (Figure 6). The other parameters of internal layers are further tuned together in the last half-part process.

4 Experiments
4.1 Experiments Settings
Large Language Models We adopt the fully open-source language model series Pythia (Biderman et al., 2023) as base models in this work. It is noted that we do not intend to achieve state-of-the-art large language model performance but rather investigate an efficient method to replace the English-centric tokenizer like Pythia. To transfer token-level knowledge from other capable large language models, tokenizers and vocabularies of Gemma (Team et al., 2024), Qwen2 (Yang et al., 2024), LLaMA2 (Touvron et al., 2023b), and LLaMA3 (Meta, 2024) are selected as the target to replace. We report hyper-parameters in Appendix A.

Corpus To reduce the risk of distribution shift from the training data, we choose the vanilla pre-training corpus Pile (Gao et al., 2020) of Pythia in the fine-tuning process. We also investigate the robustness of the corpus used in the vocabulary alignment by replacing it with Slimpajama (Soboleva et al., 2023). Corpora of downstream tasks and multiple languages are applied in cross-lingual and cross-model knowledge transfer experiments (Section 4.2.1 and 4.2.2).

Evaluation Tasks Following the common practices to evaluate large language models (Lin et al., 2022; Biderman et al., 2023; Zhang et al., 2024), there are 10 datasets, including commonsense reasoning (Clark et al., 2018; Mihaylov et al., 2018; Zellers et al., 2019; Ponti et al., 2020; Bisk et al., 2020; Sakaguchi et al., 2020) and reading comprehension (Clark et al., 2019) tasks, used in this work. To avoid the randomness from the prompt and evaluation method, we adopt the default prompt from the commonly used language model evaluation harness framework (Gao et al., 2024). Further information about the evaluation tasks is reported in Appendix C.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="6">High</th>
    <th colspan="4">Medium</th>
    <th colspan="3">Low</th>
    <th rowspan="2">Avg ↓</th>
  </tr>
  <tr>
    <th>ar</th><th>de</th><th>en</th><th>ja</th><th>zh</th><th>hn</th><th>ko</th><th>th</th><th>uk</th><th>vi</th><th>ta</th><th>te</th><th>ur</th>
  </tr>
  <tr>
    <td>Qwen2 sb</td>
    <td>4.7</td><td>11.1</td><td>15.7</td><td>6.0</td><td>4.6</td><td>2.4</td><td>3.3</td><td>2.6</td><td>5.7</td><td>3.3</td><td>2.8</td><td>3.4</td><td>4.0</td><td>5.3</td>
  </tr>
  <tr>
    <td>Pythia_1b</td>
    <td>7.6</td><td>15.4</td><td>21.7</td><td>9.9</td><td>13.2</td><td>3.4</td><td>5.6</td><td>4.3</td><td>6.7</td><td>6.3</td><td>2.9</td><td>3.3</td><td>5.8</td><td>8.2</td>
  </tr>
  <tr>
    <td>w/Focus Init.</td>
    <td>1.1<sup>e-3</sup></td><td>1.7<sup>e-3</sup></td><td>1.8<sup>e-3</sup></td><td>2.1<sup>e-3</sup></td><td>9.6<sup>e-3</sup></td><td>6.5<sup>e-3</sup></td><td>1.0<sup>e-3</sup></td><td>5.6<sup>e-3</sup></td><td>1.0<sup>e-3</sup></td><td>8.4<sup>e-3</sup></td><td>5.0<sup>e-3</sup></td><td>1.0<sup>e-3</sup></td><td>1.9<sup>e-3</sup></td><td>3.1<sup>e-3</sup></td>
  </tr>
  <tr>
    <td>+ LAT</td>
    <td>8.3</td><td>21.7</td><td>59.7</td><td>14.0</td><td>14.0</td><td>3.8</td><td>7.3</td><td>5.9</td><td>8.5</td><td>8.6</td><td>4.3</td><td>4.3</td><td>12.4</td>
  </tr>
  <tr>
    <td>w/ZeTT Init.</td>
    <td>3.0<sup>e-2</sup></td><td>4.2<sup>e-2</sup></td><td>1.3<sup>e-2</sup></td><td>1.2<sup>e-2</sup></td><td>2.4<sup>e-2</sup></td><td>3.0<sup>e-2</sup></td><td>2.4<sup>e-2</sup></td><td>3.3<sup>e-2</sup></td><td>2.0<sup>e-2</sup></td><td>2.4<sup>e-2</sup></td><td>1.8<sup>e-2</sup></td><td>4.7<sup>e-2</sup></td><td>3.4<sup>e-2</sup></td>
  </tr>
  <tr>
    <td>+ LAT</td>
    <td>7.1</td><td>15.7</td><td>26.4</td><td>10.0</td><td>10.3</td><td>2.8</td><td>5.0</td><td>3.6</td><td>5.9</td><td>4.9</td><td>2.6</td><td>2.7</td><td>4.2</td><td>7.8</td>
  </tr>
  <tr>
    <td>w/TokAlign Init.</td>
    <td>1.2<sup>e-2</sup></td><td>2.2<sup>e-2</sup></td><td>1.0<sup>e-2</sup></td><td>3.6<sup>e-2</sup></td><td>1.2<sup>e-2</sup></td><td>46.5</td><td>60.1</td><td>70.8</td><td>1.5<sup>e-2</sup></td><td>49.2</td><td>61.0</td><td>1.1<sup>e-2</sup></td><td>50.9</td><td>1.2<sup>e-2</sup></td>
  </tr>
  <tr>
    <td>+ LAT</td>
    <td>6.3</td><td>13.9</td><td>23.6</td><td>8.9</td><td>9.0</td><td>2.4</td><td>4.4</td><td>3.2</td><td>5.2</td><td>4.4</td><td>2.3</td><td>2.4</td><td>3.7</td><td>6.9</td>
  </tr>
  <tr>
    <td>Qwen2sb</td>
    <td>3.9</td><td>8.1</td><td>11.8</td><td>4.9</td><td>3.8</td><td>2.1</td><td>2.9</td><td>2.3</td><td>3.8</td><td>2.9</td><td>2.3</td><td>2.6</td><td>3.3</td><td>4.2</td>
  </tr>
  <tr>
    <td>Pythia_1b</td>
    <td>5.9</td><td>10.8</td><td>16.7</td><td>7.9</td><td>9.9</td><td>3.0</td><td>4.6</td><td>3.7</td><td>4.9</td><td>4.9</td><td>2.6</td><td>2.9</td><td>4.8</td><td>6.3</td>
  </tr>
  <tr>
    <td>w/Focus Init.</td>
    <td>6.9<sup>e-3</sup></td><td>1.6<sup>e-3</sup></td><td>1.2<sup>e-3</sup></td><td>2.4<sup>e-3</sup></td><td>1.3<sup>e-3</sup></td><td>2.5<sup>e-4</sup></td><td>7.2<sup>e-4</sup></td><td>3.2<sup>e-3</sup></td><td>1.9<sup>e-3</sup></td><td>7.9<sup>e-3</sup></td><td>1.7<sup>e-3</sup></td><td>1.5<sup>e-3</sup></td><td>1.7<sup>e-3</sup></td><td>2.8<sup>e-3</sup></td>
  </tr>
  <tr>
    <td>+ LAT</td>
    <td>6.4</td><td>17.6</td><td>39.3</td><td>10.8</td><td>11.1</td><td>2.5</td><td>10.0</td><td>3.3</td><td>5.2</td><td>4.8</td><td>2.3</td><td>2.7</td><td>4.7</td><td>8.8</td>
  </tr>
  <tr>
    <td>w/TokAlign Init.</td>
    <td>1.2<sup>e-2</sup></td><td>2.1<sup>e-2</sup></td><td>81.4</td><td>7.5<sup>e-2</sup></td><td>1.3<sup>e-2</sup></td><td>57.5</td><td>53.3</td><td>66.2</td><td>4.0<sup>e-2</sup></td><td>49.2</td><td>49.2</td><td>1.9<sup>e-2</sup></td><td>48.7</td><td>1.4<sup>e-2</sup></td>
  </tr>
  <tr>
    <td>+ LAT</td>
    <td>5.2</td><td>9.9</td><td>17.8</td><td>7.4</td><td>7.9</td><td>2.1</td><td>3.8</td><td>2.8</td><td>4.0</td><td>3.7</td><td>2.1</td><td>2.1</td><td>3.1</td><td>5.5</td>
  </tr>
  <tr>
    <td>△ Length (%) ↓</td>
    <td>-44.5</td><td>-13.1</td><td>-0.8</td><td>-32.4</td><td>-50.0</td><td>-22.2</td><td>-52.2</td><td>-46.1</td><td>-15.5</td><td>-51.7</td><td>-20.3</td><td>-2.9</td><td>-28.5</td><td>-29.2</td>
  </tr>
</table>

Table 1: The normalized perplexity on the valid corpus of CulturaX. The perplexity is normalized to the vocabulary of Pythia following Wei et al. (2023). “High”, “Medium”, and “Low” indicates the available amount of linguistic resources. “w/ xxx Init.” denotes the performance of the model after initialization without any tuning steps.

Baselines We introduce the following vocabulary adaptation methods as baseline methods in this work:

• Random Initialization for each token \( t \in \{ \mathcal{V}_t \setminus (\mathcal{V}_t \cap \mathcal{V}_s) \} \) employs the default initialization method of huggingface Transformers and reuses the parameters of token \( t \in \{ \mathcal{V}_t \cap \mathcal{V}_s \} \), which belongs to both vocabularies.

• Random Permutation initializes each token \( t \in \{ \mathcal{V}_t \setminus (\mathcal{V}_t \cap \mathcal{V}_s) \} \) using the parameter of randomly chosen token from the source vocabulary. The parameters of shared tokens are also reused.

• Multivariate initializes each token \( t \in \{ \mathcal{V}_t \setminus (\mathcal{V}_t \cap \mathcal{V}_s) \} \) by sampling from the multivariate Gaussian distribution with the mean and covariance of source embedding \( E_s \).

• Mean use the mean of source embedding \( E_s \) to initialize all tokens \( t \in \{ \mathcal{V}_t \setminus (\mathcal{V}_t \cap \mathcal{V}_s) \} \).

• WECHSEL (Minixhofer et al., 2022) linearly transfers embeddings of source tokens into target tokens by tokenizing and recomposing additional word embeddings \( W^s \) and \( W^t \), which are aligned with a bilingual dictionary.

• OFA (Liu et al., 2024) factorizes the embeddings of source model \( E_s \) into the primitive embedding \( P \) and source coordinate \( F_s \) that is further re-composed by multilingual word embedding \( W \) to the target coordinate \( F_t \). The assembled primitive embedding \( P \) and target coordinate \( F_t \) yield the target embedding \( E_t \).

• Focus (Dobler and de Melo, 2023) initializes the embedding parameters of token \( t \in \{ \mathcal{V}_t \setminus (\mathcal{V}_t \cap \mathcal{V}_s) \} \) using the weighted sum of the ones from the token \( t \in \{ \mathcal{V}_t \cap \mathcal{V}_s \} \). It largely depends on the size of \( \| \mathcal{V}_t \cap \mathcal{V}_s \| \), and performs poorly when the overlapping percentage of \( \mathcal{V}_t \) and \( \mathcal{V}_s \) is low.

• ZeTT (Minixhofer et al., 2024) trains an additional hypernetwork \( H_0 \) to generate the parameters for each token \( t \in \mathcal{V}_t \). The added hypernetwork brings a lot of training costs.

4.2 Main Results

We first report the final results of two applications after replacing vocabulary: cross-lingual transfer (Section 4.2.1) and cross-model knowledge transfer (Section 4.2.2), then show vocabulary adaptation results of methods (Section 4.3).

4.2.1 Cross-lingual Transfer

When applied to new domains or languages, tokenizers with higher compression rates can speed up the learning and inference of large language models. From the view of token co-occurrence, tokens from other languages can be aligned and initialized by the tokens with similar semantics in the source vocabulary, which can boost the cross-lingual knowledge transfer. Therefore, we replace the English-centric tokenizer of Pythia with the one of Qwen2 to evaluate the performance on cross-lingual transfer settings.

As shown in Table 1, the perplexity of Pythia initialized using TokAlign (1.2e\(^{-2}\)) is significantly
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="6">XNLI</th>
    <th colspan="6">PAWS-X</th>
    <th colspan="6">XCOPA</th>
    <th colspan="6">XStoryCloze</th>
    <th rowspan="2">Avg</th>
  </tr>
  <tr>
    <th>en</th><th>de</th><th>zh</th><th>ar</th><th>th</th><th>vi</th>
    <th>en</th><th>ja</th><th>ko</th><th>zh</th><th>th</th><th>vi</th><th>ta</th>
    <th>en</th><th>zh</th><th>ar</th><th>th</th><th>vi</th><th>ta</th>
  </tr>
  <tr>
    <td>Pythia<sub>1B</sub></td>
    <td>51.0</td><td>37.8</td><td>42.6</td><td>35.9</td><td>34.8</td><td>37.0</td>
    <td>49.6</td><td>49.3</td><td>54.8</td><td>54.9</td><td>52.9</td><td>54.0</td><td>53.2</td><td>55.4</td>
    <td>64.3</td><td>48.6</td><td>48.0</td><td>52.9</td><td>48.0</td>
  </tr>
  <tr>
    <td>+ Focus Init.</td>
    <td>52.8</td><td>32.2</td><td>33.6</td><td>33.6</td><td>32.5</td><td>32.0</td>
    <td>48.8</td><td>44.9</td><td>45.7</td><td>44.8</td><td>44.1</td><td>52.4</td><td>48.6</td><td>57.0</td>
    <td>45.9</td><td>47.8</td><td>48.5</td><td>46.5</td><td>42.2</td>
  </tr>
  <tr>
    <td>+ LAT</td>
    <td>46.0</td><td>35.1</td><td>34.9</td><td>32.9</td><td>32.5</td><td>34.4</td>
    <td>50.6</td><td>45.5</td><td>55.9</td><td>53.4</td><td>55.3</td><td>53.8</td><td>52.6</td><td>55.4</td>
    <td>55.8</td><td>48.8</td><td>47.6</td><td>50.4</td><td>46.1</td>
  </tr>
  <tr>
    <td>w/ZeTT Init.</td>
    <td>45.9</td><td>34.6</td><td>32.9</td><td>32.8</td><td>33.5</td><td>33.6</td>
    <td>51.5</td><td>50.3</td><td>54.8</td><td>51.5</td><td>53.5</td><td>52.6</td><td>48.2</td><td>55.6</td>
    <td>53.2</td><td>46.9</td><td>46.9</td><td>48.1</td><td>45.3</td>
  </tr>
  <tr>
    <td>+ LAT</td>
    <td>48.6</td><td>38.6</td><td>40.6</td><td>36.9</td><td>36.0</td><td>39.3</td>
    <td>53.0</td><td>51.0</td><td>55.8</td><td>54.8</td><td>55.3</td><td>55.8</td><td>50.8</td><td>54.0</td>
    <td>60.3</td><td>49.3</td><td>47.2</td><td>52.1</td><td>48.1</td>
  </tr>
  <tr>
    <td>w/TokAlign Init.</td>
    <td>49.9</td><td>36.6</td><td>33.2</td><td>31.8</td><td>33.2</td><td>34.1</td>
    <td>54.1</td><td>52.1</td><td>56.1</td><td>54.7</td><td>55.5</td><td>53.6</td><td>48.0</td><td>55.2</td>
    <td>61.0</td><td>47.6</td><td>47.1</td><td>51.0</td><td>46.7</td>
  </tr>
  <tr>
    <td>+ LAT</td>
    <td>50.0</td><td>39.2</td><td>42.7</td><td>37.4</td><td>37.4</td><td>40.3</td>
    <td>55.7</td><td>54.6</td><td>50.2</td><td>55.9</td><td>54.9</td><td>55.3</td><td>55.2</td><td>53.0</td>
    <td>64.0</td><td>51.1</td><td>47.8</td><td>53.5</td><td>49.1</td>
  </tr>
  <tr>
    <td>Pythia<sub>6B</sub></td>
    <td>54.4</td><td>39.0</td><td>46.2</td><td>39.3</td><td>39.8</td><td>39.3</td>
    <td>36.4</td><td>43.8</td><td>40.2</td><td>50.2</td><td>54.2</td><td>50.2</td>
    <td>56.2</td><td>54.4</td><td>52.2</td>
    <td>70.4</td><td>53.9</td><td>50.3</td><td>53.8</td><td>48.6</td>
  </tr>
  <tr>
    <td>+ Focus Init.</td>
    <td>51.5</td><td>31.3</td><td>33.0</td><td>32.6</td><td>33.4</td><td>32.2</td>
    <td>32.6</td><td>44.8</td><td>42.4</td><td>52.7</td><td>45.5</td><td>44.7</td>
    <td>52.2</td><td>48.6</td><td>55.6</td>
    <td>44.5</td><td>47.1</td><td>47.8</td><td>47.1</td><td>42.1</td>
  </tr>
  <tr>
    <td>+ LAT</td>
    <td>52.6</td><td>34.9</td><td>36.6</td><td>35.1</td><td>33.6</td><td>39.0</td>
    <td>34.5</td><td>51.1</td><td>43.8</td><td>55.9</td><td>55.3</td><td>55.4</td>
    <td>54.2</td><td>52.4</td><td>53.8</td>
    <td>61.0</td><td>48.7</td><td>47.7</td><td>53.7</td><td>47.3</td>
  </tr>
  <tr>
    <td>w/TokAlign Init.</td>
    <td>53.3</td><td>36.3</td><td>35.0</td><td>34.6</td><td>34.6</td><td>33.0</td>
    <td>33.8</td><td>48.8</td><td>44.6</td><td>56.2</td><td>55.7</td><td>55.3</td>
    <td>54.6</td><td>52.2</td><td>54.6</td>
    <td>66.8</td><td>48.6</td><td>47.7</td><td>50.0</td><td>47.1</td>
  </tr>
  <tr>
    <td>+ LAT</td>
    <td>55.2</td><td>35.8</td><td>43.5</td><td>40.4</td><td>40.2</td><td>43.0</td>
    <td>37.1</td><td>43.2</td><td>45.8</td><td>50.8</td><td>55.8</td><td>55.5</td>
    <td>54.0</td><td>57.0</td><td>54.0</td>
    <td>70.2</td><td>54.4</td><td>49.3</td><td>53.9</td><td>49.7</td>
  </tr>
</table>

Table 2: Zero-shot in-context learning results of cross-lingual transfer. Refer to Table 8 for few-shot results.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="2">ARC-E</th>
    <th colspan="2">BoolQ</th>
    <th colspan="2">HellaSwag</th>
    <th colspan="2">OpenbookQA</th>
    <th colspan="2">PIQA</th>
    <th colspan="2">Winogrande</th>
    <th rowspan="2">Avg</th>
  </tr>
  <tr>
    <th>0</th><th>5</th>
    <th>0</th><th>5</th>
    <th>0</th><th>5</th>
    <th>0</th><th>5</th>
    <th>0</th><th>5</th>
    <th>0</th><th>5</th>
  </tr>
  <tr>
    <td>Pythia<sub>1B</sub></td>
    <td>56.52</td><td>58.71</td>
    <td>60.43</td><td>57.37</td>
    <td>37.68</td><td>37.66</td>
    <td>18.80</td><td>19.00</td>
    <td>70.40</td><td>71.49</td>
    <td>53.20</td><td>52.01</td>
    <td>49.55</td><td>49.37</td>
  </tr>
  <tr>
    <td>+ Direct tuning</td>
    <td>57.19</td><td>55.64</td>
    <td>70.70</td><td>72.11</td>
    <td>41.24</td><td>41.60</td>
    <td>35.40</td><td>38.40</td>
    <td>69.04</td><td>70.08</td>
    <td>54.70</td><td>54.78</td>
    <td>53.70</td><td>53.77</td>
  </tr>
  <tr>
    <td>+ Sentence distill</td>
    <td>52.27</td><td>53.41</td>
    <td>67.49</td><td>67.06</td>
    <td>39.03</td><td>39.08</td>
    <td>21.80</td><td>22.80</td>
    <td>66.97</td><td>68.91</td>
    <td>51.85</td><td>52.17</td>
    <td>49.90</td><td>50.58</td>
  </tr>
  <tr>
    <td>w/Gemina<sub>2B</sub></td>
    <td>55.39</td><td>56.99</td>
    <td>67.18</td><td>69.69</td>
    <td>38.53</td><td>37.26</td>
    <td>19.00</td><td>20.80</td>
    <td>68.82</td><td>69.21</td>
    <td>52.33</td><td>53.51</td>
    <td>49.88</td><td>51.58</td>
  </tr>
  <tr>
    <td>w/Qwen<sub>2B</sub></td>
    <td>62.06</td><td>64.07</td>
    <td>70.18</td><td>71.38</td>
    <td>41.21</td><td>41.24</td>
    <td>23.20</td><td>23.80</td>
    <td>73.76</td><td>74.71</td>
    <td>55.13</td><td>55.41</td>
    <td>52.41</td><td>52.83</td>
  </tr>
  <tr>
    <td>w/LLMA<sub>3B</sub></td>
    <td>64.02</td><td>64.56</td>
    <td>73.91</td><td>74.19</td>
    <td>42.11</td><td>42.44</td>
    <td>24.20</td><td>27.60</td>
    <td>73.74</td><td>73.83</td>
    <td>55.49</td><td>56.43</td>
    <td>55.41</td><td>56.49</td>
  </tr>
  <tr>
    <td>Pythia<sub>6B</sub></td>
    <td>65.00</td><td>69.23</td>
    <td>64.82</td><td>62.02</td>
    <td>47.56</td><td>47.64</td>
    <td>25.00</td><td>27.00</td>
    <td>74.65</td><td>75.41</td>
    <td>60.46</td><td>62.43</td>
    <td>56.08</td><td>57.29</td>
  </tr>
  <tr>
    <td>+ Direct tuning</td>
    <td>66.25</td><td>66.20</td>
    <td>79.30</td><td>78.87</td>
    <td>52.21</td><td>53.00</td>
    <td>33.20</td><td>33.00</td>
    <td>72.74</td><td>74.48</td>
    <td>62.90</td><td>61.72</td>
    <td>61.13</td><td>61.28</td>
  </tr>
  <tr>
    <td>+ Sentence distill</td>
    <td>61.70</td><td>65.36</td>
    <td>76.64</td><td>76.88</td>
    <td>48.51</td><td>51.33</td>
    <td>28.20</td><td>30.40</td>
    <td>70.18</td><td>71.55</td>
    <td>58.96</td><td>62.19</td>
    <td>57.44</td><td>59.62</td>
  </tr>
  <tr>
    <td>w/Gemina<sub>2B</sub></td>
    <td>67.59</td><td>68.94</td>
    <td>76.06</td><td>75.68</td>
    <td>47.83</td><td>48.36</td>
    <td>28.40</td><td>31.40</td>
    <td>73.78</td><td>75.52</td>
    <td>59.04</td><td>61.47</td>
    <td>58.78</td><td>60.67</td>
  </tr>
  <tr>
    <td>w/Qwen<sub>2B</sub></td>
    <td>71.72</td><td>73.77</td>
    <td>79.85</td><td>80.00</td>
    <td>50.78</td><td>51.12</td>
    <td>29.20</td><td>34.00</td>
    <td>77.26</td><td>77.91</td>
    <td>61.33</td><td>64.56</td>
    <td>61.69</td><td>63.48</td>
  </tr>
  <tr>
    <td>w/LLMA<sub>3B</sub></td>
    <td>67.05</td><td>69.78</td>
    <td>77.83</td><td>78.78</td>
    <td>48.83</td><td>50.15</td>
    <td>26.00</td><td>32.00</td>
    <td>74.21</td><td>76.22</td>
    <td>60.22</td><td>60.93</td>
    <td>59.02</td><td>61.31</td>
  </tr>
</table>

Table 3: The main results of token-level distillation on six downstream tasks with only 235M tokens. “+Sentence distill” denotes the sentence-level distillation results with Qwen2<sub>7B</sub>(Yang et al., 2024), which fine-tunes on the output from Qwen2<sub>7B</sub> given questions as prompt.

better than other two strong baseline methods Focus (2.9e<sup>5</sup>) and ZeTT(3.4e<sup>2</sup>). The length of tokens after text tokenization has reduced by 29.2% on average across these languages. After only 2k steps of Language Adaptation Tuning (“+LAT”), TokAlign improved 14.5% over the vanilla model on average, while Focus still performed worse. It is noted that the performance of Pythia using TokAlign on three low-resource languages even outperforms the ones of Qwen2 with a similar parameter amount.

Table 2 and 8 in Appendix B.5 further report zero-shot and few-shot in-context learning results on four multilingual datasets. We can find that TokAlign brings a better-initialized model than the baseline method Focus (+4.4%), and transfers the knowledge into other languages like Japanese (ja, +2.3%) and Vietnamese (vi, +2.2%).

It is interesting to find that the perplexity of Pythia<sub>1B</sub> initialized by TokAlign reaches 1.2e<sup>2</sup>, while the in-context learning results are comparable with the ones of Focus after adapting on the multilingual corpus. We argue that it arises from the reserved English ability with TokAlign (54.2%), which significantly outperforms Focus (40.8%).

4.2.2 Cross-model Transfer

Unifying vocabulary with capable LLMs enables token-level distillation and transfers the knowledge learned into smaller models to decrease inference costs. In this section, training samples from downstream tasks and the corpus of Pile are used in the token-level distillation experiments. The logit of each token from the teacher model is taken as the soft label for Pythia to learn. Specifically, we add the KL-divergence loss between the logit from the teacher and student models to the original next token prediction loss on the training samples. The proportion of training samples is empirically set to 15% to avoid a significant degradation in language modeling performance (Wei et al., 2023). There are two baseline methods: “+ Direct tuning”, where models directly fine-tune on the training samples, and “+ Sentence distill” for comparison, where models fine-tune on the output text from the teacher model given the question as a prompt.

Table 3 reports the results of two baseline methods and token-level distillation from three teacher models using 235M tokens. It can be found that token-level distillation is significantly better than sentence-level distillation. In the neural machine
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">#GPU Hour</th>
    <th colspan="5">ARC-E</th>
    <th colspan="5">BoolQ</th>
    <th colspan="5">HellaSwag</th>
    <th colspan="5">OpenbookQA</th>
    <th colspan="5">PIQA</th>
    <th colspan="5">WinoGrande</th>
    <th colspan="5">Avg</th>
  </tr>
  <tr>
    <th>0</th><th>5</th><th>0</th><th>5</th><th>0</th><th>5</th><th>0</th><th>5</th><th>0</th><th>5</th><th>0</th><th>5</th><th>0</th><th>5</th><th>0</th><th>5</th><th>0</th><th>5</th><th>0</th><th>5</th><th>0</th><th>5</th><th>0</th><th>5</th><th>0</th><th>5</th><th>0</th><th>5</th><th>0</th><th>5</th>
  </tr>
  <tr>
    <td>Pythia1B</td><td>—</td><td>—</td><td>56.82</td><td>58.71</td><td>60.43</td><td>57.37</td><td>37.68</td><td>37.66</td><td>18.80</td><td>19.00</td><td>70.40</td><td>71.49</td><td>53.20</td><td>52.01</td><td>49.55</td><td>49.37</td>
  </tr>
  <tr>
    <td>w/ Rand. Init.</td><td>99.70</td><td>31.36</td><td>31.61</td><td>37.83</td><td>49.11</td><td>26.35</td><td>26.40</td><td>14.00</td><td>12.60</td><td>54.57</td><td>55.33</td><td>49.17</td><td>49.17</td><td>35.55</td><td>37.37</td>
  </tr>
  <tr>
    <td>w/ Rand. Perm.</td><td>99.70</td><td>31.69</td><td>32.95</td><td>37.77</td><td>54.80</td><td>26.43</td><td>26.39</td><td>14.00</td><td>12.60</td><td>55.50</td><td>55.98</td><td>47.04</td><td>50.07</td><td>35.40</td><td>38.90</td>
  </tr>
  <tr>
    <td>w/ Multivariate</td><td>99.70</td><td>32.79</td><td>34.18</td><td>45.08</td><td>49.72</td><td>27.67</td><td>27.87</td><td>15.20</td><td>16.20</td><td>56.09</td><td>57.83</td><td>50.51</td><td>50.12</td><td>37.89</td><td>39.32</td>
  </tr>
  <tr>
    <td>w/ Mean</td><td>99.70</td><td>44.87</td><td>46.97</td><td>53.39</td><td>55.20</td><td>31.59</td><td>31.67</td><td>16.20</td><td>17.00</td><td>61.32</td><td>62.46</td><td>49.25</td><td>51.85</td><td>42.77</td><td>44.19</td>
  </tr>
  <tr>
    <td>w/ OFA</td><td>99.70</td><td>38.17</td><td>37.79</td><td>55.14</td><td>52.35</td><td>28.29</td><td>28.62</td><td>14.40</td><td>12.20</td><td>58.43</td><td>58.54</td><td>49.96</td><td>50.99</td><td>40.72</td><td>40.08</td>
  </tr>
  <tr>
    <td>w/ WECHSEL</td><td>99.70</td><td>43.45</td><td>43.35</td><td>56.61</td><td>54.14</td><td>32.27</td><td>32.46</td><td>19.20</td><td>18.00</td><td>63.82</td><td>63.80</td><td>52.89</td><td>52.89</td><td>41.96</td><td>43.98</td>
  </tr>
  <tr>
    <td>w/ Focus</td><td>99.70</td><td>41.55</td><td>48.37</td><td>56.21</td><td>57.78</td><td>32.27</td><td>32.46</td><td>19.20</td><td>18.00</td><td>63.82</td><td>63.80</td><td>52.89</td><td>52.89</td><td>41.96</td><td>43.98</td>
  </tr>
  <tr>
    <td>w/ ZeTT</td><td>418.94</td><td>47.14</td><td>49.03</td><td>57.06</td><td>53.70</td><td>34.06</td><td>34.06</td><td>18.40</td><td>19.40</td><td>64.15</td><td>65.34</td><td>52.09</td><td>51.22</td><td>45.48</td><td>45.46</td>
  </tr>
  <tr>
    <td>w/ TokAlign</td><td>99.70</td><td>54.46<sup>*</sup></td><td>56.86<sup>*</sup></td><td>58.90<sup>*</sup></td><td>52.26</td><td>36.16<sup>*</sup></td><td>36.27<sup>*</sup></td><td>21.00<sup>*</sup></td><td>20.20<sup>*</sup></td><td>67.74<sup>*</sup></td><td>68.50<sup>*</sup></td><td>52.25<sup>*</sup></td><td>50.91</td><td>48.42</td><td>47.50</td>
  </tr>
  <tr>
    <td>w/ SlimPajama</td><td>99.70</td><td>53.54</td><td>55.68</td><td>57.55</td><td>53.85</td><td>36.10</td><td>35.99</td><td>19.40</td><td>20.20</td><td>67.03</td><td>67.52</td><td>52.09</td><td>51.22</td><td>47.62</td><td>47.41</td>
  </tr>
  <tr>
    <td>+ Align Rep.</td><td>99.70</td><td>54.25</td><td>56.65</td><td>59.33</td><td>54.68</td><td>37.08</td><td>36.91</td><td>20.20</td><td>19.40</td><td>67.36</td><td>68.17</td><td>54.38</td><td>52.80</td><td>48.77</td><td>48.10</td>
  </tr>
  <tr>
    <td>Pythia2SB</td><td>—</td><td>—</td><td>63.80</td><td>67.00</td><td>63.91</td><td>65.14</td><td>45.32</td><td>45.04</td><td>24.00</td><td>25.20</td><td>74.05</td><td>74.43</td><td>58.64</td><td>60.77</td><td>54.95</td><td>56.26</td>
  </tr>
  <tr>
    <td>w/ Rand. Init.</td><td>194.78</td><td>30.47</td><td>32.91</td><td>38.20</td><td>51.07</td><td>26.46</td><td>26.69</td><td>14.40</td><td>14.20</td><td>55.17</td><td>55.06</td><td>48.30</td><td>50.51</td><td>35.50</td><td>38.24</td>
  </tr>
  <tr>
    <td>w/ Rand. Perm.</td><td>194.78</td><td>31.48</td><td>31.86</td><td>37.83</td><td>50.46</td><td>26.48</td><td>26.49</td><td>13.60</td><td>14.40</td><td>54.03</td><td>54.95</td><td>50.20</td><td>48.86</td><td>35.60</td><td>37.84</td>
  </tr>
  <tr>
    <td>w/ OFA</td><td>194.78</td><td>50.13</td><td>54.12</td><td>60.89</td><td>61.47</td><td>36.39</td><td>36.88</td><td>18.00</td><td>19.00</td><td>65.18</td><td>64.80</td><td>54.06</td><td>54.85</td><td>47.44</td><td>48.52</td>
  </tr>
  <tr>
    <td>w/ WECHSEL</td><td>194.78</td><td>52.48</td><td>54.92</td><td>59.42</td><td>56.76</td><td>36.79</td><td>37.30</td><td>19.20</td><td>20.80</td><td>64.04</td><td>64.25</td><td>54.53</td><td>55.72</td><td>48.06</td><td>48.29</td>
  </tr>
  <tr>
    <td>w/ Focus</td><td>194.78</td><td>54.06</td><td>58.14</td><td>62.38</td><td>58.38</td><td>39.09</td><td>39.09</td><td>20.00</td><td>20.20</td><td>68.44</td><td>68.28</td><td>56.02</td><td>56.04</td><td>49.53</td><td>50.77</td>
  </tr>
  <tr>
    <td>w/ ZeTT</td><td>194.78</td><td>61.07</td><td>61.42</td><td>61.68</td><td>60.05</td><td>41.38</td><td>41.38</td><td>21.00</td><td>20.20</td><td>71.11</td><td>71.16</td><td>56.16</td><td>56.16</td><td>52.96</td><td>52.96</td>
  </tr>
  <tr>
    <td>w/ TokAlign</td><td>194.78</td><td>61.62<sup>*</sup></td><td>65.15<sup>*</sup></td><td>63.82</td><td>65.47</td><td>43.13</td><td>43.18<sup>*</sup></td><td>23.40<sup>*</sup></td><td>25.80<sup>*</sup></td><td>72.14<sup>*</sup></td><td>72.42<sup>*</sup></td><td>58.17<sup>*</sup></td><td>61.17<sup>*</sup></td><td>53.71</td><td>55.53</td>
  </tr>
  <tr>
    <td>+ Align Rep.</td><td>194.78</td><td>61.66</td><td>65.66</td><td>64.56</td><td>65.66</td><td>43.97</td><td>44.09</td><td>22.40</td><td>25.00</td><td>73.01</td><td>73.23</td><td>58.09</td><td>60.54</td><td>53.95</td><td>55.70</td>
  </tr>
</table>

Table 4: The main results of replacing the vocabulary of Pythia to Gemma. The best performance among the eight methods is displayed in bold. * indicates statistically significant improvements of 5% level. “+Align Rep.” denotes the GloVe embeddings for tokens are converted into relative representations using 300 common tokens in both vocabularies before alignment following (Moschella et al., 2023).

translation domain, token-level distillation outperforms sentence-level distillation when using larger student models, simpler texts, and abundant decoding information (Kim and Rush, 2016; Wei et al., 2024). Given the same teacher model Qwen2SB, the improvement of Pythia over the sentence-level distillation result reaches 4.4%. The performance of Pythia1B is even comparable with the vanilla Pythia1B after token-level distillation. It is also noted that the knowledge transfer between models will be constrained in sentence-level distilling without unifying vocabulary, which further demonstrates the importance of unifying tokenizers between models.

4.3 Vocabulary Adaptation Results

We show experimental results of replacing the Pythia vocabulary (50.3k) with the Gemma vocabulary (256.0k) using all methods in Table 4. Given the same amount of tokens to fine-tune, it can be found that TokenAlign performs better than other baseline methods. The average improvement of TokenAlign over the strong baseline method ZeTT reaches 2.4%, and 97.6% performance of the vanilla model is reserved after vocabulary replacement. ZeTT requires more computation to train a hypernetwork for the parameters prediction, e.g., 661.2 GPU hours for Pythia2SB, while our method only costs less than two hours on a CPU server with 128 cores to train GloVe embeddings and align tokens. Replace the corpus to train the GloVe embedding with 1B SlimPajama (Soboleva et al., 2023) tokens brings comparable results (the “w/ SlimPajama” row). It demonstrates the robustness of our method on the pre-training corpus for token embedding and alignment matrix. Following Moschella et al. (2023), we also evaluate the method that converts token representations into relative ones using 300 common tokens in both vocabularies as anchors before calculating the alignment matrix \( M_{s \rightarrow t} \), which brings better performance.

4.4 Analysis

The loss curves of Pythia2SB with different methods during the first 2.5k steps are shown in Figure 4. We find that TokAlign brings a better initialization and decreases the first-step training loss from 17.8 (Focus) to 9.5. Moreover, the training process with TokAlign is faster than other methods, which reaches 2.75 at the 1.3k step and is 1.92x (2.5/1.3) speed up than Focus.

![The training loss of Pythia2SB.](page_1012_1442_324_324.png)

Figure 4: The training loss of Pythia2SB.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="3">ARC-E</th>
    <th colspan="3">BoolQ</th>
    <th colspan="3">HellaSwag</th>
    <th colspan="3">OpenbookQA</th>
    <th colspan="3">PIQA</th>
    <th colspan="3">WinoGrande</th>
    <th colspan="3">Avg</th>
  </tr>
  <tr>
    <th>#V (k)</th>
    <th>0</th>
    <th>5</th>
    <th>0</th>
    <th>5</th>
    <th>0</th>
    <th>5</th>
    <th>0</th>
    <th>5</th>
    <th>0</th>
    <th>5</th>
    <th>0</th>
    <th>5</th>
    <th>0</th>
    <th>5</th>
    <th>0</th>
    <th>5</th>
    <th>0</th>
    <th>5</th>
    <th>0</th>
    <th>5</th>
  </tr>
  <tr>
    <td>Pythia<sub>1B</sub></td>
    <td>50.3</td>
    <td>56.82</td>
    <td>58.71</td>
    <td>60.43</td>
    <td>57.37</td>
    <td>37.68</td>
    <td>37.66</td>
    <td>18.80</td>
    <td>19.00</td>
    <td>70.40</td>
    <td>71.49</td>
    <td>53.20</td>
    <td>52.01</td>
    <td>49.55</td>
    <td>49.37</td>
  </tr>
  <tr>
    <td>Gemma</td>
    <td>256.0</td>
    <td>54.46</td>
    <td>56.86</td>
    <td><b>58.90</b></td>
    <td>52.26</td>
    <td>36.16</td>
    <td>36.27</td>
    <td>21.00</td>
    <td>20.20</td>
    <td>67.74</td>
    <td>68.50</td>
    <td>52.25</td>
    <td>50.91</td>
    <td>48.42</td>
    <td>47.50</td>
  </tr>
  <tr>
    <td>Qwen2</td>
    <td>152.1</td>
    <td>54.46</td>
    <td>57.07</td>
    <td>54.80</td>
    <td>49.79</td>
    <td>37.18</td>
    <td>37.04</td>
    <td>19.20</td>
    <td>18.40</td>
    <td>68.44</td>
    <td><b>70.24</b></td>
    <td>53.35</td>
    <td>52.80</td>
    <td>47.91</td>
    <td>47.56</td>
  </tr>
  <tr>
    <td>LLaMA2</td>
    <td>32.0</td>
    <td>49.45</td>
    <td>52.02</td>
    <td>58.32</td>
    <td>55.75</td>
    <td>35.38</td>
    <td>35.45</td>
    <td>18.80</td>
    <td>17.80</td>
    <td>66.32</td>
    <td>66.65</td>
    <td>53.91</td>
    <td>50.91</td>
    <td>47.03</td>
    <td>46.43</td>
  </tr>
  <tr>
    <td>LLaMA3</td>
    <td>128.0</td>
    <td><b>54.63</b></td>
    <td><b>57.28</b></td>
    <td>55.84</td>
    <td><b>53.70</b></td>
    <td>37.34</td>
    <td>37.43</td>
    <td>20.20</td>
    <td>20.40</td>
    <td>69.04</td>
    <td>70.18</td>
    <td><b>54.46</b></td>
    <td><b>53.43</b></td>
    <td><b>48.59</b></td>
    <td><b>48.74</b></td>
  </tr>
  <tr>
    <td>Pythia<sub>9B</sub></td>
    <td>50.3</td>
    <td>63.80</td>
    <td>67.00</td>
    <td>63.91</td>
    <td>65.14</td>
    <td>45.32</td>
    <td>45.04</td>
    <td>24.00</td>
    <td>25.20</td>
    <td>74.05</td>
    <td>74.43</td>
    <td>58.64</td>
    <td>60.77</td>
    <td>54.95</td>
    <td>56.26</td>
  </tr>
  <tr>
    <td>Gemma</td>
    <td>256.0</td>
    <td>61.62</td>
    <td>65.15</td>
    <td>63.82</td>
    <td>65.47</td>
    <td>43.18</td>
    <td>43.18</td>
    <td>23.40</td>
    <td><b>25.80</b></td>
    <td>72.14</td>
    <td>72.42</td>
    <td>58.17</td>
    <td><b>61.17</b></td>
    <td><b>55.71</b></td>
    <td><b>55.53</b></td>
  </tr>
  <tr>
    <td>Qwen2</td>
    <td>152.1</td>
    <td><b>62.54</b></td>
    <td><b>66.04</b></td>
    <td>62.35</td>
    <td>63.55</td>
    <td>44.46</td>
    <td>44.39</td>
    <td>23.20</td>
    <td>24.60</td>
    <td>73.50</td>
    <td>73.56</td>
    <td>59.04</td>
    <td>59.59</td>
    <td>54.18</td>
    <td>55.29</td>
  </tr>
  <tr>
    <td>LLaMA3</td>
    <td>128.0</td>
    <td>61.83</td>
    <td>64.60</td>
    <td><b>64.40</b></td>
    <td>63.94</td>
    <td><b>44.62</b></td>
    <td><b>44.59</b></td>
    <td><b>23.80</b></td>
    <td>25.60</td>
    <td>73.45</td>
    <td>73.29</td>
    <td>57.54</td>
    <td>58.72</td>
    <td><b>54.27</b></td>
    <td>55.12</td>
  </tr>
  <tr>
    <td>Pythia<sub>10B</sub></td>
    <td>50.3</td>
    <td>65.99</td>
    <td>69.23</td>
    <td>62.84</td>
    <td>62.02</td>
    <td>47.56</td>
    <td>47.64</td>
    <td>25.00</td>
    <td>27.00</td>
    <td>74.65</td>
    <td>75.41</td>
    <td>60.46</td>
    <td>62.43</td>
    <td>56.08</td>
    <td>57.29</td>
  </tr>
  <tr>
    <td>Gemma</td>
    <td>256.0</td>
    <td>65.40</td>
    <td>68.35</td>
    <td>62.36</td>
    <td>59.57</td>
    <td>45.75</td>
    <td>45.86</td>
    <td>22.00</td>
    <td>25.60</td>
    <td>73.39</td>
    <td>74.10</td>
    <td>60.38</td>
    <td>61.17</td>
    <td>54.89</td>
    <td>55.77</td>
  </tr>
  <tr>
    <td>Qwen2</td>
    <td>152.1</td>
    <td>65.57</td>
    <td><b>68.43</b></td>
    <td><b>64.07</b></td>
    <td>57.61</td>
    <td>46.84</td>
    <td>46.91</td>
    <td>25.60</td>
    <td>25.40</td>
    <td>73.45</td>
    <td>74.65</td>
    <td>61.17</td>
    <td>63.14</td>
    <td>56.12</td>
    <td>56.02</td>
  </tr>
  <tr>
    <td>LLaMA3</td>
    <td>128.0</td>
    <td><b>66.46</b></td>
    <td>68.35</td>
    <td>63.79</td>
    <td>60.64</td>
    <td>47.28</td>
    <td><b>47.31</b></td>
    <td><b>25.60</b></td>
    <td><b>28.20</b></td>
    <td><b>74.48</b></td>
    <td><b>75.84</b></td>
    <td><b>61.48</b></td>
    <td><b>63.30</b></td>
    <td><b>56.52</b></td>
    <td><b>57.27</b></td>
  </tr>
</table>

Table 5: The benchmark results of replacing different tokenizers using TokAlign. The overlapping ratio between the vocabulary of Pythia and other models are 6.23% (Gemma), 26.92% (Qwen2), 28.10% (LLaMA2), 32.85% (LLaMA3).

![BLEU-1(C_t, C'_t) vs Initial Training Loss](page_101_1012_234_180.png)
(a) BLEU-1(C_t, C'_t)

![BERTScore(C, C') vs Initial Training Loss](page_335_1012_234_180.png)
(b) BERTScore(C, C')

Figure 5: The relationship between initial training loss and BLEU-1 (a) or BERTScore (b) for Pythia<sub>1B</sub>.

Better alignment brings better initialization. We further investigate the impact of the learned alignment matrix \( M_{s \rightarrow t} \) by changing the hyper-parameters of GloVe. It is noted that different alignment matrices \( M_{s \rightarrow t} \) bring different initial parameters, and also result in different BLEU-1 scores on the same evaluation corpus. Figure 5(a) illustrates the negative relationship between the first-step training loss and BLEU-1. The sentence embedding model named “all-mnmt-base-v2” (Song et al., 2020) is adopted in the BERTScore evaluation. As shown in Figure 5(b), it also shows a clear negative relationship with the initial training loss. In other words, the higher the BLEU-1 score or BERTScore for the alignment matrix \( M_{s \rightarrow t} \), the better the initial parameter is.

More overlapping comes to faster convergence and higher performance. TokAlign is further applied to the other three target tokenizers: Qwen2, LLaMA2, and LLaMA3. Table 5 reports the performance of models after replacing vocabulary on six datasets. TokAlign recovers 98.0% performance of the base model on average with only 5k steps. Given a target vocabulary with more tokens than the one of Pythia (50.3k), it can be found that a higher overlapping ratio brings a better performance of model replaced (97.6% for Gemma to 99.1% for LLaMA3). The zero-shot in-context learning results for Pythia<sub>9B</sub> with LLaMA3 vocabulary even surpass the vanilla base model. The results of Pythia<sub>1B</sub> with LLaMA2 vocabulary are only 94.5%, which is inferior to the average result. We argue that it may come from the missing 75.0M parameters (7.4% for Pythia<sub>1B</sub>) after switching to a 32.0k vocabulary from the 50.3k vocabulary.

Figure 9 in Appendix B.3 shows the training loss curve. The replacing process of the Gemma tokenizer is the slowest, which may come from the only 6.23% overlapping ratio between two vocabularies. It is in line with the result of random initialization in Figure 10. Appendix B.3 reports more quantitative results by shuffling the alignment matrix, which further demonstrates the importance of token alignment.

Two-stage tuning brings a more stable convergence. To replace the tokenizer and keep the performance of the vanilla model, we only fine-tune the vocabulary-related parameters at the first stage. The main reason for two-stage tuning is to take these parameters as the adapters of different tokenizers and avoid the well-trained parameters of the internal layer being distracted by the new initialized parameters.

Figure 6 illustrates that our two-stage tuning method makes the convergence more stable under a high learning rate like 6.4e^{-4}, which comes to better performance after vocabulary adaptation. It is noted that the loss spike also occurs at the first stage, fine-tuning vocabulary-related parameters only, under such a high learning rate like 2.56e^{-3} in Figure 7.
(a) Learning rate 2e^5        (b) Learning rate 8e^5        (c) Learning rate 6.4e^4

Figure 6: The loss curve of Pythia1B under two-stage tuning or direct full parameters tuning.

Figure 7: The training loss curve of Pythia1B for learning rate used during replacing to the Gemma tokenizer.

5 Conclusion and Future Work

In this paper, we introduce a method named TokAlign to replace the tokenizer of large language models from a token-token co-occurrence view. Extensive experiments demonstrate that TokAlign restores the performance of vanilla models after vocabulary adaptation, which enables cross-lingual knowledge transfer and deep knowledge transfer between models like token-level distillation.

Beyond replacing the vocabulary of large language models, our method can be extended to replace the vocabulary of multi-modal models by aligning different modal tokens. The other direction is to develop a faster method, e.g., incorporating meta-learning in the two-stage tuning method to speed up the convergence.

Limitations

The first limitation comes from the assumption that the pre-training data distribution is available. We conduct experiments on Pythia with different parameter amounts, which provide public model weights and pre-training corpus. Due to the limited computation resource budget, open-source language models with unknown pre-training corpus like Mistral (Jiang et al., 2023) are not investigated in this work. However, the pre-training corpus distribution of open-weighted large language models can be roughly inferred by the BPE vocabulary (Hayase et al., 2024). It can re-construct a similar pre-training corpus to conduct replacing tokenizer experiments.

Another limitation is the additional 5k steps for vocabulary adaptation to replace a tokenizer. From the loss curve of TokAlign (Figure 9), we find that the start of full parameters tuning can be faster, which may result in a better balance between performance and computational budget. Appendix B.4 reports a preliminary result with only 2k steps, where TokAlign also shows a promising result.

Acknowledgements

We would like to thank the anonymous reviewers for their helpful discussions and valuable comments. The research work was supported by the National Key R&D Program of China (No. 2022ZD0160602) and the Strategic Priority Research Program of Chinese Academy of Sciences (No. XDA04080400).

References

Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen Marcus McAleer, Albert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck. 2024. Llemma: An open language model for mathematics. In The Twelfth International Conference on Learning Representations.

Gunjan Balde, Soumyadeep Roy, Mainack Mondal, and Niloy Ganguly. 2024. Medvoc: vocabulary adaptation for fine-tuning pre-trained language models on medical text summarization. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI '24.

Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. 2003. A neural probabilistic language model. Journal of Machine Learning Research, 3:1137–1155.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hal-lahan, Mohammad Aflah Khan, Shivanshu Purohit, Usvsn Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar Van Der Wal. 2023. Pythia: A suite for analyzing large language models across training and scaling. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 2397–2430. PMLR.

Yonatan Bisk, Rowan Zellers, Ronan Le bras, Jianfeng Gao, and Yejin Choi. 2020. Piqa: Reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7432–7439.

Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. arXiv preprint arXiv:1607.04606.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc.

Ethan C. Chau, Lucy H. Lin, and Noah A. Smith. 2020. Parsing with multilingual BERT, a small corpus, and a small treebank. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1324–1334, Online. Association for Computational Linguistics.

Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555.

Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2924–2936, Minneapolis, Minnesota. Association for Computational Linguistics.

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. Preprint, arXiv:1803.05457.

Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475–2485, Brussels, Belgium. Association for Computational Linguistics.

Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. Flashattention: Fast and memory-efficient exact attention with io-awareness. In Advances in Neural Information Processing Systems, volume 35, pages 16344–16359. Curran Associates, Inc.

Konstantin Dobler and Gerard de Melo. 2023. FOCUS: Effective embedding initialization for monolingual specialization of multilingual models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13440–13454, Singapore. Association for Computational Linguistics.

C.m. Downey, Terra Blevins, Nora Goldfine, and Shane Steinert-Threlkeld. 2023. Embedding structure matters: Comparing methods to adapt multilingual vocabularies to new languages. In Proceedings of the 3rd Workshop on Multi-lingual Representation Learning (MRL), pages 268–281, Singapore. Association for Computational Linguistics.

Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. The pile: An 800gb dataset of diverse text for language modeling. Preprint, arXiv:2101.00027.

Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muenninghoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2024. A framework for few-shot language model evaluation.

Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, and et al. 2024. Chatglm: A family of large language models from glm-130b to glm-4 all tools. Preprint, arXiv:2406.12793.

Jonathan Hayase, Alisa Liu, Yejin Choi, Sewoong Oh, and Noah A Smith. 2024. Data mixture inference: What do bpe tokenizers reveal about their training data? arXiv preprint arXiv:2407.16607.

S Hochreiter. 1997. Long short-term memory. Neural Computation MIT-Press.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. 2022. Training compute-optimal large language models. Preprint, arXiv:2203.15556.

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825.

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. Preprint, arXiv:2001.08361.

Yoon Kim and Alexander M. Rush. 2016. Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1317–1327, Austin, Texas. Association for Computational Linguistics.

Denis Kocetkov, Raymond Li, Loubna Ben allal, Jia LI, Chenghao Mou, Yacine Jernite, Margaret Mitchell, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, Leandro Von Werra, and Harm de Vries. 2023. The stack: 3 TB of permissively licensed source code. Transactions on Machine Learning Research.

Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66–75, Melbourne, Australia. Association for Computational Linguistics.

Viet Lai, Chien Nguyen, Nghia Ngo, Thuat Nguyen, Franck Demontcourt, Ryan Rossi, and Thien Nguyen. 2023. Okapi: Instruction-tuned large language models in multiple languages with reinforcement learning from human feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 318–327, Singapore. Association for Computational Linguistics.

Chong Li, Shaonan Wang, Jiajun Zhang, and Chengqing Zong. 2024. Improving in-context learning of multilingual generative language models with cross-lingual alignment. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 8058–8076, Mexico City, Mexico. Association for Computational Linguistics.

Haoran Li, Jiajun Zhang, and Chengqing Zong. 2017. Implicit discourse relation recognition for english and chinese with multiview modeling and effective representation learning. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP), 16(3):1–21.

Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuhui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, and Xian Li. 2022. Few-shot learning with multilingual generative language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9019–9052, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Yihong Liu, Peiqin Lin, Mingyang Wang, and Hinrich Schutze. 2024. OFA: A framework of initializing unseen subword embeddings for efficient large-scale multilingual continued pretraining. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 1067–1097, Mexico City, Mexico. Association for Computational Linguistics.

Ilya Loschilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations.

Jinliang Lu, Ziliang Pang, Min Xiao, Yaochen Zhu, Rui Xia, and Jiajun Zhang. 2024. Merge, ensemble, and cooperate! a survey on collaborative strategies in the era of large language models. Preprint, arXiv:2407.06089.

Meta. 2024. Introducing meta llama 3: The most capable openly available llm to date. Qwen blog.

Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. 2018. Mixed precision training. In International Conference on Learning Representations.

Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381–2391, Brussels, Belgium. Association for Computational Linguistics.

Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781.

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013b. Distributed representations of words and phrases and their compositionality. arXiv preprint arXiv:1310.4546.

Benjamin Minixhofer, Fabian Paischer, and Navid Rekabsaz. 2022. WECHSEL: Effective initialization of
subword embeddings for cross-lingual transfer of monolingual language models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3992–4006, Seattle, United States. Association for Computational Linguistics.

Benjamin Minixhofer, Edoardo Maria Ponti, and Ivan Vulić. 2024. Zero-shot tokenizer transfer. Preprint, arXiv:2405.07883.

Edoardo Mosca, Mohamed Hesham Ibrahim Abdalla, Paolo Basso, Margherita Musumeci, and Georg Groh. 2023. Distinguishing fact from fiction: A benchmark dataset for identifying machine-generated scientific papers in the LLM era. In Proceedings of the 3rd Workshop on Trustworthy Natural Language Processing (TrustNLP 2023), pages 190–207, Toronto, Canada. Association for Computational Linguistics.

Luca Moschella, Valentino Maiorca, Marco Fumero, Antonio Norrelli, Francesco Locatello, and Emanuele Rodolà. 2023. Relative representations enable zero-shot latent space communication. In The Eleventh International Conference on Learning Representations.

Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Haley Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. 2023. Crosslingual generalization through multitask finetuning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15991–16111, Toronto, Canada. Association for Computational Linguistics.

Nandini Mundra, Aditya Nanda Kishore Khandavally, Raj Dabre, Ratish Puduppully, Anoop Kunchukuttan, and Mitesh M Khapra. 2024. An empirical comparison of vocabulary expansion and initialization approaches for language models. In Proceedings of the 28th Conference on Computational Natural Language Learning, pages 84–104, Miami, FL, USA. Association for Computational Linguistics.

Thuat Nguyen, Chien Van Nguyen, Viet Dac Lai, Hieu Man, Nghia Trung Ngo, Franck Dernoncourt, Ryan A. Rossi, and Thien Huu Nguyen. 2024. CulturaX: A cleaned, enormous, and multilingual dataset for large language models in 167 languages. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 4226–4237, Torino, Italia. ELRA and ICCL.

Thuat Nguyen, Chien Van Nguyen, Viet Dac Lai, Hieu Man, Nghia Trung Ngo, Franck Dernoncourt, Ryan A. Rossi, and Thien Huu Nguyen. 2023. CulturaX: A cleaned, enormous, and multilingual dataset for large language models in 167 languages. arXiv preprint arXiv:2309.09400.

OpenAI. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.

Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar. Association for Computational Linguistics.

Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. 2020. XCOPA: A multilingual dataset for causal commonsense reasoning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2362–2376, Online. Association for Computational Linguistics.

Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. OpenAI blog.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog.

Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’20, page 3505–3506, New York, NY, USA. Association for Computing Machinery.

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Winogrande: An adversarial winograd schema challenge at scale. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):8732–8740.

BigScience Workshop: Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, and Alexandra Sasha Luccioni et al. 2023. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100.

Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany. Association for Computational Linguistics.

Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. 2023. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama.

Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2020. Mpnet: Masked and permuted pre-training for language understanding. In Advances in Neural Information Processing Systems, volume 33, pages 16857–16867. Curran Associates, Inc.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. 2024. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295.

Alexey Tikhonov and Max Ryabinin. 2021. It’s All in the Heads: Using Attention Heads as a Baseline for Cross-Lingual Transfer in Commonsense Reasoning. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3534–3546, Online. Association for Computational Linguistics.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a. Llama: Open and efficient foundation language models. Preprint, arXiv:2302.13971.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023b. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.

Ke Tran. 2020. From english to foreign languages: Transferring pre-trained language models. arXiv preprint arXiv:2002.07306.

Ahmet Üstün, Viraa Aryabumi, Zheng Yong, Wei-Yin Ko, Daniel D’souza, Gbemileke Onilude, Neel Bhandari, Shivalka Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargas, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadaee, Julia Kreutzer, and Sara Hooker. 2024. Aya model: An instruction fine-tuned open-access multilingual language model. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15894–15939, Bangkok, Thailand. Association for Computational Linguistics.

Shaonan Wang, Jiajun Zhang, and Chengqing Zong. 2018. Learning multimodal word representation via dynamic fusion methods. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 5973–5980. AAAI Press.

Jingxuan Wei, Linzhuang Sun, Yichong Leng, Xu Tan, Bihui Yu, and Ruifeng Guo. 2024. Sentence-level or token-level? A comprehensive study on knowledge distillation. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI 2024, Jeju, South Korea, August 3-9, 2024, pages 6531–6540. ijcai.org.

Tianwen Wei, Liang Zhao, Lichang Zhang, Bo Zhu, Lijie Wang, Haihua Yang, Biye Li, Cheng Cheng, Weiwei Lu, Rui Hu, et al. 2023. Skywork: A more open bilingual foundation model. arXiv preprint arXiv:2310.19341.

T Wolf. 2019. Huggingface’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771.

Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. Preprint, arXiv:1609.08144.

Yangyifan Xu, Jinliang Lu, and Jiajun Zhang. 2024. Bridging the gap between different vocabularies for LLM ensemble. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 7140–7152, Mexico City, Mexico. Association for Computational Linguistics.

Atsuki Yamaguchi, Aline Villavicencio, and Nikolaos Aletras. 2024. An empirical study on cross-lingual vocabulary adaptation for efficient language model inference. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 6760–6785, Miami, Florida, USA. Association for Computational Linguistics.

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Owen2 technical report. arXiv preprint arXiv:2407.10671.

Wen Yang, Chong Li, Jiajun Zhang, and Chengqing Zong. 2023. Bitranslate: Augmenting large language models with multilingual translation capability over 100 languages. arXiv preprint arXiv:2305.18098.

Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019. PAWS-X: A cross-lingual adversarial dataset for paraphrase identification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3687–3692, Hong Kong, China. Association for Computational Linguistics.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, Florence, Italy. Association for Computational Linguistics.
Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. 2024. Tinyllama: An open-source small language model. arXiv preprint arXiv:2401.02385.

Wenhao Zhu, Yunzhe Lv, Qingxiu Dong, Fei Yuan, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. 2023. Extrapolating large language models to non-english by aligning languages. arXiv preprint arXiv:2308.04948.

A Hyper-parameters

GloVe Training  We empirically train GloVe vectors with 1B tokens, which covers most tokens from Gemma (95.10%), Qwen2 (93.40%), LLaMA2 (99.35%), and LLaMA3 (98.04%). The dimension size is set to 300. The max training iteration and the size of the slide window are 15.

Model Tuning  The optimizer adopted in this work is AdamW (Loshchilov and Hutter, 2019), where \( \beta_1 = 0.9 \) and \( \beta_2 = 0.999 \). The learning rate for baseline methods is set to 5e-5 to reduce the loss spike in Figure 6(b) and Figure 6(c). We adopt bf16 mixed precision training, ZeRO-1, and flash-attention to save GPU memory cost and speed up the training process (Micikevicius et al., 2018; Rasley et al., 2020; Dao et al., 2022). Following Biderman et al. (2023), the batch size is set to 2M tokens and the max sequence length is 2048.

B Additional Results

B.1 Tokenizer Compression Rate

Table 6 reports detailed compression rates of tokenizers across different domains and languages. We randomly sample 10 subsets or languages from vanilla datasets (Azerbayev et al., 2024; Kocetkov et al., 2023) to estimate the compression rate. Following Lai et al. (2023), the division of languages between “High”, “Medium” and “Low” is determined by the available amount resource on CommonCrawl.

B.2 GloVe Vectors

We show the effects of different token amounts for the GloVe vectors training in Figure 8. It can be found that 1B tokens used in this work provide a high vocabulary coverage (>90%) and better initialization for Pythia1B. Due to the limited computation budget, experiments with more than 1B tokens are not conducted.

B.3 Convergence Analysis

To investigate the effect of overlapping rate between two tokenizers to the convergence of training, we plot Figure 10 for the random initialization baseline method. The convergence of Gemma tokenizer is slower than the other tokenizers and comes to worse results, which are similar to the case in Figure 9.

Moreover, we randomly shuffle the alignment matrix learned in TokAlign to imitate the case that other worse methods rather than cosine similarity to calculate the alignment matrix. Figure 11 shows that the higher percentage of randomly shuffle comes to higher initial training loss and slower convergence.

B.4 Fast Vocabulary Adaptation Results

We further investigate a challenge condition that fine-tunes only 2B tokens to adapt the target vocabulary. To meet the requirement, we reduce the batch size to 1M tokens and set the number of fine-tuning steps to 2k. Table 7 shows the results of adapting to the other 3 tokenizers using TokAlign. It can be found that 95.66% performance of the vanilla model is recovered on average, which further demonstrates the effectiveness of our method.

B.5 In-context Learning Results during Cross-lingual Transfer

Table 2 and 8 report the 0-shot and 5-shot in-context learning results on 4 multilingual datasets. The average improvement over the baseline method Focus is 2.35% after language adaptation pre-training. We can find that the model initialized by TokAlign is comparable to the one of Focus after language adaptation pre-training, which mainly comes from the strong English performance preserved by TokAlign.

Case study of multilingual token alignment. Table 9 provides nine new tokens from three languages with their top 3 tokens in the source vocabulary for qualitative analyses. In most cases, a clear semantic relationship between two aligned tokens cannot be found. We argue that it may come from the following two reasons:

• BPE algorithm (Sennrich et al., 2016) divides words into the sub-word units, also called tokens, from the statistical co-occurrence information. There may be less superficial semantic information in the tokens divided compared with words in the natural language.
<table>
  <tr>
    <th rowspan="2">Domain</th>
    <th rowspan="2">Subset / Language</th>
    <th colspan="6">Tokenizer</th>
  </tr>
  <tr>
    <th>Gemma</th>
    <th>LLaMA3</th>
    <th>LLaMA2</th>
    <th>Qwen2</th>
    <th>Pythia</th>
  </tr>
  <tr>
    <td rowspan="5"><b>Math</b><br>(Azerbayev et al., 2024)</td>
    <td><i>ArXiv</i></td>
    <td>2.8561</td>
    <td>2.7765</td>
    <td>2.7040</td>
    <td>2.7445</td>
    <td>2.8489</td>
  </tr>
  <tr>
    <td><i>Textbooks</i></td>
    <td>4.0883</td>
    <td>4.3270</td>
    <td>3.6500</td>
    <td>4.2899</td>
    <td>3.9464</td>
  </tr>
  <tr>
    <td><i>Wikipedia</i></td>
    <td>3.1753</td>
    <td>3.2049</td>
    <td>2.8792</td>
    <td>3.0312</td>
    <td>3.2898</td>
  </tr>
  <tr>
    <td><i>ProofWiki</i></td>
    <td>2.7538</td>
    <td>2.8115</td>
    <td>2.5996</td>
    <td>2.7900</td>
    <td>2.7363</td>
  </tr>
  <tr>
    <td><i>StackExchange</i></td>
    <td>3.2062</td>
    <td>3.2814</td>
    <td>3.0094</td>
    <td>3.2107</td>
    <td>3.2222</td>
  </tr>
  <tr>
    <td><i>WebPages</i></td>
    <td>3.9885</td>
    <td>4.0655</td>
    <td>3.5070</td>
    <td>3.8720</td>
    <td>4.1136</td>
  </tr>
  <tr>
    <td rowspan="8"><b>Code</b><br>(Kocetkov et al., 2023)</td>
    <td><i>Python</i></td>
    <td>3.3401</td>
    <td>4.1331</td>
    <td>3.0072</td>
    <td>4.0339</td>
    <td>3.2328</td>
  </tr>
  <tr>
    <td><i>Java</i></td>
    <td>3.7175</td>
    <td>4.4900</td>
    <td>3.2193</td>
    <td>4.4141</td>
    <td>3.4914</td>
  </tr>
  <tr>
    <td><i>Go</i></td>
    <td>2.9274</td>
    <td>3.4797</td>
    <td>2.5189</td>
    <td>3.3870</td>
    <td>2.8542</td>
  </tr>
  <tr>
    <td><i>VHDL</i></td>
    <td>2.1038</td>
    <td>2.4814</td>
    <td>1.8724</td>
    <td>2.2961</td>
    <td>2.1395</td>
  </tr>
  <tr>
    <td><i>ActionScript</i></td>
    <td>3.3470</td>
    <td>3.9717</td>
    <td>2.7852</td>
    <td>3.9180</td>
    <td>3.2949</td>
  </tr>
  <tr>
    <td><i>Scheme</i></td>
    <td>2.7178</td>
    <td>3.3045</td>
    <td>2.4586</td>
    <td>2.9713</td>
    <td>2.9326</td>
  </tr>
  <tr>
    <td><i>Haml</i></td>
    <td>3.2423</td>
    <td>3.8429</td>
    <td>2.9588</td>
    <td>3.8002</td>
    <td>3.1016</td>
  </tr>
  <tr>
    <td><i>Xbase</i></td>
    <td>2.8739</td>
    <td>3.4325</td>
    <td>2.3300</td>
    <td>3.3475</td>
    <td>2.7837</td>
  </tr>
  <tr>
    <td><i>Mako</i></td>
    <td>3.4387</td>
    <td>4.0746</td>
    <td>3.1238</td>
    <td>4.0311</td>
    <td>3.2844</td>
  </tr>
  <tr>
    <td><i>EmberScript</i></td>
    <td>1.4104</td>
    <td>1.9017</td>
    <td>1.3819</td>
    <td>1.4082</td>
    <td>2.1540</td>
  </tr>
  <tr>
    <td rowspan="8"><b>High-Langs</b><br>(Nguyen et al., 2023)</td>
    <td><i>English</i></td>
    <td>4.4971</td>
    <td>4.6042</td>
    <td>3.8647</td>
    <td>4.4875</td>
    <td>4.4505</td>
  </tr>
  <tr>
    <td><i>Russian</i></td>
    <td>6.7529</td>
    <td>5.8131</td>
    <td>4.9275</td>
    <td>5.3559</td>
    <td>3.5802</td>
  </tr>
  <tr>
    <td><i>Spanish</i></td>
    <td>4.6068</td>
    <td>3.8416</td>
    <td>3.4517</td>
    <td>3.8330</td>
    <td>3.3655</td>
  </tr>
  <tr>
    <td><i>German</i></td>
    <td>4.4605</td>
    <td>3.6314</td>
    <td>3.4417</td>
    <td>3.6041</td>
    <td>3.1096</td>
  </tr>
  <tr>
    <td><i>French</i></td>
    <td>4.2258</td>
    <td>3.7378</td>
    <td>3.4445</td>
    <td>3.7243</td>
    <td>3.3565</td>
  </tr>
  <tr>
    <td><i>Chinese</i></td>
    <td>3.7378</td>
    <td>3.2373</td>
    <td>1.8434</td>
    <td>3.9859</td>
    <td>1.9896</td>
  </tr>
  <tr>
    <td><i>Italian</i></td>
    <td>4.2211</td>
    <td>3.4952</td>
    <td>3.3320</td>
    <td>3.4573</td>
    <td>3.1928</td>
  </tr>
  <tr>
    <td><i>Portuguese</i></td>
    <td>4.2731</td>
    <td>3.6030</td>
    <td>3.2031</td>
    <td>3.5850</td>
    <td>3.2022</td>
  </tr>
  <tr>
    <td><i>Polish</i></td>
    <td>3.5583</td>
    <td>2.8548</td>
    <td>2.6639</td>
    <td>2.9464</td>
    <td>2.4333</td>
  </tr>
  <tr>
    <td><i>Japanese</i></td>
    <td>5.7640</td>
    <td>4.2796</td>
    <td>2.4701</td>
    <td>4.7059</td>
    <td>2.9326</td>
  </tr>
  <tr>
    <td rowspan="8"><b>Medium-Langs</b><br>(Nguyen et al., 2023)</td>
    <td><i>Czech</i></td>
    <td>3.3402</td>
    <td>3.2875</td>
    <td>2.5978</td>
    <td>2.4490</td>
    <td>2.3884</td>
  </tr>
  <tr>
    <td><i>Vietnamese</i></td>
    <td>4.5376</td>
    <td>4.2766</td>
    <td>1.9699</td>
    <td>4.2877</td>
    <td>2.0382</td>
  </tr>
  <tr>
    <td><i>Persian</i></td>
    <td>5.6465</td>
    <td>5.3015</td>
    <td>1.7938</td>
    <td>3.1923</td>
    <td>2.3707</td>
  </tr>
  <tr>
    <td><i>Hungarian</i></td>
    <td>3.2337</td>
    <td>2.6008</td>
    <td>2.6311</td>
    <td>2.5500</td>
    <td>2.3878</td>
  </tr>
  <tr>
    <td><i>Greek</i></td>
    <td>4.4691</td>
    <td>4.5671</td>
    <td>1.8544</td>
    <td>2.1225</td>
    <td>3.0283</td>
  </tr>
  <tr>
    <td><i>Romanian</i></td>
    <td>3.5558</td>
    <td>3.0566</td>
    <td>2.8355</td>
    <td>3.0083</td>
    <td>2.8981</td>
  </tr>
  <tr>
    <td><i>Swedish</i></td>
    <td>3.7087</td>
    <td>3.1398</td>
    <td>2.9214</td>
    <td>3.0977</td>
    <td>2.9620</td>
  </tr>
  <tr>
    <td><i>Ukrainian</i></td>
    <td>5.5141</td>
    <td>5.5985</td>
    <td>4.5904</td>
    <td>3.6179</td>
    <td>3.0702</td>
  </tr>
  <tr>
    <td><i>Finnish</i></td>
    <td>3.2659</td>
    <td>2.6748</td>
    <td>2.4176</td>
    <td>2.6473</td>
    <td>2.6112</td>
  </tr>
  <tr>
    <td><i>Korean</i></td>
    <td>3.3556</td>
    <td>3.6957</td>
    <td>1.5977</td>
    <td>3.3330</td>
    <td>1.5667</td>
  </tr>
  <tr>
    <td rowspan="7"><b>Low-Langs</b><br>(Nguyen et al., 2023)</td>
    <td><i>Hebrew</i></td>
    <td>4.0487</td>
    <td>1.8592</td>
    <td>1.7875</td>
    <td>4.3773</td>
    <td>2.0380</td>
  </tr>
  <tr>
    <td><i>Serbian</i></td>
    <td>4.8596</td>
    <td>3.9234</td>
    <td>4.2642</td>
    <td>3.6267</td>
    <td>2.9896</td>
  </tr>
  <tr>
    <td><i>Tamil</i></td>
    <td>5.6161</td>
    <td>2.0279</td>
    <td>2.2615</td>
    <td>2.4759</td>
    <td>1.9765</td>
  </tr>
  <tr>
    <td><i>Albanian</i></td>
    <td>2.8919</td>
    <td>2.6536</td>
    <td>2.2945</td>
    <td>2.6037</td>
    <td>2.3631</td>
  </tr>
  <tr>
    <td><i>Azerbaijani</i></td>
    <td>2.8585</td>
    <td>2.4857</td>
    <td>2.0407</td>
    <td>2.3797</td>
    <td>2.1534</td>
  </tr>
  <tr>
    <td><i>Kazakh</i></td>
    <td>3.8172</td>
    <td>2.9176</td>
    <td>3.0869</td>
    <td>2.9263</td>
    <td>2.3236</td>
  </tr>
  <tr>
    <td><i>Urdu</i></td>
    <td>4.4364</td>
    <td>2.8462</td>
    <td>1.7260</td>
    <td>2.7174</td>
    <td>1.9458</td>
  </tr>
  <tr>
    <td><i>Georgian</i></td>
    <td>3.8237</td>
    <td>1.4828</td>
    <td>2.5595</td>
    <td>2.6951</td>
    <td>2.2077</td>
  </tr>
  <tr>
    <td><i>Armenian</i></td>
    <td>3.2133</td>
    <td>1.1658</td>
    <td>1.7000</td>
    <td>1.8531</td>
    <td>1.3922</td>
  </tr>
  <tr>
    <td><i>Icelandic</i></td>
    <td>2.7964</td>
    <td>2.4860</td>
    <td>2.3050</td>
    <td>2.4330</td>
    <td>2.3185</td>
  </tr>
</table>

Table 6: The compression rates (bytes/token) of different tokenizers.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="3">ARC-E</th>
    <th colspan="3">BoolQ</th>
    <th colspan="3">HellaSwag</th>
    <th colspan="3">OpenbookQA</th>
    <th colspan="3">PIQA</th>
    <th colspan="3">WinoGrande</th>
    <th colspan="3">Avg</th>
  </tr>
  <tr>
    <th>#\( \mathcal{V} \) (k)</th>
    <th>0</th>
    <th>5</th>
    <th>0</th>
    <th>5</th>
    <th>0</th>
    <th>5</th>
    <th>0</th>
    <th>5</th>
    <th>0</th>
    <th>5</th>
    <th>0</th>
    <th>5</th>
    <th>0</th>
    <th>5</th>
    <th>0</th>
    <th>5</th>
    <th>0</th>
    <th>5</th>
    <th>0</th>
    <th>5</th>
  </tr>
  <tr>
    <td>Pythia<sub>1B</sub></td>
    <td></td>
    <td>50.3</td>
    <td>56.82</td>
    <td>58.71</td>
    <td>60.43</td>
    <td>57.37</td>
    <td>37.68</td>
    <td>37.66</td>
    <td>18.80</td>
    <td>19.00</td>
    <td>70.40</td>
    <td>71.49</td>
    <td>53.20</td>
    <td>52.01</td>
    <td>49.55</td>
    <td>49.37</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>\( \rightarrow \) Gemma</td>
    <td></td>
    <td>256.0</td>
    <td>51.09</td>
    <td>52.44</td>
    <td>53.12</td>
    <td>52.35</td>
    <td>35.00</td>
    <td>35.05</td>
    <td>20.20</td>
    <td>18.60</td>
    <td>64.80</td>
    <td>65.83</td>
    <td>53.12</td>
    <td>51.62</td>
    <td>46.22</td>
    <td>45.98</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>\( \rightarrow \) Qwen2</td>
    <td></td>
    <td>152.1</td>
    <td>53.41</td>
    <td>55.47</td>
    <td>53.52</td>
    <td>55.81</td>
    <td>36.12</td>
    <td>36.38</td>
    <td>20.80</td>
    <td>18.00</td>
    <td>68.50</td>
    <td>68.88</td>
    <td>54.38</td>
    <td>52.80</td>
    <td>47.79</td>
    <td>47.89</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>\( \rightarrow \) LLaMA3</td>
    <td></td>
    <td>128.0</td>
    <td>51.73</td>
    <td>55.09</td>
    <td>59.05</td>
    <td>55.08</td>
    <td>36.42</td>
    <td>36.52</td>
    <td>19.40</td>
    <td>19.60</td>
    <td>67.68</td>
    <td>68.34</td>
    <td>53.43</td>
    <td>53.75</td>
    <td>47.95</td>
    <td>48.06</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 7: The main results of replacing the vocabulary of Pythia for TokAlign using 2B tokens from the Pile corpus.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="6">XNLI</th>
    <th colspan="6">PAWS-X</th>
    <th colspan="6">XCOPA</th>
    <th colspan="6">XStoryCloze</th>
    <th colspan="6">Avg</th>
  </tr>
  <tr>
    <th>en</th><th>de</th><th>zh</th><th>ar</th><th>th</th><th>vi</th>
    <th>en</th><th>ja</th><th>ko</th><th>zh</th>
    <th>th</th><th>vi</th><th>ta</th>
    <th>en</th><th>zh</th><th>ar</th><th>te</th>
    <th>Avg</th>
  </tr>
  <tr>
    <td>Pythia<sub>1B</sub></td>
    <td>16.2</td><td>38.6</td><td>38.9</td><td>36.9</td><td>35.2</td><td>38.9</td>
    <td>34.9</td><td>48.9</td><td>48.3</td><td>52.9</td>
    <td>53.3</td><td>54.1</td><td>53.4</td><td>52.6</td><td>55.4</td><td>65.3</td><td>48.6</td><td>48.2</td><td>52.2</td>
    <td>47.5</td>
  </tr>
  <tr>
    <td>w/ Focus Init.</td>
    <td>32.8</td><td>32.2</td><td>33.6</td><td>33.6</td><td>33.5</td><td>32.0</td>
    <td>32.8</td><td>44.8</td><td>46.0</td><td>48.9</td>
    <td>44.8</td><td>44.7</td><td>51.4</td><td>47.6</td><td>55.6</td><td>45.9</td><td>48.6</td><td>48.5</td><td>46.8</td>
    <td>42.3</td>
  </tr>
  <tr>
    <td>+ LAT</td>
    <td>47.0</td><td>36.7</td><td>35.4</td><td>34.3</td><td>33.5</td><td>35.1</td>
    <td>33.9</td><td>51.5</td><td>48.6</td><td>53.7</td>
    <td>51.2</td><td>54.0</td><td>54.4</td><td>51.6</td><td>55.6</td><td>55.8</td><td>48.7</td><td>47.5</td><td>50.4</td>
    <td>46.3</td>
  </tr>
  <tr>
    <td>w/ TokAlign Init.</td>
    <td>44.9</td><td>37.4</td><td>34.0</td><td>32.8</td><td>35.3</td><td>35.2</td>
    <td>34.5</td><td>50.2</td><td>50.3</td><td>52.0</td>
    <td>53.1</td><td>54.4</td><td>54.4</td><td>50.0</td><td>54.4</td><td>61.2</td><td>48.3</td><td>47.6</td><td>50.0</td>
    <td>46.3</td>
  </tr>
  <tr>
    <td>+ LAT</td>
    <td>44.4</td><td>39.0</td><td>38.7</td><td>35.6</td><td>35.1</td><td>37.8</td>
    <td>35.5</td><td>51.9</td><td>49.3</td><td>54.7</td>
    <td>53.1</td><td>50.6</td><td>54.2</td><td>54.0</td><td>52.8</td><td>64.7</td><td>50.8</td><td>48.0</td><td>52.4</td>
    <td>47.5</td>
  </tr>
  <tr>
    <td>Pythia<sub>0.0B</sub></td>
    <td>53.0</td><td>40.7</td><td>41.7</td><td>38.9</td><td>37.3</td><td>41.3</td>
    <td>35.1</td><td>49.4</td><td>47.1</td><td>52.9</td>
    <td>52.2</td><td>52.4</td><td>55.0</td><td>53.6</td><td>53.6</td><td>73.1</td><td>54.6</td><td>49.9</td><td>53.9</td>
    <td>49.2</td>
  </tr>
  <tr>
    <td>w/ Focus Init.</td>
    <td>31.5</td><td>31.3</td><td>33.0</td><td>32.6</td><td>34.3</td><td>32.2</td>
    <td>32.6</td><td>44.8</td><td>46.4</td><td>52.3</td>
    <td>51.2</td><td>54.5</td><td>52.4</td><td>47.4</td><td>56.0</td><td>44.9</td><td>47.3</td><td>48.5</td><td>47.6</td>
    <td>43.1</td>
  </tr>
  <tr>
    <td>+ LAT</td>
    <td>45.1</td><td>37.7</td><td>35.3</td><td>34.3</td><td>35.0</td><td>38.1</td>
    <td>33.8</td><td>49.5</td><td>49.0</td><td>52.6</td>
    <td>54.5</td><td>55.3</td><td>52.0</td><td>51.2</td><td>53.8</td><td>61.5</td><td>48.3</td><td>47.3</td><td>53.4</td>
    <td>46.7</td>
  </tr>
  <tr>
    <td>w/ TokAlign Init.</td>
    <td>50.8</td><td>39.1</td><td>34.4</td><td>34.5</td><td>33.9</td><td>34.6</td>
    <td>35.2</td><td>50.0</td><td>47.7</td><td>53.9</td>
    <td>54.3</td><td>55.2</td><td>53.2</td><td>51.2</td><td>53.2</td><td>68.0</td><td>48.5</td><td>47.8</td><td>50.2</td>
    <td>47.1</td>
  </tr>
  <tr>
    <td>+ LAT</td>
    <td>49.2</td><td>41.5</td><td>37.8</td><td>36.9</td><td>38.7</td><td>41.9</td>
    <td>34.7</td><td>51.2</td><td>49.5</td><td>53.5</td>
    <td>54.8</td><td>55.4</td><td>53.4</td><td>59.8</td><td>52.8</td><td>73.0</td><td>53.9</td><td>49.2</td><td>53.6</td>
    <td>49.5</td>
  </tr>
</table>

Table 8: Five-shot in-context learning results of cross-lingual transfer.

<table>
  <tr>
    <th rowspan="2">Top-3</th>
    <th colspan="3">French</th>
    <th colspan="3">Chinese</th>
    <th colspan="3">Korean</th>
  </tr>
  <tr>
    <th>dire(speak)</th>
    <th>aller(go)</th>
    <th>oui(are)</th>
    <th>吃(eat)</th>
    <th>科学(science)</th>
    <th>智能(intelligence)</th>
    <th>능(competence)</th>
    <th>집(house)</th>
    <th>왜(why)</th>
  </tr>
  <tr>
    <td colspan="10"><i>Owen2 (Target Tokenizer)</i></td>
  </tr>
  <tr>
    <td>1</td>
    <td>ada</td>
    <td>Gsta</td>
    <td>Gsalv</td>
    <td>allel</td>
    <td>Gantagon</td>
    <td>_[[</td>
    <td>Si</td>
    <td>GBart</td>
    <td>bst</td>
  </tr>
  <tr>
    <td>2</td>
    <td>ays</td>
    <td>GA-</td>
    <td>Gvas</td>
    <td>Gindicator</td>
    <td>Gign</td>
    <td>liquid</td>
    <td>uria</td>
    <td>GPAT</td>
    <td>rains</td>
  </tr>
  <tr>
    <td>3</td>
    <td>G-</td>
    <td>Gdetermin</td>
    <td>Gexplos</td>
    <td>Gbasic</td>
    <td>Gritic</td>
    <td>Layer</td>
    <td>ost</td>
    <td>GEdgar</td>
    <td>irc</td>
  </tr>
  <tr>
    <td colspan="10"><i>Gemma (Target Tokenizer)</i></td>
  </tr>
  <tr>
    <td>1</td>
    <td>Gj</td>
    <td>Cor</td>
    <td>Tools</td>
    <td>kernel</td>
    <td>GLed</td>
    <td>Gcommittee</td>
    <td>Gmang</td>
    <td>Gcru</td>
    <td>Gcholesterol</td>
  </tr>
  <tr>
    <td>2</td>
    <td>Gdar</td>
    <td>Gequality</td>
    <td>directed</td>
    <td>sentence</td>
    <td>COUNT</td>
    <td>GUND</td>
    <td>ial</td>
    <td>Gcal</td>
    <td>Gmolecule</td>
  </tr>
  <tr>
    <td>3</td>
    <td>ba</td>
    <td>Lex</td>
    <td>afx</td>
    <td>messages</td>
    <td>Gglycine</td>
    <td>Gfactors</td>
    <td>Grebut</td>
    <td>Gmalt</td>
    <td>apor</td>
  </tr>
</table>

Table 9: The case study of new tokens from other languages in the target vocabulary with top-3 source tokens aligned. The language family of French, Chinese, and Korean are Indo-European, Sino-Tibetan, and Koreanic, respectively.
Figure 8: The average vocabulary coverage (a) and initial training loss of Pythia1B (b) under different amount tokens to train the GloVe vector.

Figure 9: The training loss curve of Pythia1B for different overlapping ratios.

Figure 10: The training loss to different tokenizers using random initialization baseline.

• The GloVe vector for each token is obtained from the token-token co-occurrence information. These aligned tokens often appear together, e.g., 科学(science) and “Gcritic”, 왜(why) and “rains”.

Therefore, it is better to choose a matric to quantify the performance of the alignment matrix learned, for example, the BLEU-1 score or BERTScore in Section 3.2.

Figure 11: The training loss of Pythia1B when replacing tokenizer to Qwen2 under different percentages of shuffling.

C Evaluation Tasks

We report the statistics of evaluation tasks used in Table 10. Here are the descriptions of these evaluation tasks:

Natural Language Inference aims to determine the semantic relationship (Entailment, neural, or contradiction) between the premise and hypothesis (Conneau et al., 2018).

Paraphrase Detection requires the model to evaluate whether the second sentence is a paraphrase of the first sentence in this task (Yang et al., 2019).

Commonsense Reasoning is a task for the model to reason the gold answer based on the semantic coherence and physic rules (Clark et al., 2018; Mi-haylov et al., 2018; Zellers et al., 2019; Ponti et al., 2020; Bisk et al., 2020; Sakaguchi et al., 2020; Tikhonov and Ryabinin, 2021).
<table>
  <tr>
    <th>Task</th>
    <th>Dataset</th>
    <th>#Lang</th>
    <th>#Class</th>
    <th>Data Curation</th>
    <th>#Train</th>
    <th>#Dev</th>
    <th>#Test</th>
  </tr>
  <tr>
    <td rowspan="2">Natural Language Inference</td>
    <td>XNLI</td>
    <td>15</td>
    <td>3</td>
    <td>Translation</td>
    <td>–</td>
    <td>2,490</td>
    <td>5,010</td>
  </tr>
  <tr>
    <td>PAWS-X</td>
    <td>7</td>
    <td>2</td>
    <td>Aligned</td>
    <td>–</td>
    <td>2,000</td>
    <td>2,000</td>
  </tr>
  <tr>
    <td rowspan="8">Paraphrase Detection</td>
    <td>ARC-Easy</td>
    <td>4</td>
    <td>–</td>
    <td>2,251</td>
    <td>570</td>
    <td>2,376</td>
    <td></td>
  </tr>
  <tr>
    <td>HellaSwag</td>
    <td>1</td>
    <td>4</td>
    <td>–</td>
    <td>39,905</td>
    <td>10,042</td>
    <td>10,003</td>
  </tr>
  <tr>
    <td>OpenbookQA</td>
    <td>1</td>
    <td>4</td>
    <td>–</td>
    <td>4,957</td>
    <td>500</td>
    <td>500</td>
  </tr>
  <tr>
    <td>PIQA</td>
    <td>1</td>
    <td>2</td>
    <td>–</td>
    <td>16,000</td>
    <td>2,000</td>
    <td>3,000</td>
  </tr>
  <tr>
    <td>XCOPA</td>
    <td>12</td>
    <td>2</td>
    <td>Translation</td>
    <td>33,810</td>
    <td>100</td>
    <td>500</td>
  </tr>
  <tr>
    <td>XStoryCloze</td>
    <td>11</td>
    <td>2</td>
    <td>Translation</td>
    <td>361</td>
    <td>–</td>
    <td>1,511</td>
  </tr>
  <tr>
    <td>WinoGrad</td>
    <td>1</td>
    <td>2</td>
    <td>–</td>
    <td>40,398</td>
    <td>1,267</td>
    <td>1,767</td>
  </tr>
  <tr>
    <td>Reading Comprehension</td>
    <td>BoolQ</td>
    <td>1</td>
    <td>2</td>
    <td>–</td>
    <td>9,427</td>
    <td>3,270</td>
    <td>–</td>
  </tr>
</table>

Table 10: Statistic of evaluation datasets used.

Reading Comprehension needs the model to infer whether the given passage can answer the query (Clark et al., 2019).

D Language Codes

We provide details of languages involved in Table 11. Following Lai et al. (2023), languages are divided by the data ratios in CommonCrawl: High (>1%), Medium (>0.1%), and Low (>0.01%).

<table>
  <tr>
    <th>ISO 639-1</th>
    <th>Language</th>
    <th>Family</th>
  </tr>
  <tr>
    <td>AR</td>
    <td>Arabic</td>
    <td>Afro-Asiatic</td>
  </tr>
  <tr>
    <td>BN</td>
    <td>Bengali</td>
    <td>Indo-European</td>
  </tr>
  <tr>
    <td>DE</td>
    <td>German</td>
    <td>Indo-European</td>
  </tr>
  <tr>
    <td>EN</td>
    <td>English</td>
    <td>Indo-European</td>
  </tr>
  <tr>
    <td>JA</td>
    <td>Japanese</td>
    <td>Japonic</td>
  </tr>
  <tr>
    <td>KO</td>
    <td>Korean</td>
    <td>Koreanic</td>
  </tr>
  <tr>
    <td>TA</td>
    <td>Tamil</td>
    <td>Dravidian</td>
  </tr>
  <tr>
    <td>TE</td>
    <td>Telugu</td>
    <td>Dravidian</td>
  </tr>
  <tr>
    <td>TH</td>
    <td>Thai</td>
    <td>Kra-Dai</td>
  </tr>
  <tr>
    <td>UR</td>
    <td>Urdu</td>
    <td>Indo-European</td>
  </tr>
  <tr>
    <td>VI</td>
    <td>Vietnamese</td>
    <td>Austroasiatic</td>
  </tr>
  <tr>
    <td>ZH</td>
    <td>Chinese</td>
    <td>Sino-Tibetan</td>
  </tr>
</table>

Table 11: Details of language codes in this work.

<table>
  <tr>
    <th>Name</th>
    <th>License</th>
  </tr>
  <tr>
    <td>Transformers</td>
    <td>Apache 2.0 license</td>
  </tr>
  <tr>
    <td>lm-evaluation-harness</td>
    <td>MIT license</td>
  </tr>
  <tr>
    <td>matplotlib</td>
    <td>PSF license</td>
  </tr>
  <tr>
    <td>Focus</td>
    <td>MIT license</td>
  </tr>
  <tr>
    <td>WECHSEL</td>
    <td>MIT license</td>
  </tr>
  <tr>
    <td>Pythia</td>
    <td>Apache 2.0 license</td>
  </tr>
  <tr>
    <td>LLaMA3</td>
    <td>Meta LLaMA 3 community license</td>
  </tr>
  <tr>
    <td>Qwen2</td>
    <td>Tongyi Qianwen license</td>
  </tr>
  <tr>
    <td>Gemma</td>
    <td>Gemma license</td>
  </tr>
  <tr>
    <td>The Pile</td>
    <td>MIT license</td>
  </tr>
</table>

Table 12: Licenses of scientific artifacts involved in this work.

E Licenses of Scientific Artifacts

We follow and report the licenses of scientific artifacts involved in Table 12.

Reviewer paper summaries: The paper develops a novel, efficient vocabulary adaptation algorithm. First, the original vocabulary is mapped to the target vocabulary by learning a one-to-one mapping matrix in an unsupervised manner. Glove is used to learn the representation of these tokens. Token IDs are aligned based on cosine similarity. BLUE-1 and BERTScore are used to evaluate the alignment matrix. TokAlign improves compression rates across languages. Extensive experimentation proves the generalizability of TokAlign’s improved performance.

The paper introduces an innovative method to align token IDs across different vocabularies in large language models (LLMs), aiming to optimize tokenization efficiency, particularly for multilingual tasks. By leveraging token co-occurrence statistics, the proposed TokAlign method achieves notable improvements in compression rates, reduces model perplexity, and enhances multilingual text processing capabilities. Furthermore, the method enables token-level knowledge transfer across models, broadening its potential applications.
Summary of strengths: 1. The paper tackles an important research problem of efficient vocabulary adaptation by aligning original vocabulary with target vocabulary by learning a token-level alignment matrix.

2. Extensive experimentation across a variety of metrics (perplexity score, downstream task performance) over a good number of datasets is conducted. Thus, the performance improvement of TokAlign can be considered generalizable.

3. The paper is well-written and is easy to understand.

1. The paper is well-organized and easy to follow.

2. It presents a novel perspective by proposing a token-token co-occurrence-based approach to replace the tokenizer in large language models.

3. The method demonstrates promising potential in both cross-lingual scenarios and knowledge transfer tasks, which could be highly impactful for multilingual model alignment and adaptation.
Summary of weaknesses: 1. In some cases, like in Table 4, the performance improvement is marginal and may need statistical significance test results to demonstrate meaningful gains.

2. The extensive experimentation does not include qualitative results and human evaluation studies. This will further improve the paper because although there is overall improvement, there may be cases where the performance drops. Identifying these failure modes will significantly improve the significance of this work. 

3. The authors commit to making the codebase publicly available but do not provide it during the review process for scrutiny.

The method seems to heavily rely on GloVe training over the pretraining corpus of the source model and the target model. This dependency could limit its general applicability, particularly when the original pretraining data is unavailable.
Comments, suggestions and typos: In Tables 1 and 2, please provide the full form of LAT as it is not introduced before in the text. Also, for all tables, please add the inference you want the readers to draw as the concluding line of the caption.

Recently, vocabulary adaptation has also been used for medical text summarization, i.e., instead of being two different languages, it adapts the open-domain-pretrained model vocabulary to an expert domain such as medicine. You may cite the following paper in the related works section: "Balde et al. 2024, “MEDVOC: Vocabulary Adaptation for fine-tuning pretrained language models on medical text summarization” [IJCAI 2024] https://doi.org/10.24963/ijcai.2024/683"

N/A
Overall assessment scores: 4

4