Model-Aware Tokenizer Transfer

## Abstract

Abstract Large Language Models (LLMs) are trained to support an increasing number of languages, yet their predefined tokenizers remain a bottleneck for adapting models to lower-resource or distinct-script languages. Existing tokenizer transfer methods typically rely on semantic heuristics to initialize new embeddings, ignoring higher-layer model dynamics and limiting transfer quality. We propose Model-Aware Tokenizer Transfer (MATT), a method that incorporates model internals into the tokenizer transfer process. MATT introduces an Attention Influence Modeling (AIM) objective that distills inter-token communication patterns from a source model into a target model with a new tokenizer, providing an efficient warm-up before standard language modeling. Unlike approaches that focus solely on embedding similarity, MATT leverages attention behavior to guide embedding initialization and adaptation. Experiments across diverse linguistic settings show that MATT recovers a large fraction of the original model’s performance within a few GPU hours, outperforming heuristic baselines. These results demonstrate that incorporating model-level signals offers a practical and effective path toward robust tokenizer transfer in multilingual LLMs.

## 1 Introduction

Recent advances in large language models (LLMs) have shifted attention from training monolingual models (Jiang et al., 2023; Touvron et al., 2023) to covering an increasing number of languages (Grattafiori et al., 2024; Team et al., 2025). Such multilingual models have become valuable tools for researchers and practitioners working with lower-resource languages. They can be used directly for downstream tasks, help translate English datasets into the target language (Rybak, 2023), or act as a robust baseline for further adaptation (Ociepa et al., 2024).
Our work focuses on the last scenario: adapting an existing LLM to a new language.

A major practical challenge in this setting is that every pretrained model is tied to a fixed tokenizer. Alternative architectures that avoid a predefined vocabulary, such as the Byte-Latent Transformer (Pagnoni et al., 2025) or H-Net (Hwang et al., 2025), are still in the experimental stage and not yet widely adopted. Tokenizers for multilingual models are usually trained to cover many scripts at once and inevitably favor high-resource languages. As a result, lower-resource languages, especially those with distinct alphabets such as Georgian, often receive a very limited share of the vocabulary. This mismatch leads not only to lower accuracy (Ali et al., 2024; Tamang & Bora, 2024), but also to slower processing and inference, which are vital for the end users.

One practical way to mitigate this problem is tokenizer transfer: replacing the original tokenizer of a pretrained model with a new one tailored to the target language and retraining the input and output embeddings (de Vries & Nissim, 2020). Even models not explicitly trained for multilinguality usually contain some cross-lingual knowledge thanks to shared alphabets or accidental language contamination (Blevins & Zettlemoyer, 2022). Consequently, if we can initialize the new embeddings well, much of the original performance can be recovered and used as a strong starting point for continual pretraining. At the same time, we should not expect this process to introduce entirely new linguistic knowledge, since several studies show that most of the model’s knowledge is stored in the feed-forward layers (Dai et al., 2022; Geva et al., 2021; Nichani et al., 2024).

Most existing tokenizer-transfer methods focus almost exclusively on the embedding layer. They construct new embeddings as linear combinations of the original ones, differing mainly in how the combination weights are computed (Minixhofer et al., 2022; Dobler & de Melo, 2023; Remy et al., 2023; 2024; Li et al., 2025). By ignoring the higher layers, these approaches overlook how the model actually processes tokens. More recent work, such as Zero-Shot Tokenizer Transfer by Minixhofer et al. (2024), leverages the full model by training a hypernetwork with a language modeling objective to predict embeddings. While effective, this strategy is computationally demanding because language modeling requires full forward and backward passes through the model.

To overcome these limitations, we introduce Model-Aware Tokenizer Transfer (MATT), a method that leverages the internal behavior of the pretrained model rather than relying only on surface semantics. At the core of MATT is Attention Influence Modeling (AIM) objective.

AIM encourages the model with the new tokenizer to reproduce the inter-token interactions of the original model’s attention layers. In effect, the original model acts as a teacher, while the model with the new tokenizer serves as a student that learns to match its attention patterns. This procedure distills structural knowledge about token relationships directly from the teacher, providing a richer and more informative initialization than relying on an embedding layer alone.

MATT is orthogonal to existing heuristics based on semantic similarity and can be combined with them. It acts as an efficient warm-up stage before conventional language model pretraining, reducing the cost of adaptation while preserving model quality.

We evaluate MATT by transferring the tokenizers of Gemma 3 (Team et al., 2025) and Qwen 3 (Team, 2025) models to extended versions that increase compression and expand coverage for several languages, including English, German, Japanese, Arabic, Swahili, and Ukrainian. Across multiple settings, MATT consistently recovers a substantial portion of the original model’s performance on both generative and discriminative tasks, while requiring only a few GPU hours and outperforming heuristic-based transfer methods.

Our contributions can be summarized as follows:

- •
Attention Influence Modeling (AIM): a novel distillation objective that aligns the attention dynamics of two models with different tokenizers.
- •
Model-Aware Tokenizer Transfer (MATT): an efficient tokenizer-transfer method that exploits model dynamics instead of relying solely on semantic relationships, achieving state-of-the-art results with substantially lower computational cost than language modeling objectives.
- •
Comprehensive evaluation: experiments across multiple languages and models demonstrating the effectiveness and efficiency of MATT.

## 2 Related Work

##### Large Language Models and Vocabulary Size

Large Language Models are becoming increasingly multilingual. Early open-source models focused almost exclusively on English (Jiang et al., 2023; Touvron et al., 2023; Almazrouei et al., 2023), but most recent releases include at least several languages and offer partial support for many more. This shift toward multilinguality has changed how researchers choose vocabulary size.

Studies show that larger vocabularies can improve model quality (Takase et al., 2025; Liang et al., 2023), but they also slow training and inference. As a result, most current foundation models use vocabularies of about 100 to 250 thousand tokens, with strongly multilingual models leaning toward the upper end. This sweet spot, first popularized by XLM-RoBERTa (Conneau et al., 2020), continues in more recent models such as Gemma (Team et al., 2025), Aya Expanse (Dang et al., 2024), and even GPT-5(^1^11[https://github.com/openai/tiktoken](https://github.com/openai/tiktoken)). Going beyond this range rarely pays off: performance gains are small, and efficiency drops sharply. As a result, tokenizers cannot achieve an optimal compression rate for every language, creating a need for techniques that allow efficient transfer of tokenizers to specific languages or domains without requiring very large vocabularies.

##### Heuristics-Based Embedding Initialization Methods

When transferring a tokenizer to a new language or domain, the main challenge is initializing embeddings for tokens that did not exist in the original model. Early work on tokenizer transfer (Artetxe et al., 2020; Gogoulou et al., 2022; de Vries & Nissim, 2020) focused on proving that transfer was possible, so embedding initialization received little attention. Simple strategies were used, including random initialization, taking the mean of existing embeddings, sampling from their distribution, copying the embedding of a random token, or using token frequency as a guide.

Later research began to exploit semantic relationships between tokens. WECHSEL (Minixhofer et al., 2022) was an influential step: it trained FastText (Bojanowski et al., 2017) embeddings for the source and target languages and used a translation vocabulary to identify the closest source tokens for each new token. New embeddings were then initialized as weighted averages of these source embeddings.
Several methods followed a similar direction. OFA (Liu et al., 2024) and Tik-to-Tok (Remy et al., 2023) refined the idea of using cross-lingual similarities, while Transtokenization (Remy et al., 2024) created its own token-level translation dictionary with FastAlign (Dyer et al., 2013). Hyper-OFA (Özeren et al., 2025) went further by training a hypernetwork to map tokens from an external multilingual space into the model’s embedding space, avoiding the need for simplistic linear combinations. TokAlign (Li et al., 2025) took a co-occurrence perspective, training two GloVe (Pennington et al., 2014) models on the same corpus to learn a one-to-one alignment matrix between tokens.

As LLMs became more multilingual, overlap between source and target vocabularies became an important resource. FOCUS (Dobler & de Melo, 2023) trains a FastText model on text tokenized with the target vocabulary, then initializes new embeddings as similarity-weighted averages of overlapping tokens. CLP Transfer (Ostendorff & Rehm, 2023) takes advantage of topological similarities of the latent space across model sizes within the same family: embeddings are first trained on a smaller related model and then aligned to the target model by measuring similarities with overlapping tokens.

##### Beyond Heuristics

While heuristics provide a practical starting point, they have limitations. An alternative is to train new embeddings directly by continuing language modeling with all other parameters frozen (de Vries & Nissim, 2020), but this is computationally costly.

Mini-Model Adaptation (Marchisio et al., 2023) reduces the cost by using only a subset of layers and training the embeddings on a language modeling task. Other work (Abagyan et al., 2025) shows that periodically resetting embeddings during pretraining makes models more robust to them, reducing the effort needed to learn new tokens afterwards.

Another approach by Minixhofer et al. (2024) trains a universal hypernetwork for a given language model by sampling tokenizers from a diverse distribution during the language modeling stage. Once the hypernetwork is trained, we can initialize embeddings for various tokenizers effortlessly, achieving a solid baseline for further continual pretraining. However, training such a hypernetwork is a compute-heavy task, requiring forward and backward passes through the whole model in every step to update the hypernetwork weights, limiting its practicality in settings where we already have a defined target tokenizer and the trained hypernetwork is not available beforehand.

## 3 Method

### 3.1 Intuition

Large Language Models generate text one token at a time. Decoder-only transformers, which form the backbone of most modern LLMs, follow the following steps: the embedding of the most recently generated token is passed through a stack of attention and feed-forward layers, and finally projected by the LM head to produce a probability distribution over the next token.

Assuming the input embedding of the last token is correct, the feed-forward layers will not damage its representation. The main source of potential distortion lies in the attention layers, where each token interacts with the context. Changing the tokenizer introduces new tokens into the context, altering these interactions and thus the internal representations that drive next-token prediction. Our goal is to train a model using a new tokenizer so that, despite these changes, its attention layers produce output embeddings similar to those generated by the original tokenizer.

### 3.2 Prerequisites

Consider an input string $s$ and a tokenization function $T$, which produces a token sequence $T(s)=(t_{1},t_{2},\dots,t_{n})$ of length $n$.

In each attention layer, the inputs are the query (${\bm{Q}}$), key (${\bm{K}}$), and value (${\bm{V}}$) state matrices, producing the output state matrix (${\bm{O}}$). Each of these can be seen as a collection of vector states for every token $t_{i}$:

$$ $\begin{array}[]{cccc}{\bm{Q}}=\left[\begin{array}[]{c}{\bm{q}}_{1}\\ {\bm{q}}_{2}\\ \dots\\ {\bm{q}}_{n}\end{array}\right]_{n\times h}&{\bm{K}}=\left[\begin{array}[]{c}{\bm{k}}_{1}\\ {\bm{k}}_{2}\\ \dots\\ {\bm{k}}_{n}\end{array}\right]_{n\times h}&{\bm{V}}=\left[\begin{array}[]{c}{\bm{v}}_{1}\\ {\bm{v}}_{2}\\ \dots\\ {\bm{v}}_{n}\end{array}\right]_{n\times h}&{\bm{O}}=\left[\begin{array}[]{c}{\bm{o}}_{1}\\ {\bm{o}}_{2}\\ \dots\\ {\bm{o}}_{n}\end{array}\right]_{n\times h}\end{array},$ $$

where $h$ is the hidden size.

Attention is computed as:

$$ ${\bm{O}}=\text{Attention}({\bm{Q}},{\bm{K}},{\bm{V}})=\text{softmax}\left(\frac{{\bm{Q}}{\bm{K}}^{T}}{\sqrt{d_{k}}}\right){\bm{V}}={\bm{A}}{\bm{V}},$ $$

where ${\bm{A}}$ is the attention matrix of shape $n\times n$, that contains weights with which the value states are aggregated into the output state.

We can break down the final matrix multiplication ${\bm{A}}{\bm{V}}$ into a chain of value states (${\bm{V}}$) averages for each token, weighted by the attention matrix ${\bm{A}}$. The output state for the token $t_{i}$ would then look the following way:

$$ ${\bm{o}}_{i}=\text{softmax}\left(\frac{{\bm{q}}_{i}{\bm{K}}^{T}}{\sqrt{d_{k}}}\right){\bm{V}}={\bm{A}}_{i,:}{\bm{V}}=\sum_{j=1}^{n}{\bm{A}}_{i,j}{\bm{v}}_{j}=\sum_{j=1}^{n}{\bm{v}}^{*}_{i,j},$ $$

where ${\bm{v}}^{*}_{i,j}={\bm{A}}_{i,j}{\bm{v}}_{j}$ is a weighted value state for the token $t_{j}$ given the query token $t_{i}$.

### 3.3 Segment-Level Interpretation of Attention

To compare attention outputs across different tokenizers, we introduce a segmentation function $S$ that splits the input string $s$ into segments $(s_{1},s_{2},\dots,s_{m})$ while respecting a set of tokenization functions
$\textstyle\tau$
:

$$ $S(s;\,\mathchoice{\raisebox{0.0pt}{\resizebox{6.93805pt}{6.83331pt}{\hbox{\raisebox{0.0pt}{$\displaystyle\tau$}}}}}{\raisebox{0.0pt}{\resizebox{6.93805pt}{6.83331pt}{\hbox{\raisebox{0.0pt}{$\textstyle\tau$}}}}}{\raisebox{0.0pt}{\resizebox{5.85301pt}{4.78334pt}{\hbox{\raisebox{0.0pt}{$\scriptstyle\tau$}}}}}{\raisebox{0.0pt}{\resizebox{5.4089pt}{3.41667pt}{\hbox{\raisebox{0.0pt}{$\scriptscriptstyle\tau$}}}}})=(s_{1},s_{2},\dots,s_{m}),\,\text{such that}$ $$

$$ $\forall\,T\in\mathchoice{\raisebox{0.0pt}{\resizebox{6.93805pt}{6.83331pt}{\hbox{\raisebox{0.0pt}{$\displaystyle\tau$}}}}}{\raisebox{0.0pt}{\resizebox{6.93805pt}{6.83331pt}{\hbox{\raisebox{0.0pt}{$\textstyle\tau$}}}}}{\raisebox{0.0pt}{\resizebox{5.85301pt}{4.78334pt}{\hbox{\raisebox{0.0pt}{$\scriptstyle\tau$}}}}}{\raisebox{0.0pt}{\resizebox{5.4089pt}{3.41667pt}{\hbox{\raisebox{0.0pt}{$\scriptscriptstyle\tau$}}}}}:\,T(s_{1})\circ T(s_{2})\circ\cdots\circ T(s_{m})=T(s),$ $$

where $\circ$ is a concatenation operator. This ensures that no segment boundary lies within any token produced by any tokenization function in
$\textstyle\tau$
.

The most intuitive approach is a function that splits the input string into words, and the rest of the section is explained in relation to this function. However, for some languages, word segmentation can be ambiguous; thus, in practice, we define our segmentation function to always choose segments of minimal length that still satisfy the above condition (see Appendix [A](https://arxiv.org/html/2510.21954v1#A1) for the algorithm).

Given $S$, we define weighted value states for a segment $s_{k}$ with respect to a query token $t_{i}$:

$$ $\bm{\mathfrak{s}}_{i,k}=\sum_{\{j\,:\,t_{j}\in T(s_{k})\}}{\bm{v}}^{*}_{i,j}$ $$

The output state for token $t_{i}$ can then be expressed as a sum over segments:

$$ ${\bm{o}}_{i}=\sum_{j=1}^{n}{\bm{v}}^{*}_{i,j}=\sum_{j=1}^{m}\bm{\mathfrak{s}}_{i,j}$ $$

To move from token-level to segment-level interpretation, we replace individual query tokens with segment representations.
Since the output state of each token is designed to predict the next token, it is natural to require that the output state of a segment should similarly carry enough information to predict the next segment.
Because the language modeling head still operates at the token level, we approximate ”predicting the next segment” by predicting the first token of that next segment.

Consider a segment $s_{i}$ whose tokens are $T(s_{i})=(t_{a},t_{a+1},\dots,t_{b})$. The final token $t_{b}$ produces the output state used to generate the next token $t_{b+1}$, which begins the following segment $s_{i+1}$. We therefore define a function $\ell_{T}$ that maps a segment index to the index of its last token:

$$ $\ell_{T}(i)=b$ $$

The query state of segment $s_{i}$ is set equal to the query state of its last token:

$$ $\bm{\mathfrak{q}}_{i}={\bm{q}}_{\ell_{T}(i)}={\bm{q}}_{b},$ $$

and the output state of the segment is computed from this query state:

$$ $\bm{\mathfrak{o}}_{i}=\text{softmax}\left(\frac{\bm{\mathfrak{q}}_{i}{\bm{K}}^{T}}{\sqrt{d_{k}}}\right){\bm{V}}=\text{softmax}\left(\frac{{\bm{q}}_{\ell_{T}(i)}{\bm{K}}^{T}}{\sqrt{d_{k}}}\right){\bm{V}}={\bm{o}}_{\ell_{T}(i)}=\sum_{j=1}^{m}\bm{\mathfrak{s}}_{\ell_{T}(i),j}$ $$

### 3.4 Attention Influence Modeling

As described in the Section [3.1](https://arxiv.org/html/2510.21954v1#S3.SS1), our goal is to train the model with a new tokenizer $T^{\prime}$ so that its output states match those of the original model with tokenizer $T$.

Since we can enforce a common segmentation function $S$, we approximate this by requiring the new model to produce the same segment-level outputs $\bm{\mathfrak{o}}^{\prime}_{i}$ as the old ones – $\bm{\mathfrak{o}}_{i}$. A more detailed objective also matches the weighted value states $\bm{\mathfrak{s}}_{\ell_{T}(i),j}$ and $\bm{\mathfrak{s}}^{\prime}_{\ell_{T^{\prime}}(i),j}$ of every segment $s_{j}$ for each query segment $s_{i}$, with the causal constraint $j\leq i$.

Given the above, we define the Attention Influence Modeling objectives (normal and simplified):

$$ $\mathcal{L}_{AIM}=\frac{2}{m(m+1)}\sum_{i=1}^{m}\sum_{j=1}^{i}\mathcal{L}^{*}(\bm{\mathfrak{s}}_{\ell_{T}(i),j},\bm{\mathfrak{s}}^{\prime}_{\ell_{T^{\prime}}(i),j}),$ $$

$$ $\mathcal{L}_{AIM^{*}}=\frac{1}{m}\sum_{i=1}^{m}\mathcal{L}^{*}(\bm{\mathfrak{o}}_{i},\bm{\mathfrak{o}}^{\prime}_{i}),$ $$

where $\mathcal{L}^{*}({\bm{x}},{\bm{y}})$ can be any loss function, that brings ${\bm{x}}$ and ${\bm{y}}$ closer. In Section [4](https://arxiv.org/html/2510.21954v1#S4), we experiment with MSE and Cosine Embedding losses.

Figure: Figure 1: Attention Influence Modeling (AIM) objective with word segmentation. For each input, the weighted value vectors ${\bm{v}}^{*}_{i,j}$ of the original tokens $t_{j}$ are aggregated into segment-level vectors $\bm{\mathfrak{s}}_{i,k}$ according to a chosen word-segmentation function. The model trained with the new tokenizer produces its own segment representations $\bm{\mathfrak{s}}^{\prime}_{i,k}$. The AIM objective encourages these new segment representations to stay close to the segment representations $\bm{\mathfrak{s}}_{i,k}$ computed from the model using the old tokenizer. All this happens with respect to the query state $\bm{\mathfrak{q}}_{5}$ of the 5th segment (_methane), which is equal to the query state of its last token – hane.
Refer to caption: x1.png

Figure [1](https://arxiv.org/html/2510.21954v1#S3.F1) illustrates an example of applying AIM to the text CH4 – formula for methane. In this case, we use a word segmentation function together with different tokenization functions for a given query state $\bm{\mathfrak{q}}_{5}$, where the segment $s_{5}$ corresponds to _methane.

Figure: Figure 2: Token-level attention alignment between teacher and student models. The left matrix shows the weighted value states of the teacher model using the original tokenizer $T$, and the right matrix shows those of the student model using the new tokenizer $T^{\prime}$. Each square represents the weighted value state ${\bm{v}}^{*}_{i,j}$ of $t_{j}$ for query token $t_{i}$ ($i$ for rows, $j$ for columns). Numbers (or matching colors) within a matrix identify tokens that are aggregated into the same segment-level state $\bm{\mathfrak{s}}_{i,k}$. Numbers (or matching colors) across the two matrices indicate corresponding pairs $\bm{\mathfrak{s}}_{\ell_{T}(i),j}$ and $\bm{\mathfrak{s}}^{\prime}_{\ell_{T^{\prime}}(i),j}$ used in the loss $\mathcal{L}^{*}$ to align the teacher and the student attention representations.
Refer to caption: x2.png

Figure [2](https://arxiv.org/html/2510.21954v1#S3.F2) presents the attention alignment matrix for the same text, where the weighted value states ${\bm{v}}^{*}_{i,j}$ are grouped into segments. These segment-level representations are then matched and optimized to be equal under the $\mathcal{L}_{AIM}$ loss.

### 3.5 Technical Details

During training, the model with the old tokenizer $T$ is kept frozen. The model with the new tokenizer $T^{\prime}$ has all layers frozen except the input embeddings. As a small modification to the basic training setup, we partially freeze the embedding matrix: tokens that are shared between the old and new tokenizers are initialized from the original model and kept fixed, while only the embeddings of new, non-overlapping tokens are updated during training.

To speed up convergence, we initialize new embeddings using FOCUS (Dobler & de Melo, 2023). We train with AdamW (Loshchilov & Hutter, 2017), a constant learning rate of $1\times 10^{-4}$, and no weight decay. However, it should be noted that we have not performed extensive hyperparameter tuning, so using learning rate scheduling, adapting the learning rate, weight decay, and other hyperparameters may yield significantly better results.

MATT offers a key advantage over standard language modeling with frozen non-embedding parameters: greater efficiency.
Since AIM is defined at the attention-layer level, we can decide how much of the model to include in the tokenizer transfer by selecting the layer depth at which AIM is applied. Specifically, by choosing a value of $n$, we take only the first $n$ layers into account. This allows us to balance efficiency and performance. An ablation study examining how $n$ affects training speed and final model quality is presented in Appendix [C](https://arxiv.org/html/2510.21954v1#A3).

Since only input embeddings are trained, tied input–output embeddings are advantageous, as the tuned input embeddings can be reused in the LM head. Models without tied embeddings still benefit from input tuning, but to a significantly lesser extent; handling untied settings is left for future work.

## 4 Experiments

We conducted a series of experiments across different languages, model families, and scales to evaluate the effectiveness of the MATT method compared to existing heuristic-based approaches. In each experiment, we first trained a tokenizer with a higher compression rate than the original one, merged it with the base tokenizer, and then applied tokenizer transfer to the extended vocabulary.

Additional experiments, including convergence speed tests (Appendix [B](https://arxiv.org/html/2510.21954v1#A2)) and ablation studies (Appendix [C](https://arxiv.org/html/2510.21954v1#A3)), are presented to complement the main results.

### 4.1 Main Results

Our primary evaluation uses the Gemma 3 12B model – both the pretrained (PT) and instruction-tuned (IT) variants (Team et al., 2025), which share the same tokenizer. We replaced their default tokenizer with an extended version that improves Ukrainian coverage, raising the compression rate from 2.98 to 4.44. This increase translates to an almost 50% speedup during inference.

We compare the following methods:

- •
WECHSEL – transfer using the English–Ukrainian vocabulary from the official implementation(^2^22[https://github.com/CPJKU/wechsel](https://github.com/CPJKU/wechsel)).
- •
Transtokenizers – token alignment via FastAlign using parallel corpora (OpenSubtitles (Lison & Tiedemann, 2016) and NLLB (NLLB Team, 2022)) and the official transtokenizers(^3^33[https://github.com/LAGoM-NLP/transtokenizer](https://github.com/LAGoM-NLP/transtokenizer)) toolkit.
- •
TokAlign – GloVe embeddings trained on 2 million Ukrainian documents (approximately 1.86 billion Gemma tokens) from the Kobza corpus (Haltiuk & Smywiński-Pohl, 2025), used to create a one-to-one alignment matrix with the official implementation(^4^44[https://github.com/ZNLP/TokAlign](https://github.com/ZNLP/TokAlign)).
- •
FOCUS – FastText embeddings trained on the same data as TokAlign, with initialization performed via the deepfocus(^5^55[https://github.com/konstantinjdobler/focus](https://github.com/konstantinjdobler/focus)) package.
- •
MATT – initialized with FOCUS embeddings and trained on around 240 million Ukrainian tokens from Kobza using the AIM objective with MSE loss on the 12th layer. Original embeddings are frozen, and all other hyperparameters remain unchanged (see Section [3.5](https://arxiv.org/html/2510.21954v1#S3.SS5)).
- •
MATT (cosine) – same setup as MATT but using cosine embedding loss instead of MSE.

We evaluate performance on Belebele (Bandarkar et al., 2024), Global MMLU (Singh et al., 2025), and Long FLORES(^6^66[https://huggingface.co/datasets/robinhad/long_flores](https://huggingface.co/datasets/robinhad/long_flores)), a modification of FLORES (NLLB Team, 2022; Goyal et al., 2021; Guzmán et al., 2019), which elevates the sentence-level translation to document-level by aggregating data points from the same sources. We only evaluate the translations from English to Ukrainian with a specific intent to validate the model’s performance on a generative task in the target language. Evaluation is performed with the lm-evaluation-harness framework (Gao et al., 2024) with a 3-shot prompt. In the case of an instruction-tuned model, no chat template is used.

**Table 1: Performance of Gemma 3 12B models with different tokenizer transfer methods on Belebele and Global MMLU (accuracy, %), and Long FLORES (BLEU). The last column reports the average of the three metrics. $\cdot^{\dagger}$ denotes that the original embeddings of the overlapping tokens were directly copied into the marked model (FOCUS already involves this step in its implementation).**
| Model | Belebele | Global MMLU | Long FLORES | Avg |
| --- | --- | --- | --- | --- |
| Gemma 3 12B PT | 89.33 | 67.03 | 14.36 | 56.91 |
| WECHSEL | 22.67 | 24.61 | 0.00 | 15.76 |
| WECHSEL^† | 34.00 | 31.58 | 0.72 | 21.86 |
| Transtokenizers | 61.89 | 46.03 | 0.04 | 35.99 |
| Transtokenizers^† | 45.44 | 40.06 | 0.41 | 28.64 |
| TokAlign | 31.44 | 32.98 | 0.00 | 21.47 |
| TokAlign^† | 33.78 | 32.57 | 0.00 | 22.12 |
| FOCUS | 48.78 | 37.14 | 1.01 | 28.98 |
| MATT | 89.56 | 64.98 | 8.70 | 54.41 |
| Gemma 3 12B IT | 89.44 | 64.10 | 28.32 | 60.62 |
| MATT | 89.67 | 62.43 | 20.54 | 57.55 |
| w/ cosine | 89.67 | 62.59 | 16.23 | 56.16 |

Table [1](https://arxiv.org/html/2510.21954v1#S4.T1) shows a clear advantage of MATT over all other methods. While heuristic-based approaches such as FOCUS and Transtokenizers can regain up to about 50% of the original model’s accuracy on discriminative tasks, they reach no more than about 7% of the original performance on the generative translation benchmark. In contrast, MATT restores over 60% of the original generative performance while maintaining accuracy on discriminative tasks close to the unmodified model. These results indicate the superiority of the model-aware approach to tokenizer transfer, especially considering the extremely low computational costs required.

Interestingly, the results for the instruction-tuned model suggest that MSE loss may be more beneficial for generative tasks than cosine embedding loss, recovering over 70% of the original model’s performance.

### 4.2 Multilingual Results

In the multilingual setting, we experiment with Gemma 3 4B PT and Qwen 3 0.6B (Team, 2025). Their new tokenizers expand coverage for five typologically diverse languages – English, German, Japanese, Arabic, and Swahili, which vary in resource availability, writing system, and language family.

As shown in Table [2](https://arxiv.org/html/2510.21954v1#S4.T2), the extended tokenizers consistently improve compression rates across all languages, including modest gains for English.

**Table 2: Comparison of original and extended tokenizers. Compression rate is the average number of characters represented by a single token (higher is better).**
| Tokenizer | Vocabulary Size | Compression Rate |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  |  | ar | de | en | ja | sw |
| Gemma |  |  |  |  |  |  |
| original | 262,145 | 2.8457 | 3.9734 | 4.3187 | 1.6846 | 2.9802 |
| extended | 387,980 | 3.9122 | 4.4997 | 4.3383 | 2.1267 | 4.2518 |
| Qwen |  |  |  |  |  |  |
| original | 151,669 | 2.5982 | 3.4737 | 4.3599 | 1.4852 | 2.5788 |
| extended | 298,833 | 3.9221 | 4.4886 | 4.4233 | 2.2867 | 4.2322 |

The transfer methods remain the same as in Section [4.1](https://arxiv.org/html/2510.21954v1#S4.SS1), except the training data is now drawn from HPLT 2.0 Cleaned (Burchell et al., 2025). FOCUS uses 2 million documents (approximately 500 million tokens per language), and MATT – only about 50 million tokens per language. We also experiment with an AIM* objective, Cosine Embedding loss, and training without freezing the original embeddings.

Performance is reported on Belebele, MMMLU Hendrycks et al. (2020), and Global MMLU. We additionally record the time and memory required to tune embeddings on a single H100 GPU.

**Table 3: Benchmark results for transferring original tokenizers to their extended versions across five languages (Arabic, German, English, Japanese, Swahili). For the proposed MATT method, peak VRAM usage and processing time required for the tokenizer transfer are also reported.**
|  | VRAM | Time | Belebele | MMMLU | Global MMLU | Avg |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model | GiB |  | ar | de | en | ja | sw | ar | de | en | ja | sw | ar | de | en | ja | sw |  |
| Gemma 3 4B PT | - | - | 69.33 | 68.00 | 82.00 | 67.44 | 59.67 | 39.24 | 43.77 | 53.89 | 41.20 | 35.71 | 45.25 | 47.50 | 59.00 | 47.25 | 38.00 | 53.15 |
| FOCUS | - | - | 32.89 | 52.44 | 80.33 | 42.00 | 25.56 | 27.89 | 36.89 | 53.81 | 32.77 | 28.23 | 30.25 | 36.75 | 59.50 | 33.00 | 26.75 | 39.94 |
| MATT | 16.6 | 4h 47m | 62.44 | 72.56 | 80.67 | 58.00 | 54.67 | 37.05 | 44.79 | 53.82 | 38.68 | 36.10 | 40.50 | 47.50 | 60.25 | 40.00 | 39.50 | 51.10 |
| w/ AIM* | 10.9 | 3h 44m | 63.78 | 70.33 | 80.78 | 56.11 | 54.89 | 37.22 | 44.17 | 53.86 | 39.15 | 35.69 | 37.75 | 45.50 | 59.75 | 40.50 | 42.25 | 50.78 |
| w/ cosine | 15.7 | 5h 01m | 62.00 | 70.56 | 80.67 | 60.22 | 59.00 | 38.33 | 44.84 | 53.88 | 40.48 | 36.22 | 42.00 | 45.25 | 60.00 | 41.50 | 41.75 | 51.78 |
| w/ unfrozen | 20.4 | 5h 34m | 59.78 | 68.78 | 80.56 | 56.22 | 55.89 | 38.24 | 44.77 | 53.96 | 40.97 | 37.95 | 40.25 | 48.50 | 60.00 | 45.00 | 41.75 | 51.51 |
| Qwen3 0.6B | - | - | 50.78 | 59.33 | 64.11 | 58.67 | 29.89 | 36.08 | 40.12 | 47.21 | 37.54 | 28.68 | 38.25 | 42.75 | 50.25 | 44.25 | 28.25 | 43.74 |
| FOCUS | - | - | 27.11 | 32.22 | 60.33 | 29.22 | 24.89 | 28.76 | 30.28 | 41.85 | 29.53 | 26.58 | 26.00 | 31.25 | 45.25 | 28.00 | 24.50 | 32.38 |
| MATT | 9.4 | 3h 38m | 36.67 | 43.22 | 60.44 | 39.44 | 28.22 | 32.78 | 35.14 | 42.41 | 33.19 | 26.71 | 29.25 | 35.50 | 45.25 | 30.50 | 25.75 | 36.30 |
| w/ AIM* | 3.5 | 2h 45m | 40.56 | 45.89 | 60.44 | 42.11 | 28.44 | 33.15 | 35.71 | 42.29 | 33.72 | 27.19 | 32.50 | 37.00 | 45.50 | 37.50 | 22.75 | 37.65 |
| w/ cosine | 8.6 | 3h 52m | 39.89 | 46.67 | 60.33 | 43.22 | 28.89 | 32.77 | 35.76 | 42.28 | 33.41 | 27.38 | 33.50 | 35.75 | 45.50 | 35.25 | 24.50 | 37.67 |
| w/ unfrozen | 10.2 | 3h 38m | 42.22 | 47.56 | 62.11 | 42.89 | 27.89 | 33.02 | 36.46 | 43.91 | 33.75 | 27.50 | 31.50 | 38.00 | 46.25 | 35.50 | 24.00 | 38.17 |

Table [3](https://arxiv.org/html/2510.21954v1#S4.T3) shows that MATT substantially narrows the performance gap between a freshly initialized model and the original, often recovering most of the accuracy and occasionally surpassing the original. The AIM* variant offers a good compromise, reducing memory and runtime while only slightly lowering accuracy. Further VRAM savings are possible with a custom kernel for AIM computation.

## 5 Conclusion

In this work, we introduced MATT, a model-aware method for tokenizer transfer that leverages the internal dynamics of LLMs. We applied MATT to extend the tokenizers of Gemma 3 and Qwen 3 models across multiple languages and settings, demonstrating that it consistently recovers a large portion of the original model’s capabilities while requiring only a few GPU hours of training. Unlike heuristic-based methods that rely solely on the embedding layer, MATT refines token representations with direct feedback from the model, thanks to the novel Attention Influence Modeling (AIM) objective, allowing it to bridge the performance gap caused by tokenizer changes more effectively.

Our experiments highlight this advantage most clearly in the transfer of the 12 billion-parameter Gemma 3 model to Ukrainian. With the extended tokenizer, MATT achieves an average score of 54.41 out of the original 56.91, whereas the best heuristic approach reaches only 35.99.

This substantial improvement underscores the value of incorporating model dynamics into tokenizer transfer and shows that high performance can be retained at a fraction of the computational cost typically required for full retraining.

#### Limitations

The first limitation lies in the fact that MATT relies on tied input and output embeddings to fully realize its advantages. We outline possible strategies to relax this requirement in the Appendix [D](https://arxiv.org/html/2510.21954v1#A4).

Second, we do not perform continual pretraining due to computational constraints and instead evaluate only the initialized model. This is sufficient to compare MATT with existing baselines, whose primary goal is to provide a strong starting point for further adaptation.

Finally, we have not tested MATT on encoder-only architectures. In principle, applying it to such models would only require removing the causal constraint in the AIM definition.

## References

- Abagyan et al. (2025)
Diana Abagyan, Alejandro R Salamanca, Andres Felipe Cruz-Salinas, Kris Cao, Hangyu Lin, Acyr Locatelli, Marzieh Fadaee, Ahmet Üstün, and Sara Hooker.
One tokenizer to rule them all: Emergent language plasticity via multilingual tokenizers.
*arXiv preprint arXiv:2506.10766*, 2025.
- Ali et al. (2024)
Mehdi Ali, Michael Fromm, Klaudia Thellmann, Richard Rutmann, Max Lübbering, Johannes Leveling, Katrin Klug, Jan Ebert, Niclas Doll, Jasper Buschhoff, Charvi Jain, Alexander Weber, Lena Jurkschat, Hammam Abdelwahab, Chelsea John, Pedro Ortiz Suarez, Malte Ostendorff, Samuel Weinbach, Rafet Sifa, Stefan Kesselheim, and Nicolas Flores-Herr.
Tokenizer choice for LLM training: Negligible or crucial?
In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), *Findings of the Association for Computational Linguistics: NAACL 2024*, pp. 3907–3924, Mexico City, Mexico, June 2024. Association for Computational Linguistics.
doi: 10.18653/v1/2024.findings-naacl.247.
URL [https://aclanthology.org/2024.findings-naacl.247/](https://aclanthology.org/2024.findings-naacl.247/).
- Almazrouei et al. (2023)
Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, et al.
The falcon series of open language models.
*arXiv preprint arXiv:2311.16867*, 2023.
- Artetxe et al. (2020)
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama.
On the cross-lingual transferability of monolingual representations.
In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pp. 4623–4637, Online, July 2020. Association for Computational Linguistics.
doi: 10.18653/v1/2020.acl-main.421.
URL [https://aclanthology.org/2020.acl-main.421/](https://aclanthology.org/2020.acl-main.421/).
- Bandarkar et al. (2024)
Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa.
The belebele benchmark: a parallel reading comprehension dataset in 122 language variants.
In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 749–775, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
doi: 10.18653/v1/2024.acl-long.44.
URL [https://aclanthology.org/2024.acl-long.44/](https://aclanthology.org/2024.acl-long.44/).
- Blevins & Zettlemoyer (2022)
Terra Blevins and Luke Zettlemoyer.
Language contamination helps explains the cross-lingual capabilities of English pretrained models.
In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), *Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing*, pp. 3563–3574, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
doi: 10.18653/v1/2022.emnlp-main.233.
URL [https://aclanthology.org/2022.emnlp-main.233/](https://aclanthology.org/2022.emnlp-main.233/).
- Bohdan Didenko (2025)
Bohdan Didenko.
Post #9194 on Telegram Channel Zaduha.
[https://t.me/zaduha/9194](https://t.me/zaduha/9194), 2025.
[Online; accessed 26 June 2025].
- Bojanowski et al. (2017)
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov.
Enriching word vectors with subword information.
*Transactions of the association for computational linguistics*, 5:135–146, 2017.
- Burchell et al. (2025)
Laurie Burchell, Ona de Gibert, Nikolay Arefyev, Mikko Aulamo, Marta Bañón, Mariia Fedorova, Liane Guillou, Barry Haddow, Jan Hajič, Erik Henriksson, et al.
An expanded massive multilingual dataset for high-performance language technologies.
*arXiv preprint arXiv:2503.10267*, 2025.
- Conneau et al. (2020)
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov.
Unsupervised cross-lingual representation learning at scale.
In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pp. 8440–8451, Online, July 2020. Association for Computational Linguistics.
doi: 10.18653/v1/2020.acl-main.747.
URL [https://aclanthology.org/2020.acl-main.747](https://aclanthology.org/2020.acl-main.747).
- Dai et al. (2022)
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei.
Knowledge neurons in pretrained transformers.
In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), *Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 8493–8502, Dublin, Ireland, May 2022. Association for Computational Linguistics.
doi: 10.18653/v1/2022.acl-long.581.
URL [https://aclanthology.org/2022.acl-long.581/](https://aclanthology.org/2022.acl-long.581/).
- Dang et al. (2024)
John Dang, Shivalika Singh, Daniel D’souza, Arash Ahmadian, Alejandro Salamanca, Madeline Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terrence Zhao, et al.
Aya expanse: Combining research breakthroughs for a new multilingual frontier.
*arXiv preprint arXiv:2412.04261*, 2024.
- de Vries & Nissim (2020)
Wietse de Vries and Malvina Nissim.
As good as new. how to successfully recycle english gpt-2 to make models for other languages.
*arXiv preprint arXiv:2012.05628*, 2020.
- Dobler & de Melo (2023)
Konstantin Dobler and Gerard de Melo.
FOCUS: Effective embedding initialization for monolingual specialization of multilingual models.
In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), *Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pp. 13440–13454, Singapore, December 2023. Association for Computational Linguistics.
doi: 10.18653/v1/2023.emnlp-main.829.
URL [https://aclanthology.org/2023.emnlp-main.829/](https://aclanthology.org/2023.emnlp-main.829/).
- Dyer et al. (2013)
Chris Dyer, Victor Chahuneau, and Noah A Smith.
A simple, fast, and effective reparameterization of ibm model 2.
In *Proceedings of the 2013 conference of the North American chapter of the association for computational linguistics: human language technologies*, pp. 644–648, 2013.
- Gao et al. (2024)
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou.
The language model evaluation harness, 07 2024.
URL [https://zenodo.org/records/12608602](https://zenodo.org/records/12608602).
- Geva et al. (2021)
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy.
Transformer feed-forward layers are key-value memories, 2021.
URL [https://arxiv.org/abs/2012.14913](https://arxiv.org/abs/2012.14913).
- Gogoulou et al. (2022)
Evangelia Gogoulou, Ariel Ekgren, Tim Isbister, and Magnus Sahlgren.
Cross-lingual transfer of monolingual models.
In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Jan Odijk, and Stelios Piperidis (eds.), *Proceedings of the Thirteenth Language Resources and Evaluation Conference*, pp. 948–955, Marseille, France, June 2022. European Language Resources Association.
URL [https://aclanthology.org/2022.lrec-1.100/](https://aclanthology.org/2022.lrec-1.100/).
- Goyal et al. (2021)
Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzmán, and Angela Fan.
The flores-101 evaluation benchmark for low-resource and multilingual machine translation.
2021.
- Grattafiori et al. (2024)
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al.
The llama 3 herd of models.
*arXiv preprint arXiv:2407.21783*, 2024.
- Guzmán et al. (2019)
Francisco Guzmán, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, and Marc’Aurelio Ranzato.
Two new evaluation datasets for low-resource machine translation: Nepali-english and sinhala-english.
2019.
- Haltiuk & Smywiński-Pohl (2025)
Mykola Haltiuk and Aleksander Smywiński-Pohl.
On the path to make Ukrainian a high-resource language.
In Mariana Romanyshyn (ed.), *Proceedings of the Fourth Ukrainian Natural Language Processing Workshop (UNLP 2025)*, pp. 120–130, Vienna, Austria (online), July 2025. Association for Computational Linguistics.
ISBN 979-8-89176-269-5.
doi: 10.18653/v1/2025.unlp-1.14.
URL [https://aclanthology.org/2025.unlp-1.14/](https://aclanthology.org/2025.unlp-1.14/).
- Hendrycks et al. (2020)
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.
Measuring massive multitask language understanding.
*arXiv preprint arXiv:2009.03300*, 2020.
- Hwang et al. (2025)
Sukjun Hwang, Brandon Wang, and Albert Gu.
Dynamic chunking for end-to-end hierarchical sequence modeling.
*arXiv preprint arXiv:2507.07955*, 2025.
- Jiang et al. (2023)
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed.
Mistral 7b, 2023.
URL [https://arxiv.org/abs/2310.06825](https://arxiv.org/abs/2310.06825).
- Li et al. (2025)
Chong Li, Jiajun Zhang, and Chengqing Zong.
TokAlign: Efficient vocabulary adaptation via token alignment.
In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), *Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 4109–4126, Vienna, Austria, July 2025. Association for Computational Linguistics.
ISBN 979-8-89176-251-0.
doi: 10.18653/v1/2025.acl-long.207.
URL [https://aclanthology.org/2025.acl-long.207/](https://aclanthology.org/2025.acl-long.207/).
- Liang et al. (2023)
Davis Liang, Hila Gonen, Yuning Mao, Rui Hou, Naman Goyal, Marjan Ghazvininejad, Luke Zettlemoyer, and Madian Khabsa.
XLM-V: Overcoming the vocabulary bottleneck in multilingual masked language models.
In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), *Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pp. 13142–13152, Singapore, December 2023. Association for Computational Linguistics.
doi: 10.18653/v1/2023.emnlp-main.813.
URL [https://aclanthology.org/2023.emnlp-main.813/](https://aclanthology.org/2023.emnlp-main.813/).
- Lison & Tiedemann (2016)
Pierre Lison and Jörg Tiedemann.
OpenSubtitles2016: Extracting large parallel corpora from movie and TV subtitles.
In Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Sara Goggi, Marko Grobelnik, Bente Maegaard, Joseph Mariani, Helene Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (eds.), *Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC‘16)*, pp. 923–929, Portorož, Slovenia, May 2016. European Language Resources Association (ELRA).
URL [https://aclanthology.org/L16-1147/](https://aclanthology.org/L16-1147/).
- Liu et al. (2024)
Yihong Liu, Peiqin Lin, Mingyang Wang, and Hinrich Schuetze.
OFA: A framework of initializing unseen subword embeddings for efficient large-scale multilingual continued pretraining.
In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), *Findings of the Association for Computational Linguistics: NAACL 2024*, pp. 1067–1097, Mexico City, Mexico, June 2024. Association for Computational Linguistics.
doi: 10.18653/v1/2024.findings-naacl.68.
URL [https://aclanthology.org/2024.findings-naacl.68/](https://aclanthology.org/2024.findings-naacl.68/).
- Loshchilov & Hutter (2017)
Ilya Loshchilov and Frank Hutter.
Fixing weight decay regularization in adam.
*CoRR*, abs/1711.05101, 2017.
URL [http://arxiv.org/abs/1711.05101](http://arxiv.org/abs/1711.05101).
- Marchisio et al. (2023)
Kelly Marchisio, Patrick Lewis, Yihong Chen, and Mikel Artetxe.
Mini-model adaptation: Efficiently extending pretrained models to new languages via aligned shallow training.
In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), *Findings of the Association for Computational Linguistics: ACL 2023*, pp. 5474–5490, Toronto, Canada, July 2023. Association for Computational Linguistics.
doi: 10.18653/v1/2023.findings-acl.338.
URL [https://aclanthology.org/2023.findings-acl.338/](https://aclanthology.org/2023.findings-acl.338/).
- Minixhofer et al. (2022)
Benjamin Minixhofer, Fabian Paischer, and Navid Rekabsaz.
WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models.
In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), *Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pp. 3992–4006, Seattle, United States, July 2022. Association for Computational Linguistics.
doi: 10.18653/v1/2022.naacl-main.293.
URL [https://aclanthology.org/2022.naacl-main.293](https://aclanthology.org/2022.naacl-main.293).
- Minixhofer et al. (2024)
Benjamin Minixhofer, Edoardo M. Ponti, and Ivan Vulić.
Zero-shot tokenizer transfer.
In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), *Advances in Neural Information Processing Systems*, volume 37, pp. 46791–46818. Curran Associates, Inc., 2024.
URL [https://proceedings.neurips.cc/paper_files/paper/2024/file/532ce4fcf853023c4cf2ac38cbc5d002-Paper-Conference.pdf](https://proceedings.neurips.cc/paper_files/paper/2024/file/532ce4fcf853023c4cf2ac38cbc5d002-Paper-Conference.pdf).
- Nichani et al. (2024)
Eshaan Nichani, Jason D. Lee, and Alberto Bietti.
Understanding factual recall in transformers via associative memories, 2024.
URL [https://arxiv.org/abs/2412.06538](https://arxiv.org/abs/2412.06538).
- NLLB Team (2022)
James Cross Onur Çelebi Maha Elbayad Kenneth Heafield Kevin Heffernan Elahe Kalbassi Janice Lam Daniel Licht Jean Maillard Anna Sun Skyler Wang Guillaume Wenzek Al Youngblood Bapi Akula Loic Barrault Gabriel Mejia Gonzalez Prangthip Hansanti John Hoffman Semarley Jarrett Kaushik Ram Sadagopan Dirk Rowe Shannon Spruit Chau Tran Pierre Andrews Necip Fazil Ayan Shruti Bhosale Sergey Edunov Angela Fan Cynthia Gao Vedanuj Goswami Francisco Guzmán Philipp Koehn Alexandre Mourachko Christophe Ropers Safiyyah Saleem Holger Schwenk Jeff Wang NLLB Team, Marta R. Costa-jussà.
No language left behind: Scaling human-centered machine translation.
2022.
- Ociepa et al. (2024)
Krzysztof Ociepa, Krzysztof WrĂłbel, Adrian GwoĹşdziej, Remigiusz Kinas, et al.
Bielik 7b v0. 1: A polish language model–development, insights, and evaluation.
*arXiv preprint arXiv:2410.18565*, 2024.
- Ostendorff & Rehm (2023)
Malte Ostendorff and Georg Rehm.
Efficient language model training through cross-lingual and progressive transfer learning.
*arXiv preprint arXiv:2301.09626*, 2023.
- Özeren et al. (2025)
Enes Özeren, Yihong Liu, and Hinrich Schuetze.
HYPEROFA: Expanding LLM vocabulary to new languages via hypernetwork-based embedding initialization.
In Jin Zhao, Mingyang Wang, and Zhu Liu (eds.), *Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop)*, pp. 79–96, Vienna, Austria, July 2025. Association for Computational Linguistics.
ISBN 979-8-89176-254-1.
doi: 10.18653/v1/2025.acl-srw.6.
URL [https://aclanthology.org/2025.acl-srw.6/](https://aclanthology.org/2025.acl-srw.6/).
- Pagnoni et al. (2025)
Artidoro Pagnoni, Ramakanth Pasunuru, Pedro Rodriguez, John Nguyen, Benjamin Muller, Margaret Li, Chunting Zhou, Lili Yu, Jason E Weston, Luke Zettlemoyer, Gargi Ghosh, Mike Lewis, Ari Holtzman, and Srini Iyer.
Byte latent transformer: Patches scale better than tokens.
In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), *Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 9238–9258, Vienna, Austria, July 2025. Association for Computational Linguistics.
ISBN 979-8-89176-251-0.
doi: 10.18653/v1/2025.acl-long.453.
URL [https://aclanthology.org/2025.acl-long.453/](https://aclanthology.org/2025.acl-long.453/).
- Pennington et al. (2014)
Jeffrey Pennington, Richard Socher, and Christopher Manning.
GloVe: Global vectors for word representation.
In Alessandro Moschitti, Bo Pang, and Walter Daelemans (eds.), *Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)*, pp. 1532–1543, Doha, Qatar, October 2014. Association for Computational Linguistics.
doi: 10.3115/v1/D14-1162.
URL [https://aclanthology.org/D14-1162/](https://aclanthology.org/D14-1162/).
- Remy et al. (2023)
François Remy, Pieter Delobelle, Bettina Berendt, Kris Demuynck, and Thomas Demeester.
Tik-to-tok: Translating language models one token at a time: An embedding initialization strategy for efficient language adaptation.
*arXiv preprint arXiv:2310.03477*, 2023.
- Remy et al. (2024)
François Remy, Pieter Delobelle, Hayastan Avetisyan, Alfiya Khabibullina, Miryam de Lhoneux, and Thomas Demeester.
Trans-tokenization and cross-lingual vocabulary transfers: Language adaptation of llms for low-resource nlp.
*arXiv preprint arXiv:2408.04303*, 2024.
- Rybak (2023)
Piotr Rybak.
Maupqa: Massive automatically-created polish question answering dataset.
*arXiv preprint arXiv:2305.05486*, 2023.
- Singh et al. (2025)
Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David Ifeoluwa Adelani, Jian Gang Ngui, Daniel Vila-Suero, Peerat Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, Raymond Ng, Shayne Longpre, Sebastian Ruder, Wei-Yin Ko, Antoine Bosselut, Alice Oh, Andre Martins, Leshem Choshen, Daphne Ippolito, Enzo Ferrante, Marzieh Fadaee, Beyza Ermis, and Sara Hooker.
Global MMLU: Understanding and addressing cultural and linguistic biases in multilingual evaluation.
In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), *Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 18761–18799, Vienna, Austria, July 2025. Association for Computational Linguistics.
ISBN 979-8-89176-251-0.
doi: 10.18653/v1/2025.acl-long.919.
URL [https://aclanthology.org/2025.acl-long.919/](https://aclanthology.org/2025.acl-long.919/).
- Takase et al. (2025)
Sho Takase, Ryokan Ri, Shun Kiyono, and Takuya Kato.
Large vocabulary size improves large language models.
In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), *Findings of the Association for Computational Linguistics: ACL 2025*, pp. 1015–1026, Vienna, Austria, July 2025. Association for Computational Linguistics.
ISBN 979-8-89176-256-5.
doi: 10.18653/v1/2025.findings-acl.57.
URL [https://aclanthology.org/2025.findings-acl.57/](https://aclanthology.org/2025.findings-acl.57/).
- Tamang & Bora (2024)
S. Tamang and D. J. Bora.
Evaluating tokenizer performance of large language models across official indian languages, 2024.
URL [https://arxiv.org/abs/2411.12240](https://arxiv.org/abs/2411.12240).
- Team et al. (2025)
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al.
Gemma 3 technical report.
*arXiv preprint arXiv:2503.19786*, 2025.
- Team (2025)
Qwen Team.
Qwen3 technical report, 2025.
URL [https://arxiv.org/abs/2505.09388](https://arxiv.org/abs/2505.09388).
- Touvron et al. (2023)
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.
Llama: Open and efficient foundation language models.
*arXiv preprint arXiv:2302.13971*, 2023.

## Appendix A Segmentation Algorithm

Instead of relying on word-based segmentation, we use an offset-based segmentation strategy. Designing a consistent word segmentation across tokenizers is challenging because tokenizers often differ in normalization rules, pre-tokenization steps, language coverage, etc. These differences make it difficult to ensure that segment boundaries match at the word level.

The offset-based method addresses this by operating directly on character offsets in the original text. Given two different tokenizations of the same string, along with the start and end positions of each token, the algorithm searches for all possible split positions that never cut through the middle of any token (see Figure [3](https://arxiv.org/html/2510.21954v1#A1.F3)).

This approach is universal: such a segmentation always exists, even if the worst case reduces to a single segment spanning the entire input. It can also lead to more precise alignments because the target tokenizer may break a word into several sub-tokens. By working with character offsets, we can introduce mid-word segment boundaries whenever they yield a better match.

For example, consider the sentence CH4 is a formula for methane. Suppose the original tokenizer produces the tokens _for, m, and ula for the word formula, while the new tokenizer produces _form and ula. A word-level strategy would force alignment at the whole-word boundary, but an offset-based method can instead match _for and m with _form, and ula with ula, which more closely respects both tokenizations.

Algorithm [1](https://arxiv.org/html/2510.21954v1#algorithm1) provides detailed pseudocode for implementation.

Figure: Figure 3: Offset-based segmentation algorithm visualization.
Refer to caption: x3.png

Figure: Algorithm 1 Offset–Based Segmentation

## Appendix B Convergence Speed

We repeated the experiment with Gemma 3 4B PT described in Section [4.2](https://arxiv.org/html/2510.21954v1#S4.SS2), but this time we saved model checkpoints every 3,000 training steps. While the results in Table [3](https://arxiv.org/html/2510.21954v1#S4.T3) were obtained after 250,000 steps, this setup allows us to observe how quickly the embeddings adapt to the new tokenizer and to evaluate whether training can be substantially shortened.

The AIM objective provides a rich learning signal for tuning the embeddings. Using a mean squared error (MSE) loss, the number of value pairs contributing to the objective is proportional to the product of the head dimensionality, the number of attention heads, the number of possible segment pairs, and the batch size. In the configuration used here – four documents per batch, each truncated to 256 tokens – this amounts to hundreds of millions of pairs at every step of the training.

Table [4](https://arxiv.org/html/2510.21954v1#A2.T4) presents the results for the first eight checkpoints on the Belebele benchmark across all tested languages, while Figure [4](https://arxiv.org/html/2510.21954v1#A2.F4) provides a visual view of the same trends.

The data show that more than 50% of the final performance gains can be achieved in under 10% of the total training steps, corresponding to fewer than five million tokens per language. This indicates that training time could be cut dramatically with only a minor loss in accuracy, especially if an adaptive data selection strategy is used to prioritize documents that contain a higher proportion of previously unseen tokens.

Figure: (a) Arabic
Refer to caption: x4.png

**Table 4: Performance on the Belebele benchmark during early training of Gemma 3 4B PT with Model-Aware Tokenizer Transfer, showing rapid gains within the first 10% of steps compared to the full run.**
|  |  | Belebele |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Steps # | Tokens # | ar | de | en | ja | sw |
| 0k | 0M | 32.89 | 52.44 | 80.33 | 42.00 | 25.56 |
| 3k | 3M | 34.78 | 56.89 | 80.33 | 45.33 | 30.33 |
| 6k | 6M | 36.44 | 59.22 | 80.44 | 48.00 | 36.44 |
| 9k | 9M | 37.00 | 63.44 | 80.44 | 50.56 | 37.67 |
| 12k | 12M | 40.78 | 64.56 | 80.44 | 49.11 | 40.56 |
| 15k | 15M | 42.56 | 65.56 | 80.44 | 48.11 | 42.00 |
| 18k | 18M | 44.22 | 67.00 | 80.44 | 49.11 | 42.22 |
| 21k | 21M | 45.89 | 67.56 | 80.56 | 49.67 | 43.22 |
| 24k | 24M | 48.00 | 68.89 | 80.56 | 49.67 | 44.33 |
| 250k | 250M | 62.44 | 72.56 | 80.67 | 58.00 | 54.67 |

## Appendix C Ablation Studies

We apply the MATT method with AIM objective, transferring Gemma 3 4B PT to a Ukrainian-centric tokenizer by Bohdan Didenko (2025), which increases the compression rate by around 50%. We conduct several ablation studies to determine the optimal training configuration, evaluating performance on the Belebele and Global MMLU benchmarks in the same setting as in Section [4](https://arxiv.org/html/2510.21954v1#S4). We report the results in Table [5](https://arxiv.org/html/2510.21954v1#A3.T5) and describe our insights below.

**Table 5: Ablation studies of MATT configurations. 3 and 5 denote the number of layers used for AIM objective.**
|  | VRAM (GiB) | Time | Belebele | Global MMLU |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | 3 | 5 | 3 | 5 | 3 | 5 | 3 | 5 |
| All Layers vs. Last Layer |  |  |  |  |  |  |  |  |
| all layers | 17.3 | 26.5 | 1h 33m | 2h 19m | 32.56 | 35.11 | 28.63 | 29.00 |
| last layer | 9.1 | 10.1 | 0h 56m | 1h 04m | 37.22 | 60.11 | 29.95 | 34.80 |
| Initialization Method |  |  |  |  |  |  |  |  |
| WECHSEL | 9.1 | 10.1 | 0h 52m | 1h 04m | 42.22 | 59.78 | 29.82 | 33.56 |
| FOCUS | 9.1 | 10.1 | 0h 55m | 1h 06m | 37.11 | 60.89 | 30.10 | 34.72 |
| Transtokenizers | 9.1 | 10.1 | 0h 55m | 1h 04m | 52.44 | 60.89 | 32.43 | 34.75 |

##### AIM objective on all layers deteriorates both efficiency and performance compared to only the last one.

Since AIM is defined for a single attention layer, we can apply it many times to any subset of layers. To balance efficiency and accuracy, we compare defining the AIM objective over all layers up to a target depth $n$ against using only the final $n$-th layer. Results in Table [5](https://arxiv.org/html/2510.21954v1#A3.T5) show that restricting AIM to the last layer requires much less VRAM and training time, while also delivering better downstream performance.

##### FOCUS and Transtokenizers perform similarly on higher layers, while WECHSEL underperforms.

Because MATT is independent of the embedding initialization method, different starting points can be tested. We compare WECHSEL, FOCUS, and Transtokenizers. FOCUS and Transtokenizers perform similarly on higher layers, while WECHSEL lags behind (see Table [5](https://arxiv.org/html/2510.21954v1#A3.T5)). Although Transtokenizers occasionally achieves the best scores, in other experiments, we find FOCUS to be more stable across models, and therefore make it our default choice. Minor efficiency differences are likely due to external factors such as checkpointing overhead.

##### AIM on higher layers leads to better results, but saturates at around one third of the model’s depth.

MATT allows selecting how deep into the model the AIM objective is applied, creating a natural trade-off between efficiency and accuracy. We evaluate different target depths and find that performance steadily improves as AIM is applied to higher layers (see Figure [5](https://arxiv.org/html/2510.21954v1#A3.F5)), but gains saturate once the objective reaches roughly one third of the model’s total depth. In contrast, memory consumption and training time continue to grow almost linearly with the number of layers, highlighting the cost of deeper alignment.

Figure: (a) Performance
Refer to caption: x9.png

## Appendix D Addressing The Tied Embeddings Requirement

While MATT is less effective when input and output embeddings are not tied, there are several promising directions to mitigate this limitation.

One practical approach is to fine-tune only the input embeddings. Even without updating the output layer, this strategy already outperforms existing baseline methods according to some early endeavors. During a later continual pre-training stage, the language model (LM) head can then be trained efficiently because it is directly connected to the loss, allowing the rest of the model to remain frozen while only the LM head is updated.

We also conducted preliminary experiments with a mapping technique that transfers input embeddings to the output embeddings space. In this setup, the output embeddings for new tokens were initialized using the mapped input embeddings after MATT fine-tuning. However, this approach underperformed compared to initializing with FOCUS and then fine-tuning only input embeddings with MATT. This can be attributed to low-capacity mapping models and requires further research.

Despite these early results, the search space remains large, and we believe that more effective strategies for untied embeddings are likely to be found with further exploration.

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 2
presentation: 3
questions: - **Q1:** The training loss only optimizes the new embeddings in their role as input embeddings, as you discuss. How does this influence the model’s ability to actually generate the new tokens (in the case of tied embeddings)? 
- **Q2:** The proposed method’s intuition, setup, and loss objective seem (very) similar to the one proposed in https://arxiv.org/abs/2505.20133, could you discuss the differences of your work?
- **Q3:** What target layer for the AIM/AIM* objectives is used in the main experiments?
- **Q4:** How well does MATT work with varying degrees of presence of the target language(s) in the models original pretraining data? Does the model need to have already been trained on target languages?

**(nit):** this does not influence my review of the paper, but could you explain the difference between the proposed AIM objective and the MATT method? Is MATT just the application of the AIM (or AIM*) objective?
soundness: 2
strengths: **S1:** The proposed method significantly outperforms (non-learning based) heuristic embedding initialization methods in the reported evaluations.

**S2:** The method’s intuition on distilling inter-token attention patterns is well-motivated.

**S3:** The paper is well-written and easy to follow, with good figures to explain the proposed method.
summary: The paper proposes a method to transfer the tokenizer of a pretrained LLM to a new tokenizer. It initializes embeddings for new tokens by optimizing a distillation objective based on matching segments of hidden states. The method is evaluated in language adaptation settings and compared against several other “heuristic” embedding initialization methods.
weaknesses: **W1:** There are several important baselines missing, which are required to properly contextualize the strong results of the proposed method. In general, the current experimental evaluation is insufficient and forgoes comparison against stronger (non-heuristic) embedding initialization baselines. **It is not surprising** that directly optimizing new embeddings on 240 million (!) tokens outperforms heuristic initializations based on some mappings between embedding spaces without any further training. Therefore, I recommend adding comparisons against the following baselines in a future revision to back up the strong claims of the paper:
- **W1.1:** The method compares only against “heuristic” embedding initialization methods which are not learning based. Apart from ZeTT (which amortizes the learning during its hyper-network pretraining), one crucial baseline is learning the new embeddings via the *classic next-token prediction objective* using the same setup as in the experiments for the proposed AIM objective (e.g. same data, computational budget, ..). **This baseline is necessary** and should have been provided in all experiments to evaluate the usefulness of the proposed (more complicated) objective vs. a simpler existing learning objective.
 - **W1.2:** As ZeTT (https://arxiv.org/abs/2405.07883) is very competitive, a comparison of the proposed method to ZeTT is desirable. Can the direct optimization of the embeddings via MATT beat the embeddings produced by ZeTT?
- **W1.3:** Another non-heuristic baseline is the method proposed in https://arxiv.org/abs/2410.05864, which extracts new token representations from intermediate layers and maps them back to the embedding spaces. This method also outperforms heuristic-based initialization methods in their evaluations.

**W2:** In l. 81ff:
> Our contributions are: [...] Model-Aware Tokenizer Transfer (MATT): an efficient tokenizer-transfer method that exploits model dynamics instead of relying solely on semantic relationships, achieving state-of-the-art results with substantially lower computational cost than language modeling objectives.

 I could not find any comparison to language modeling objectives (i.e. next-token prediction with cross-entropy?) in the paper, could you point me to the correct Figure/Table?

**W3:** In l. 137ff you write:  
> Other work (Abagyan et al., 2025) shows that periodically resetting embeddings during pretraining makes models more robust to them, reducing the effort needed to learn new tokens afterwards. 

but I cannot find support for this claim anywhere in the paper by Abagyan et al., 2025. Could you elaborate?




**Score summary:** I recommend to **`reject`** the paper in its current state due to (1) missing baselines which are necessary to evaluate the strength of the proposed method compared to prior work / simpler methods (see W1), (2) claims for which I could not find support for in the paper/cited references (W2 & W3) and (3) questions regarding novelty compared to insufficiently discussed prior work (Q2). If these points are sufficiently addressed, I am open to raising my score.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: - Line 337: What does it mean to merge tokenizers? This is a bit confusing to me as it sounds that the training process is then agnostic to tokenizers, but for each input you still need to segment based on each tokenizers separately to obtain inputs to your loss function, right?
- Line 361: Why is the MSE loss chosen to be on the 12th layer? I see in Table 5 all layers were tested, but were there also other single layers tested except layer 12? Could it be that some other layers provide more useful information? It might be interesting to look at earlier or intermediate layers as well.
soundness: 3
strengths: - The paper is well motivated and the idea is novel to the best of my knowledge
- The results seem to be promising compared to other baselines and heuristics
- The method enables efficient transfer from a source language to a variety of target languages.
summary: This work aims to tackle transfer of pre-trained language models on a source language to a certain target language. This is critical as the distribution of data availability is not uniform across languages and knowledge of a pre-trained model may enable more efficient transfer to obtain capable language models for target languages. To this end, the authors propose model-aware tokenizer transfer (MATT) which, contrary to existing methods, leverages the idea that a tokenizer for a target language should preserve the attention weight patterns of the source language to maximize performance. This is instantiated via Attention Influence Modeling (AIM) that aligns softmax weights between source and target tokenizers at overlapping string segments. MATT outperforms commonly used heuristics on various target languages demonstrating strong performance and improved compression rate.
weaknesses: Generally, I am positive about the paper, however there are some clarifications required for me to fully comprehend the relevance and results reported in this work.

**Baseline comparison**

I am not convinced that Table 1 provides a fair comparison, as the authors mentioned, most of the competitors are mere heuristics that do not involve any optimization procedure whereas MATT involves an optimization process.
The comparison would be more fair if the amount of compute that was used for MATT optimization is invested in LM training for the heuristics, as this is what they usually rely on as a second step. This could be matched by the amount of compute that is needed for MATT and allocate the same amount of compute (like shown in Table 3) to tuning the embedding layer for heuristics on the language modeling task.
Eventually, this would add a new set of results that shows different compute budgets and a comparison for those of MATT to others, which would add strength and rigor to the paper.

**Clarity**

The clarity of the experimental design could be improved. For me it was sometimes not clear whether the evaluation was done in the source or in the target task and it was not explicitly stated. 
For example, Belebele spans 122 languages, was the evaluation in Table 1 only done on the Ukrainian subset? Same question for GlobalMMLU and LongFLORES.

In Table 3 there are some languages for which the base model is still better even after tokenizer transfer, like arabic or japanese. As far as I understand this provides negative results -- why would you do tokenizer transfer if the base model performs better on certain langauges anyways? Maybe I misunderstood this part, but I believe it is central to clarify the relevance of this work. If it is indeed a negative result, is there a way to find a proxy for when tokenizer transfer would work and when it doesn't?

Finally, it is not entirely clear to me whether after tokenizer transfer during inference one still needs to sort of switch between tokenizers depending on the language one wants to predict for, i.e. do I need to know during inference whether I evaluate on a target language or not? In my view this knowledge is required, but from the way certain parts of the text are written it sounds like it is not, can the authors please elaborate on that?

**Significance of results**

The performance difference between different components of MATT in Table 1 and Table 3 is sometimes marginal, therefore it is not clear what setup generally works best or should be used by default. I recommend to add error bars at least for the ablation studies to see whether different setups exhibit statistically significant performance.


**Limitations**

If I understood correctly, another limitation that wasn't mentioned is that for collecting the targets for the optimization process you need an extra forward pass, once with the original tokenizer and once with the external tokenizer which is then used for backpropagation to obtain gradients for the embedding weights.

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 3
questions: 1. The attention-related definitions in Section 3 only consider a single-head scenario. Given that almost all transformer-based models use multi-head attentions, this aspect warrants clarification (at least by a footnote).

2. How the paper chooses $n$ (i.e., how many layers are considered for the loss computation) is unclear in the main body of the paper. This should be clearly explained in the main text, as it is the core hyperparameter of the method.

3. On Weakness 1, what happens when an LM head is tuned using MATT? Does it improve performance?

4. On Weakness 2 (i), why does the paper consider only Ukrainian? The paper should expand its evaluation to different typological languages.

5. On Weakness 2 (ii), the paper should consider languages that are not supported in Qwen3. For instance, Amharic is not on the list. There are many other languages that are not on the list but are included in FLORES, Belebele, and Global MMLU.

6. On Weakness 2 (iii), the paper should consider another generation task like summarization. I believe most of the tested languages in this paper are included in XL-SUM.

7. On Weakness 2 (iv), the paper should consider including results on actual inference speedups.

8. On Weakness 2 (v), the paper should try to use the same set of tasks and models across the paper. If this is not possible, the paper should give justifications.

9. On Weakness 2 (vi), the paper should include an additional discussion on the observed performance difference between languages and models in Section 4.2.

10. On Weakness 3, the paper should conduct continual pre-training for all baselines using the same budget as MATT. Moreover, I would recommend including some full continual pre-training results for the proposed method and the best-performing baseline to examine the extent to which the advantage of the proposed method holds.

11. On Weakness 4, the paper should expand the evaluation further to make the corresponding statement reliable. If this is not possible, the discussion requires revision.
soundness: 1
strengths: 1. This paper focuses on an important problem of how to efficiently transfer vocabulary from source to target with a particular focus on vocabulary expansion settings. While investigated by many previous studies, this paper proposes a fundamentally new approach that seeks to incorporate attention-weight information between source and target tokenizers for transfer, accounting for higher-layer model dynamics. This aspect has been neglected in previous studies and can be seen as one of the major contributions of this study in terms of originality and significance.
2. The paper is mostly well-written and easy to read up until Section 3. In particular, the intuition and motivation are pretty clear in the paper.
summary: This paper proposes a novel method to initialize the embeddings of newly added tokens for cross-lingual transfer effectively. The proposed method aims to incorporate higher-layer model dynamics into the initialization of new embeddings by forcing the model with the target (adapted) tokenizer to generate output embeddings similar to those produced by the source tokenizer. This is achieved with the attention influence modeling objective and a relatively small amount of training. The experimental results on both Gemma3 and Qwen3 demonstrate that the proposed approach outperforms existing initialization baselines that do not go through training.
weaknesses: 1. The fundamental limitation of this work is that the method is only applicable to input embeddings. Some frontier models, like Qwen3, do not tie weights. In that case, the effectiveness of the proposed method substantially diminishes compared to models with weight tying (Table 3).

2. The experiments and evaluation are neither extensive nor comprehensive. Specifically,  
    * (i) The main adaptation experiments in 4.1 are only conducted in Ukrainian without justifications, which fails to demonstrate the applicability to different languages with different scripts.
    * (ii) all target languages in the paper are officially supported in the tested model - Qwen3 (i.e., it lacks experiments on unseen languages or unsupported languages.) While Gemma3 does not provide a list of supported languages, it is highly likely to support the target languages considering the vocabulary size and multilingual performance reported in the technical paper. Therefore, it is not clear whether the proposed method exhibits a similar advantage of better downstream performance when applied to unseen or unsupported languages.
    * (iii) The paper considers only English-to-target machine translation as a generation task, which lacks the diversity of evaluation.
    * (iv) While the paper reports compression rates as the average number of characters, it fails to report the actual inference speedups (e.g., tokens/sec used in previous related papers - See Hong et al. (2024) and Yamaguchi et al. (2024, 2025). This prevents us from evaluating the actual efficiency impact of using the proposed method.
https://arxiv.org/abs/2401.10660
https://arxiv.org/abs/2406.11477
https://arxiv.org/abs/2412.11704
    * (v) The main experiments in 4.1 and the multilingual experiments in 4.2 use different models without justifications. Also, 4.2 lacks generation evaluation. This hinders comparison between monolingual and multilingual settings.
    * (vi) The multilingual evaluation lacks in-depth analysis regarding observed differences between languages and models.

3. The comparison against baselines seems unfair, making it impossible to understand the true gain from the proposed method. While the paper argues that evaluating only the initialized model is sufficient to compare MATT with existing baselines (L477-479), MATT indeed involves training. Given that the other baselines do not include training of the base model, the paper should continually pre-train all baselines using the same compute budget to make them comparable. Moreover, one can assume that the longer the continual pre-training, the smaller the difference between initialization methods. Therefore, it is also important to demonstrate how well the performance advantage of the proposed method holds when conducting continual pre-training.

4. The discussion on the instruction-tuned model in L405-407 is entirely not reliable, given that (i) generation evaluation is conducted only on an English-to-target machine translation task and (ii) only Ukrainian is tested in the evaluation.

---

confidence: 5
contribution: 3
initial_rating: 8
presentation: 4
questions: * Have you considered how performance improves after a more traditional finetuning of the model using its natural autoregressive finetuning?
* Do you have an explanation for the reason why FOCUS and Transtokenizers perform about as well after MATT finetuning, but Transtokenizers perform much better alone? Are the correlations learnt by Transtokenizers' token alignment similar to the ones recovered by your approach?
soundness: 4
strengths: * Well-explained intuition and very reasonable assumptions.
* Outstanding and well-summarized review of the state-of-the-art in tokenization trade-offs and embedding adaptation.
* Numerous baselines and ablation studies, which bring confidence in the results, and offer many potential insights.
* Very reasonable compute requirements for the achieved inference speed improvements achieved.
summary: In this work, the authors take existings LLMs and swap their tokenizers for more efficient inference (and if desired finetuning) in a desired target language (e.g. Ukrainian). The novelty of their approach is to not only consider token embeddings on their own while reinitializing the embddings, but to also consider their perception by the model's attention layers. While considering the attention outputs does require undergoing a slight finetuning after embedding reinitialization, the efficiency of their propsed attention alignment objective is probably higher than traditional autoregressive finetuning. The authors show that their training methodology is lightweight, achieves better outcomes than embedding reinitialization alone, and bridges the gap between simpler and more advanced initializations.
weaknesses: * Unlike the selected baselines, MATT involves training time on a GPU. A fairer baseline would thus consider how much a FOCUS or Transtokenizers baseline would improve after seeing ~3h of embedding finetuning on a H100 (using a more traditional autoregressive loss). After reading the paper, I assume that the proposed approach would likely perform better at a small training regime, but it might saturate sooner as the signal is only an approximation of the real objective. This would be good to objectify.
* Most LLMs do not use tied input/output embeddings, which reduce the effectiveness of the proposed method for these popular use cases. For the output embeddings, a more effective embedding transfer method and autoregresive finetuning might thus serve the use case better.


Decision: Reject