Token Distillation: Attention-Aware Input Embeddings for New Tokens
Authors: Konstantin Dobler
Desmond Elliott
Gerard de Melo
Venue: ICLR 2026 Poster (2026)
Keywords: embedding initialization
tokenizer
vocabulary adaptation
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We propose Token Distillation to obtain high-quality input embeddings for new tokens by distilling representations obtained using the original tokenization.

Abstract: Current language models rely on static vocabularies determined at pretraining time, which can lead to decreased performance and increased computational cost for domains underrepresented in the original vocabulary.
New tokens can be added to solve this problem, when coupled with a good initialization for their new embeddings. However, existing embedding initialization methods require expensive further training or pretraining of additional modules.
In this paper, we propose Token Distillation and show that by distilling representations obtained using the original tokenization, we can quickly learn high-quality input embeddings for new tokens.
Experimental results with a wide range of open-weight models show that Token Distillation outperforms even strong baselines.

Published as a conference paper at ICLR 2026
TOKEN DISTILLATION: ArTENTION-AWAR INPUT EMBEDDINGS FOR NEW TOKENS
Konstantin Dobler
Desmond Elliott
Gerard de Melo
Hasso Plattner Institute
University of Copenhagen
Hasso Platmer Institute ELLIS Unit Potsdam gerard deme oeopol.pp.oooo
ELLIS Unit Potsdam
ELLIS Unit Copenhagen
konstant in. www.edobbl.....oo.
de@di. ku.
ABSTRACT
Current language models rely static vocabularies determined at pretraining time, which can lead to decreased performance and increased computational cost for domains underrepresented in the original vocabulary. New tokens can be added to solve this problem, when coupled with a good initialization for their new embeddings. However, existing embedding initialization methods require expensive further training or pretraining of additional modules. In this paper, we propose Token Distillation and show that by distilling representations obtained using the original tokenization, we can quickly learn high-quality input embeddings for new tokens. Experimental results with a wide range of beenhw models show that Token Distillation outperforms even strong baselines
INTRODUCTION
Pretrained language models are trained with fixed tokenizer that often fragments domain-specinf novel terms into multiple subtokens. This excessive tokenization not only leads to reduced perfor- mance on downstream tasks (Rust et al. 2021 Ali et al., 2024) but also increases the computational (and therefore also financial) cost due to inflated sequence lengths (Ahia et al. 2023 Yamaguchi
al. 2024a). Although adding new tokens to a model's vocabulary can reduce over-token is crucial to choose good initialization for the new embeddings (Gee et al. 2022 Minixhoter et al.
2022 Dobler & de Melo 2023: Yamaguchi al., 2024a).
In this paper. we argue that many recent methods for embedding initialization are fundamentally limited. Whenever we wish to add a new token to pretrained model's vocabulary, this new token may be split up into multiple subtokens in the original model's tokenization. However, these subtokens might not be individually informative about the semantics of the entire new token (consider, e.g., <_pal> <at> <able>). The semantics of a word composed of multiple subtokens will largely not be stored in their raw input embeddings at all but rather constructed by the Transformer's 's
caattiionfeedffowwwar layer stack during contextualization (Elhage al. 2022 Lad et al. 2024
Kaplan et al. 2025).
Therefore, methods that do not exploit the information encoded in Transformer layer weights are at serious disadvantage. Motivated by this insight, we propose a novel method for input embedding initialization that captures information stored in all Transformer layers in addition to the existing input embeddings. Our method optimizes new token embeddings with a distillation-base objective match the model's behavior compared to when a new token split up into its original subtokens. We demonstrate the efficacy of our method, dubbed "Token Distillation", in Section 5 We illustrate our method in Figure and describe it in [Section 3
We compare against strong baselines, including standard embedding initialization procedures, training of the embedding matrices with causal language modeling, as well as pretrained hyper-networks that predict new token embeddings. Extensive experiments on a wide range of open-weight models, including question-answering benchmarks and the generation of definitions for the newly added tokens confirm the effectivvness of our approach. Our experimental setup is detailed in Section 4 Additionally, we describe related work in Section and explicitly discuss limitations our method in Appendix A In summary, contributions are as follows.
We propose Token Distillation, a novel method for providing high-quality input embeddings when adding new tokens to pretrained language models.
'Our code is available at http:/t.w.t.............//oo//aa

---

Published as a conference paper at ICLR 2026
palatable
food
is
good
Frogen pretrained LLM
: new token embedding (only this is trained)
original subtoken embedddings of new token
context sequence token embeddings
MSE (match behavior)
hidden states using original subtokens
= hidden states using single new token
Frozen pretzained
LLM
hidden states of original subtokens [except for last one); discarded in loss computation
pal
ALt
alble
food
is
good
Figure I: Illustration of Token Distillation Given a sequence containing our new target token, we first obtain the model's hidden states on that sequence using the original tokenization and then quickly learn new embedding by reducing the mean squared error (MSE) between the original hidden states and the hidden states of the model when using single token embedding to replace the original subtokens.
We motivate our proposed method by describing the fundamental limitations of current embed- ding initialization methods and empirically verify our claims.
Extensive experiments and in-depth analyses of our method using wide range of open open-weight model checkpoints and strong baselines confirm its effectiveness.
2
BACKGROUND
Most state-of-the-ar Large Language Models (LLMs) trained using a static tokenizer, usually derived by byte-pair encoding scheme before model training (Sennrich et al.) (016). However, single tokenizer predetermined before model training might not be equally well-suited for the many different use cases that a model will need to address later on. Suboptimal tokenization of new languages or -specfi text. increases the cost of further training and inference (Ahia et Yamaguchi et al. (2024) and has also been tied to reduced performance (Rust et al. [02]] Ali
2023
ct al. 2024). Furthermore, Lesci et al. (2025) show that in practice, words which are not a single token the vocabulary are significantly less likely to be generated.
solution to this problem is to modify the existing vocabulary to suit the specific needs. However, simply modifying the vocabulary insufficient we also need suitable embeddings for the new tokens. A common approach is simple random initialization followed by a phase of training only the new embedding layers the target corpus to "warm up" the new embedding weights. However, random initialization of new token embeddings has been widely shown to underperforn better methods (e.g.. Gee et al. 2022: Minixhofer et al.] 2022; Dobler & de Melo 2023). This has led to notable interest in devising ways of obtaining more informative embeddings for new tokens.
common alternative is to initialize a new token's 'ss embedding as the mean its subtoken embeddings
(Sachidananda al.) 2021 Koto et al.) 2021] Gee et al. 2022) -potentially weighted based on the subtoken position in the new token (Nakash et al.) 2025) ) or based on other heuristics (Downey al. 2023). Alternatively, weighted mean of existing embeddings can also be computed based
similario scores in auxiliary embedding spaces (Wang et al. 2019 Tran 2020 Minixhofee et al.
2022 Ostendorff & Rehm 2023 Dobler & de Melo, 2223
2023] Liu et al. 2024 inter alia). In fact, most recent research into embedding initialization for new tokens proposes some variation of a weighted
linear combination
copy of existing embedding vectors (Mosin al. 2023 Zeng et 2023
Mundra et al.] [2024] Yamaguchi et al. 2024a b) Remy et al.) 2024 Goddard & Neto 2025 Lee
al. 2025 Singh et al. 2025).. Nonetheless, the resulting embeddings still require further tuning to achieve good results (Minixhofer et al., 2024). We refer to this category of initialization methods as
"model-gradient free". as they do not consider any training signal from the model.
Only few methods based on access model gradients have been investigated in recent years. Model gradients can be used by directly optimizing just the new token embeddings to minimize a cross-entropy loss relevant contexts (Lampinen & McClelland, 2018), the common practice of
2

---

Published as a conference paper at ICLR 2026
training all embeddings while freezing other parts of the model (Atetxx al. 2020; de Vries &
Nissim 2021) - potentially only backpropagating through parts of the model (Marchisio et al. 2023)
-or via pore-itraining a hypeer-ettorrk (Ha et al.) (2017) that is able to predict embeddings for new
tokens in the target embedding space (Pinter et al. 2017; Schick & Schüitze| 2019. 2020; Teehan
at 2024 Minixhofer et al.) 2024). Alternatiely, using PatchScopes (Ghandeharioun et al. 2024).
"Tokens to Words" (Kaplan et al. (025) locates layers at which a word fragmented into multiple subtokens is represented by single contextual embedding in the residual stream, and then projects these hidden states into the input/ouiput embedding spaces via trained mappings. These methods - hyper-networks in particular have been shown to yield immediately useful representations but require additional compute for ptra-ntraining. In the case of hyper-networks, it additionally might be necessary to train different hyper-networks for different target domains, further increasing the complexity Motivated by these difficulties, we propose an alternative method in this paper that does not require extensive pretraining of an embedding prediction hyper-network or additional modules and still achieves competitive results, even outperforming pretrained syper-networks.
3 METHOD: TOKEN DISTILLATION
Our goal is to initialize input embeddings for new tokens such that, when inserted into frozen pretrained Transformer, they faithfully reproduce its original behavior now using a single new token instead of multiple subtokens. Specifically, we seek an embedding e' for new token t* such that all downstream hidden states match those obtained when the model would instead have seen the original subtokens ( (n.. which 5* would have been tokenized into according to the original vocabulary. In what follows, we (1) explain why existing methods fall short and motivate our method, (2) formalize our method, and (3) describe implementation details.
3.1 INTUITION
As illustrated in |Section I individual embeddings for subtokens t,...., <able>) of new token t² (e.g., <_palatable>> do not necessarily store the semantics of the specific concatenated form t*. Instead, their contexualized representation is gradually constructed in the Transformer layers via neural detokenization process (Elhage al. 2022 Lad al. 2024 Kaplan et al. 2025) and then attended to by other token positions in the sequence, which are influenced by this contextualized representation. The various prior approaches that solely rely on information encoded in the embedding matrices of model thus ignore the crucial functional knowledge stored in the attention and feed-forward weights of Transformer layers, which imposes a fundamental limitation on their performance.
fn (e.g., <_pal> <at>
Precisely pinpointing the specifically involved attention heads is difficult and can require manual inspection (Elhage et al.) (002).. Instead, we propose to distill (Hinton et al., 2015 Snell et al.) 2022) the impact that the multiple subtokens f1.-.., L,, have on other tokens attending to them into single token embedding e* Our intuition is as follows: If we identify an embedding e* for t such that the model produces similar hidden states in the succeeding positions after seeing t* and t"'s original subtokens ..,..., tn. we have extracted the relevant information stored in the model's Transformer layers without requiring a specific localization within the model weights. Thus, we propose to optimize e by minimizing the mean-squared error between hidden states produced by the input sequences containing ,,,...,44 in and their counterparts using L* We will show in [Section that this empirically outperforms existing training-free embedding initialization methods, embedding tuning with next-token prediction objective, as well as pretrained embedding prediction hyper-networks. Next, we formally describe our method.
3.2 METHOD DESCRIPTION
Let 7 be the tokenizer of a pretrained Transformer. Given a new token string t* with original subtokens r(t") = [t....., tt,], we require a small corpus of example sequences E S each containing the new token string L*. We denote the new tokenizer, which includes 1*, as r², and the tokenization of 8 using t or r* as 8, and 87,, respectively. The target for learning the embedding of L' are the hidden states at specified layer / produced by the language model when processing s,, which we denote as (s+). We wish to find an embedding c* for new token t* such that hidden states
Minixhofer et al. 2024) train separate hyper-networks for mistralal a /Mi stral -7B-v0. for En- Code and multi innuall target tokenizers.
3

---

Published as a conference paper at ICLR 2026
produced by processing the input sequence 8," instead of s, are similar to our target 10 (s,). We denote these new hidden states using our new embedding e* as H!! (s,-). As the two input sequence indices are not aligned due to replacing subtokens by single new token, define set of mapped positions (i.j) E _M(s,, Sy.). where i and j are the positions of the same token in 8,* and 87, respectively. Additionally, (Msr, only includes pairs (i.j) where position in 8,* would attend to t*. We subscripts such as (s+); to signify the hidden state at the j-th token position. We learn c' by minimizing the mean-squared error between hidden states for given target layer I:
min Bews
Σ
( - (s))s
(I)
Elie
((.)))((......))
In practice, we simply use the last layer's 's hidden state but analyze this choice in Section 5.3
3.3 FURTHER DETAILS
Retrieving relevant contexts for new tokens. Like other methods optimizing embeddings based on contexts, method also needs input sequences. Firstly, note that randomly sampling texts from a domain-specific or general corpus is inefficient for our goal of learning newly added embeddings: Typically, the new tokens will only make up small fraction, so most gradient updates will actually not affect the new input embeddings at all and simply learn to minimize their log- pog-robbabylity in the case of output embeddings. As we aim to have a fast method, we need a better approach. We propose two different approaches: (1) Our main approach is to simply retrieve snippets that contain our target tokens from a Nooainn-speciff or general corpus. This can be implemented efficiently using the algorithm proposed by AAh & Corasick (1975). Then we can truncate the snippets to small window around our target token to optimize computational efficiency. (2) For causal language models, we can simply generate snippets by prompting the model with our target token. We provide implementation details in [Apeendix C.8] In our main experiments, we focus on the first approach but note the availability of the second approach in case reference corpus is not available. In most cases of domain and language adaptation, the availability of such a corpus a reasonable assumption. Nevertheless, study ablation instead using the generative approach in /ecciin 5.3
Output embeddings. Since method backpropagates gradients back from the hidden states, we do not learn output embeddings with our distillation-bar based objective. In fact, this is not possible, as our new tokens are not part of the original model that serves as the "teacher". In practice, for learning output embeddings, we can simply add next-token prediction objective just for the output embeddings minimal computational overhead or freely combine our method with any other method for initializing output embeddings. Recent work suggests that input and output embeddings should in fact be treated differently (Nakash et al. 2025 Huang et al. 2025). Indeed, the Transformer architecture handle tokens that are input-only. Therefore, if not paired with additional initialization method for output embeddings, we have the choice of either adding new tokens only to the input embedding matrix or - for compatibility with existing frameworks setting their output embeddings to a vector of zeros.
Hyperparameters. We employ simple setup and use the AdamW (Kingma & Ba 2017 Loshchilov & Hutter 2019) optimizer for all trainable parameters. We set the batch size to op- timize throughput and run all experiments on Nvidia H100 80GB GPUs. We do not use weight decay and maintain constant learning rate with linear warmup. For fair comparison, we sweep for the best learning rate for all methods that require learning rate. Since our method is aimed to serve as an initialization rather than as full-scale further training, we restrict the number of example sequences to a maximum of 25 per target token and truncate to a context length of 50 tokens. These restrictions ensure that our method is quick to run, initializing 2,500 new tokens under 10 minutes. In Sectien 5.2 we additionally report experiments with a larger compute budget (100 snippets per token) and with continued training for vocabulary adaptation. We use the same data for all training-baat based methods, including baselines and variations of our method. We provide all details in Appendix C.
single GPU in
4

---

Published as a conference paper at ICLR 2026
4 EXPERIMENTAL SETUP
4.1 EVALUATION
Common use cases for adding new tokens pretrained model's tokenizer language and domain adaptation, especially for the biomedical domain (Poerner et al.) 2020 Gee et al. 2023 Hasan et al. 2024 Singh et al. 2024 Balde al. 2024). This is because it is particularly challenging domain with highly complex domain-specific terminology, which can serve as a benchmark to stress test embedding initialization methods. Therefore, we evaluate our method on collection of standard benchmarks in the biomedical domain (Pal et al. (2024) and add frequently occurring words as new tokens. Additionally, for a more in-depth evaluation of the quality of new representations provided by our methods, we prompt the adapted models to generate definitions for such new tokens and evaluate their quality with an LLM-as---Judee (Li et al. 2023). We also evaluate Token Distillation for language adaptation, choosing French as an exemplary language. We evaluate our method on set of multiple-choice benchmarks from FrenchBench (Faysse et al. 2025) and add frequently occurring words new tokens, as in the biomedical domain adaptation experiments.
Furthermore, to push the limits of embedding initialization methods, we move beyond domain and language adaptation and apply our method multi-word tokens, which have recently gained further interest (Gee et al., 2023 Liu et al.] 2025 Huang et al., (2025). We follow the zero-shot tokenizer transfer setting our evaluations (Minixhofer et al. 2024). Le., we evaluate initialization methods without further training on a corpus, as we want to directly judge the quality of the resulting representations. In In[Section 5.2, we additionally experiment with a short continued training phase after initializing new embeddings.
We select target tokens that actually occur frequently in the chosen benchmarks so that we can effectively judge the quality of newly generated embeddings. We select words that occur more frequently than a threshold and additionally ensure that all individual benchmarks are represented. We report the full methodology in Appendiix C.I For our multi-word token experiments, we prompt gpt -04 -mini-- maini-high to generate suitable candidates (full details in Appendix C.3). For our experi- ments in the biomedical domain, we retrieve contexts from ncbi puwbedd reference corpus. For For
our experiments on language adaptation to French, we use HuggingE wttaaaaaaeeaeeaeeaaaaaaaaaa our morti-word token experiments, we prompt the original models to generate sequences containing the new tokens. For the L.LM--a---Juddee evaluations, we use L.l a.1ama.... 1.maa.337777 3- Insto---sstuuu ruct as the judge model. We evaluate the general correctesss of the generated definitions to test the quality and completeness of the resulting representations as well as semantic similarity with the target model's output (Villegas et al. 2025). as inducing as little behavior change as possible can be an important desideratum.
4.2 MODELING
Considered base models. We conduct experiments using wide range of open-weight model checkpoints to ensure our results are not merely a function of peculiarities in any specific model's 's embedding space. In particular, we choose the following models: Mistral-7B--o OLM02-7B-11124-nnttmn
L.lama3. 3. 188 In the remain- der of the paper, we denote "-Instruct" variants with a simple "-i" suffix. Through this extensive evaluation, we study different model families, model sizes (3B, 7B, and 8B), instruction-tuned and base models, as well as models with separate input/output embeddings and tied (shared) embeddings between input and output. For experiments French language adaptation, we experiments with the Mistral-7B-v0
Llamaa38P, I,l T I.l 3. and 1,1 ama3. 3. 2-3B-Instrre
a.
and Llama3-88 (-Inst ruct) models and additionally consider Qwen -Ba--8-BB,,,,,, which has received more extensive multilingual training.
Baselines. For all results, we report the performance of the unmodified base model using the original tokenization. To establish a lower bound, we also report results for initializing the new token embeddings randomly from a normal distribution with a per-channel mean and standard deviation of the original embeddings (Hewitt) 2021). Furthermore, we report results for the commonly used method of taking the subtoken mean (Sachidananda et al., 2021 Koto et al. 2021 Gee et al. 2022) initialization for new token, which has been shown to perform similarly to more sophisticated initialization methods that also use a weighted average of existing embeddings (Minixhofer et al.
2024 Yamaguchi et al. 2024b).
5

---

Published as a conference paper at ICLR 2026
Mistol778
Elams--8--i
L.mm331-688
1amm3.1-884
Mo2---7--
Aug.
Original lokemization
70.6
61.3
66.5
Random
Howis 2021
51.860.5
39.380.5
6
1..600.4
NIP
nu (necssmms)
55.7a0.5
63.8e0.1
42.440.
51.5a0.66
33.5
Mean at.
63.8
63.4
66.9
54
58.0
60.3
NTP J.impinh 2r 20.8
6] 26014
65.660.3
65 320.6
56 020.5
58, (8.990.2
56 66166
61.3e03
58.7al16
63.0
2:17
[Smmiidere ara 20124
62.7
66.1
66.3
1okeh esccunzoonn
82.560.5
67.3
07.62l.3
07.340.5
71,029.2
30.2+1.0
63.1+8.2
01.220.3
64.6
TTkkas Disillation + NTP
B7 2401
76.7116
57 240.1
70.640.4
57 77.6001
62.240.1
59.5005
64.3
+ Tokmm Distillatonn aNTP
02 2203
07.628.4
67.8et1
67.458.4
70.9803
57.9
52.5526
61.2p11.
64.7
Table 1: Benchmark results on biomedical domain adaptation for different initialization methods. We report a macro-average + standard deviation over five random seeds on the tasks in the Open Medical-LLMM leaderboard (see SScciinn 4.I). The best initialization result for each model is given in boldface, while all results that are not significantly worse (one-sided Welch's is (-test with Bonferroni correction, p < 0.05) are underlined. Methods without #x.x are deterministic. -; We only report results for TTTT where pretrained hyper-networks are available. *: Our method(s).
Mistal 7B
Liuma3 SB
Llama3 3B
LiemaS.1
Limu3.1 SB
Liama3
Llama3.2 3B
OLMo2
Ang.
Coer
Sin
Coor
Sim
Corr
Cuer
Sim
Cozr
Sin
Original tnkemizztion
96.5
100,0
100.0
98.7
103.0
96.4
95.2
100.0
93.8
99.4
95.8
Randun Hexin
2021
0.0
0.4
0.0
0.0
0.0
0.2
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0,0
0.0
0.0
0.0
0.1
Mear
GB ee t 2012
2.5
4.3
16.8
16.2
17.5
20.7
25.6
25.6
25.2
28,3
15.2
15.2
12.1
14.6
20.1
24.3
16.5
18,5
NIP
zeleenmeert
26.8
30.3
33.2
39.5
54.4
43.4
28.5
15.0
41.4
50.0
19.7
20.7
18.6
21.1
22.5
25.6
25.1
33.5
NTP
amsioes et 20r5
49.8
56.2
45.3
49.6
65.4
58.0
38.6
05.6
60.2
69.9
51.6
56.4
50.0
58.8
52.1
57.0
53.0
99.4
2:T1
Meahaaofrr ot 20
63.5
68.6
69.5
73.6
70.7
50.3
Tiken upsiindd
79.7
85.2
72.7
76.3
79.7
91.0
75.8
30.1
$1.6
59.8
0.0
0.0
75.0
$3.0
83.8
89.5
68.5
74.4
Token Distiliznon + NTP
71.3
82.6
08.4
73.8
75.8
56.3
12.3
79.9
77.3
56.1
02.3
09.1
05.2
75.8
65.4
74.6
70.5
78.5
T.ken Diuiillation oNTP
92.0
71.7
75.2
80.9
91.6
$1.2
39.5
72.7
79.5
71.0
80.5
81.2
86.0
Table 2: Results for prompting models to generate definitions for newly initialized biomedical domain tokens. We report similarity with the original target model's definition (Sim) and correctness (Corr), both judged by Ll ama 3-705 Ins- ruct. We bold the best initialization result in each column. *: Our (s) For full judge prompting details, see Appendix C.7
Since our proposed method conducts a short optimization on a few reference sequences per new token, we compare against the common approach of training embeddings using causal language modeling using the same data. Specifically, we report results for "classic" embedding tuning using the next-token prediction objective denoted as NTP (tune all embeddings) - as well as masking updates to the original embeddings such that only new embeddings are optimized (denoted as just NTP), which corresponds to the method used by Lampinen & McClelland (2018). These methods use the subtoken mean as their starting point. A strong alternative method for obtaining embeddings for new tokens are neterrs specifically pretrained for this task. We report results for ZeTT (Minixhofer et al.) 2024) using their provided heper-network checkpoints.
Our method. As the initial embedding input into our method, we also use the subtoken mean. We investigate alternatives to this choice in Section 5.3] Our proposed objective can also be combined with the NTP- objectives. We report results for combining Token Distillation and NTP. Since the NTP and Token Distillation objectives have different scales, a simple addition is suboptimal. In fact, NTP is usually of larger magnitude, which leads it to overpower Token Distillation. Therefore, we also consider an "autoscaled" variant Token Distillation + aNTP where the NTP loss is scaled by sttopgrrad ent(1 olss__sss_ sttss_stte__eeeee i st lation loss_nts before summing. Here, stopp_grad ent: prevents gradient flow through the scaling factor o
5
RESULTS
5.1 MAIN EXPERIMENTS
Main results. We provide our main results on benchmarks in Table (biomedical domain adapta- tion) and Table (French language adaptation). In Table 2 we report our experiments on definition generation for newly added biomedical tokens (reporting similarity to the original target model's generation and correctness as judged by a larger LLM).. On average, our method outperforms all other baseline embedding initialization methods. In particular, Token Distillation shows superior results on benchmarks and in its ability to generate correct definitions, which is indicative of higher-quality
Multti-bbeective optimization is a rich field with extensive literature. We leave thorough investigation of alternatives such GradNorm (Chen et al. 2018) for future work.
6

---

Published as a conference paper at ICLR 2026
Mismal78B
LJmm33-88
Llama3-88-i
Qwen3-88
Avg.
Original tokenization
69.5
69.4
72.1
81.7
73.2
Random (Hewi 2021
51.0
46.1
51.5
62.1
52.7
Mean
56.3
58.4
61.7
69.6
61.5
NTP
Lampieen el al 2018
64.7
67.0
70.1
812
70.8
7c11
Minikhoter
66.1
61.3"
65.2'
Token Disitation
68.5
68.9
72.9
81.5
72.9
Table 3: Performance on FrenchBench multiple-choice tasks after adding new French whole-word tokens. f; For ZeTTT, only Mistral-7B has a multilingual-tune hyper-network, while Llama3-8B(-i) uses the general English+Code hyper-network.
representations. Additionally, Token Distillation also exhibits greater similarity to the original to- kenization behavior than all other methods, which substantiates our claim that distillation-hase objective is able to better approximate complex multi-subtoken interactions into a single token embedding.
Note that even with its very competitive performance, Token Distillation does not fully attain the level of the original tokenization, which benefits from the massive pretraining of the original model. This is expected in our zero-shot tokenizer transfer setting without any further training (see) Minixhofer et al., 2024), although remarkably, Token Distillation is able to outperform the original tokenization when adding French tokens to L. 3- without any training of the Transformer layers. We seek to directly investigate the embedding initialization quality and the advantages of our distillation- objective compared next-token prediction rather than the effects of further training. However, results can naturally be improved via further training: we argue that Token Distillation will provide the best starting point for such further processing. InfScciion 5.2 we additionally experiment with giving Token Distillation additional computational budget as well as moderate continued training phase, which does enable Token Distillation to closely match the original tokenization's 's performance while offering significant speedups due to reduced token counts. In the following, we analyze various aspects of the results in further detail.
Note that in Table 2| only for L1 ama 2-BBB, Token Distillation obtains results that are on par with random initialization. I. ama3. .2 (-i) are the only models in our lineup with tied embedding weights. Our objective does not explicitly enforce a bound on the norm of the new embedding, which in this case led to the failure mode of always generating a specific new embedding with very large norm. Note that this does not always happen for checkpoints with tied weights, as evidenced by the results for Llamaa. 2 3-- However, we additionally propose simple modification alleviate the occurrence of this issue by combining Token Distillation with the next-token prediction objective, which (implicitly) acts as regularizer on the output embedding norm. In support of our argument that subtoken attention distillation objective is superior to next-token prediction for learning new token embeddings, the combination via a sum of the two objectives (Token Distillation + NTP) generally yields worse results than Token Distillation alone. However, we can add a dynamic downweighting factor (see /Section 4.2) to the next-token prediction objective (Token Distillation + aNTP), which mostly alleviates the negative interference while keeping the regularizing effect.
Freezing original embeddings. Vanilla NTP-tuning of all embeddings (denoted as tune all em- beddings) yields disappointing results, even underperforming the subtoken mean initialization. We investigate this and observe a degradation of the original token embeddings. Note that for Token Distillation, we only optimize new token embeddings. Therefore, we also compare against the NTP baseline, where we similarly optimize only the new token embeddings. NTP outperforms NTP (lune all embeddings), supporting our analysis. However, even with this improvement, Token Distillation in turn still further surpasses NTP.
We note that masking gradient updates to the original embeddings during further training is not commonly done when adding new token embeddings, even during initial "embedding tuning" phases where all weights but the embedding layers are frozen. In these phases, the goal is to improve the initialized embeddings for new tokens or "adapt" the new embedding matrix to the existing Transformer layers (de Vries & Nissim 2021). When normally sampling from large-scale training corpus, the degradation of original embeddings will be less dramatic, as the number of new tokens per batch will be much lower. However, this is much less computationally efficient and at least initially still potentially harmful. We leave a further investigation of this as a practically useful direction for future work.
7

---

Published as a conference paper at ICLR 2026
Hyper-networks. ZeTT, which uses pretrained hyper-network, is the strongest baseline compare against. We only compare against ZeTT when a pretrained hyper-network by Minixhofer et al. (2024) is available" We first note that ZcTT yields quite impressive results, outperforming our optimized NTP baseline. Evidently, the pretraining of ZeTT's hyper-network has internalized a better embedding prediction than the iterative gradient descent optimization on selected samples we employ for NTP. Nevertheless, our proposed method Token Distillation outperforms even ZeTT without needing any hyper-network pretraining. It is important note that at inference time, hyper- network based methods such as ZeTT are actually faster than Token Distillation (and NTP--aaed baselines), since only a single forward pass through the hyper-network per token is required. However, they require expensive pretraining for each new model and - in some cases target domain. For example, ZeTT with hyper-network pretrained only on English naturally underperforms on language adaptation to other languages- as this is out of distribution from the hyper-network training, whereas the multilingually pretrained variant available for one of the models performs more favorably (sce Table 3). Also, we can actually use ZeTT- --g-errretde embeddings them using our method for even better results. We analyze this possibility in Section 5.3
R
starting point and further tune
5.2 COMPRESSION-ORIENTED TOKENS AND CONTINUED TRAINING
In our main experiments, we select new tokens based on frequently occurring whole words (yielding moderate sequence compression of 7-10%). In practice, however, vocabulary adaptation is often pursued for computational efficiency, which calls for selecting tokens that maximize sequence compression even if these tokens to initialize well (see our discussion in Appendix A). We therefore include two experiments that explicitly target this regime.
whole words and are therefore inherently more challenging
Compression-orientes tokens on French (Table 4). We repeat French vocabulary adaptation, but (i) increase the number of snippets per new token from 25 to 100 (4x), and (ii) select 5,000 new tokens using Adapti Vocab (Nakash et al. 2025). Importantly, Adapti Vocab does not restrict tokens to whole words, which increases sequence compression but also makes initialization harder (tokens more ambiguous and more depeedttee than whole words). As a result, init-only scores in this experiment are not directly comparable to our main FrenchBench results in Table 3| which use different (whole-word) token set. Adapti Vocab yields substantially stronger compression, reducing average token counts by about 20% on the evaluated FrenchBench datasets. Despite the increased difficulty of the token set, Token Distillation closely matches the original tokenization while providing the compression benefit.
Method
Llama- (A tokens)
Qwen3-SB (A tokens)
Method
Init only
Continued training
A tokens
Original tokenization
72.1 (0%)
81.7 (0%)
Original tokenization
47.4
46.9
0%
Suhtoken Mean
60.9 (-21 (-21%)
69.2 (-20%)
Subtoken Mean
36.2
45.4
-36%
NTP
66.5 (-21 %
79.1 (-20%)
40.6
-36%
Distillation
(x21%)
(-20%)
Token Distillation
44.2
47.5
-36%
Table 4: Performance on FrenchBench multiple- choice tasks after adding 5,000 new French to- kens selected via Adapti Vocab (Nakash et al.] 2025). Token Distillation and NTP use 100 snip- pets per new token.
Table 5: Performance on Arabic multiple-choice tasks after adding 1,000 new Arabic tokens to Mistral-7B selected via AdaptiVocal (Nakash ct al. 2025). Token Distillation and NTP use 100 snippets per new token.
Compression-orientes tokens + continued training on Arabic (Table 5). We next consider setting where added tokens come from a language that is very poorly supported in the source tokenizer. Vocab to Mistral-7B. reducing token counts on Arabic benchmarks by 36%. Note that the reduction of token counts by 36% corresponds to compute savings of up to 50%. due to the quadratic scaling of attention with respect sequence lengths (see (Appendixx B.4). Using the same increased initialization budget as in the previous experiment on French compression-oriente outperforms other initialization baselines ("Init only") but does not yet match the original tokenization.
We add the top 1,000 Arabic tokens selected via Adapti Vocab
tokens (100 snippets per new token), Token Distillation already
We then run a moderate continued-training phase for vocabulary adaptation based on the methodology proposed by Nakash et al. (2025), which trains the embedding matrices as well as the first and last
Minixhofer et al. 2024) do not provide checkpoint trained specifically for the "- Inst ruct" variant of 1.1 1m1maa3 8B but show that transfer from the hase version is possible.
8

---

Published as a conference paper at ICLR 2026
Mistol-78
Elams--8--i
1Jamx3.1688 1Jmm3.1-88.
Mo2-77B--
Original lokenization
97.8
Howi RUAT
0.0
7.4
Implien
$4.0
71.6
20.0
70.0
Ze11
[xmmmretee endle 20126
15.3
25.1
23.4
1okel Decandrnn
46.3
68.0
96.3
61.8
81.3
31.6
57.5
84.3
64.6
Tokes Desillatinn e NIP
48.3
68.0
K3.3
00.2
79.7
38.2
63.4
MI.9
65.5
Tikes Disiilation + oNTP
46.9
71.9
85.7
62.0
80.
57.2
68.4
83.7
66.6
Table 6: Correctness of generated definitions for newly initialized multi-word tokens (famous people.
places, entities, sayings and concepts) as judged by Ll ama3. Instruttt
Mistral77 Ulamad-S
Llamak
Llam332-39.
Tooeen Datillation (genemsted
67.1
67.0
57.1
Dutillation (retniveed
67,603
Table 7: Comparison of using data generated from the model instead of retrieving from a corpus (ncbi/ /pubmed). We only run single seed with generated data and report average performance on biomedical benchmarks.
Transformer layers. After this continued training, Token Distillation matches or slightly exceeds the original tokenization while retaining the substantial token-count reduction. The original tokenization is not able to benefit from this moderate continued training, likely due heavy over- wver-faamentttion.
5.3 IN-DEPTH ANALYSIS
Multi-word tokens. In Table 6 we report L.-M-as-a-Judeee results on the correctness of generated definitions. This tests the limits of our method, as the new tokens are now more complicated, such as "Software Development" or even common phrases such as "spill the beans". this setting, even using the original tokenization sometimes is not able to generate correct definitions. In turn, the gap Token Distillation to the original tokenization is also larger in this experiment. However, Token Distillation outperforms all other baselines by very large margin. ZeTT (Minixhofer et al. 2024) in particular struggles in the multi-word token setup, likely because is out-of-distribution from the hyper-network pretraining.
Retrieved vs. generated data. In Sectionn 3] we discuss generating contexts containing our tokens by prompting the model to generate some text containing the new token. In our main experiments, we instead retrieve relevant contexts from a corpus. We compare the two approaches in Table 7 In general, data generation performs slightly worse than retrieving contexts but is still competitive, outperforming next-token prediction objective on retrieved contexts and even pretrained hyper- works from ZeTT.. This highlights the applicability of our method even in scenarios where no corpus containing relevant contexts might be available. For example, we use context generation instead of retrieval in our multi-word token experiments in Table 6
Choice of initial embedding for Token Distillation. We find that the subtoken mean initialization as starting point yields better results than random initialization (see "Random + Token Distillation" in Table 8) We the subtoken mean for its simplicity. In settings where tokens have low lexical overlap with existing tokens, better results could be achieved via mapping-based initialization methods (e.g.. Minixhofer et al.) 2022 Dobler & de Melo 2023). If pretrained embedding prediction hyper-network is available, we can instead use these predicted embeddings as the starting point. We this experiment using ZeTT; this combination further improves performance compared using the subtoken mean (see Table 8). We also compare refining the hyper-networts embeddings with NTP. This greatly improves over NTP but yields only minimal gains over using the hyper-netwer embeddings themselves without further NTP.
Different distillation objectives. Our main proposed method Token Distillation computes the distillation objective based on a mean-squared error (MSE) of last layer hidden states. A natural question is whether we can also use the "traditional" knowledge distillation objective (Hinton et al.) 2015) based on the Kullback-Leinn
divergence (KL) (Kullback & Leibler 1951). We analyze this and report the results in Table 9 We additionally include computing the MSE between logits instead of hidden states. For the KL-based results, we include different variations of combining the objective with NTPP-based objective. In general, the choice of distillation objective does not.

---

Published as a conference paper at ICLR 2026
matter much. This is hardly surprising, as the logits are merely linear projection (potentially with an additional normalization layer) of the hidden state; the log-probabilities used for KL are then in turn merely additional log- normalization via log-softmax. Combining the KL-based results with NTP-based objectives follows similar trends as their combinations with hidden state MSF:-hased objectives. We opt for the MSE on hidden states for reasons: (1) In terms of implementation convenience, the objectives with a projection to logits over the vocabulary require a masking step for the "student" model, as the original "teacher" model does not have the new tokens in its vocabulary. (2) Choosing hidden states allows us to explore other layers than just the last one with potentially large computational speedups we explore this next.
Misiua-7B
Liams? SB
Llama--B--
Avg.
Original Tekesitation
64.5
69.8
70.6
68.3
60
Moss
Coare 2024
56.5
61.8
656
618
2cTl
62.7
66.1
64.3
65.0
(Mass-ua
61.5
+Oaon +) Thiee Ihsillatou
-Randnn + Tekes Disillatio
K3.5
62.5
65.3
67.3
68.0
65.9
66.3
66.6
62
Learring Toaee
+2rT1 Taken Diaillatoo
64.2
56.8
+Meen
Toles Disillatonn
+XeTT
Taken Disiilation
NTP
64.1
67.9
68.7
66.9
Table 8: Biomedical benchmark results with different starting points for Token Distillation. We report results for models where a pretrained hyper-network from ZeTT is available and con- sider the same single seed for all methods.
56
Target Laper
Figure 2: Average performance on biomedi- cal benchmarks with different target layers of Llama3-8B- -a-tt-----Isstccc for Token Distillation.
Target layer. our main experiments, we apply the Token Distillation objective on the hidden states after the last layer. However, the last hidden state might not be the most optimal, as it might be overspecialized for next-token prediction (Rogers et al. 2020 In Figure 2 we plot the average performance on our biomedical benchmark datasets using Llama 3B Instruct while varying the target layer for extracting hidden states for our Token Distillation objective. Indeed we do see a slight downward trend as we approach the last layer choosing different layer than the last one can further boost our already strong results. We repeat the analysis with L. amaa..... 18BB- Inst ruct in Appendix B.3 with matching results.
Interestingly, good results can already be achieved with very early layers, e.g., after layer four. This suggests that much of the subtoken-specific contextual information is already added to the residual stream in early layers, which echoes findings of previous work (Vulié et al. 20200; Elhage et al. 2022 Lad et al.] 2024; Nakash et al. 2025 Kaplan et al., 2025). This opens up an avenue to further speed up our method by terminating the forward pass after
target layer (and also saving that compute on the backward pass), which can be much faster if we select early target layers. We leave further exploration of this as an exciting direction for future work. For our main experiments, we choose to keep the last layer because this choice does not necessitate model-specific sweep over target layers. Also, the last layer is a principled choice, as guarantees that no subtoken interactions that are only modeled in later layers are excluded from the objective.
6 CONCLUSION
In this work, we have described a fundamental limitation of many existing embedding initialization methods, which only exploit knowledge stored in a model's embedding matrices, whereas much of the knowledge about token compositions actually resides in the Transformer layers (Elhage et al. 2022; Lad et al. 2024). To address this limitation, we have proposed Token Distillation, which systematically incorporates this knowledge via distilling contextual information from hidden states. Experimental results confirm that our method not only clearly outperforms methods aggregating embedding matrix rows but also yields better performance than training embeddings using traditional causal language modeling well as hyper-networks extensively pretrained for new token embedding prediction [Miixhoffer et al.) 2024). In future work, we believe research into embedding initialization should move beyond aggregation of information stored in the embedding tables. In particular, a more localized identification of subtoken contextualization (e.g. specific attention heads that aggregate subtokens) is promising direction for a more targeted distillation objective.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
We thank the German Federal Ministry of Research, Technology and Space for their compute grant through the project =Kl-Servicezentrrm Berlin Brandenburga (161S22092). Konstantin Dobler further thanks the European Laboratory for Learning and Intelligent Systems (ELLIS) PhD program for support. The research was supported by a research grant (VIL53122) from VILLUM FONDEN, and by the European Union's Horizon 2020 research and innovation program under grant agreement No. 101 135661771 (TrustLLM).
REFERENCES
Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David R. Mortensen, Noah A. Smith, and Yulia Tsvetkov. Do All Languages Cost the Same? Tokenization in the Era of Commercial Lan- guage Models, May 2023. /ttp/ttt/////arav.orro//aa////// [es].
13707 arXiv:2305.13707
Alfred V. Aho and Margaret J. Corasick. Efficient string matching: An aid to bibliographic search. Commun. ACM, 18663333,-340 June 1975. ISSN 0001-0782. doi: 10. (0.1145/360825.360855. URL
http::/doi. acm. http.///0///0111133333333333
Mehdi Ali, Michael Fromm, Klaudia Thellmann, Richard Rutmann, Max Lübbering, Johannes Level- ing, Katrin Klug, Jan Ebert, Niclas Doll, Jasper Buschhoff, Charvi Jain, Alexander Weber, Lena Ju- rkschat, Hammam Abdelwahab, Chelsea John, Pedro Ortiz Suarez, Malte Ostendorff, Samuel Wein- bach, Rafet Sifa, Stefan Kesselheim, and Nicolas Flores-Herr. Tokenizer Choice For LLM Training: Negligible or Crucial? In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Findings of the Association for Computational Linguistics: NAACL 2024, pp. 3907-3924, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10. 1/001 findings-naacl.222 URL httpps::/aclaattholllaaa
org 2024 wtt.0.2...iin...... -id.l-n--a--g-aaac222444-
Mikel Arterxe, Sebastian Ruder, and Dani Yogatama. On the Cross-lingual Transferability of Monolingual Representations. Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4623-4637, Online, July 2020. Association for Computational Linguistics. doi: 18.3vv53/v1/ a02in.cl-main.42) URL httpp:://act antho ogy. wwwoorgg/00000 ac mmain.... ii.in.. 4217
Gunjan Balde, Soumyadeep Roy, Mainack Mondal, and Niloy Ganguly. MEDVOC:
Vocabulary Adap- tation for Fine-tuning Pre-trained Language Models Medical Text Summarization. Proceedings of the Thing-7hhixdlnternattonnd Joint Conference on Artificial Intelligence, pp. 6180-6188, August 20t996///ccai2222/6888 URL https: :/s::://wwwe //wwww. ijcai .ar.ii.rr//ae proceed ings/ 2024/6833 Conference Name: Thirty-Third International Joint Conference on Artificial Intelli- gence (JCAAI-24) ISBN: 978 9781956792041 I Place: Jeju, South Korea Publisher: International Joint Conferences on Artificial Intelligence Organization.
2024. doi:
Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Alham Fikri Aji, Pawan Sasanka Ammanamanchi, Sidney Black, Jordan Clive, Anthony DiPofi, Julen Etxaniz, Benjamin Fattori, Jessica Zosa Forde, Charles Foster, Jeffrey Hsu, Mimansa Jaiswal, Wilson Y. Lee, Haonan Li, Charles Lovering, Niklas Muennighoff, Ellie Pavlick, Jason Phang, Aviya Skowron, Samson Tan, Xiangru Tang, Kevin A. Wang. Genta Indra Winata, François Yvon, and Andy Zou. Lessons from the Trenches Reproducible Evaluation of Language Models, May 2024. URL.pttp://axxxv.... wwt.:/wwwww..oaeooooooooooooeoee abs
http//bs//44... 2405. 14782 arXiv:2405.14 14782 [cs].
Zhao Chen, Vijay Badrinarayanan, Chen- Yu Lee, and Andrew Rabinovich. GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks, June 2018. URL http://arxiv.co org/ htsh//bb//l111 02257 arXiv:1711.0222 [cs].
Wietse de Vries and Malvina Nissim. As Good as New. How to Successfully Recycle English GPT-2 to Make Models for Other Languages. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.). Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 836-846. Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/ 2021. findings-acl.74, URL https://acclantho ogy org.org/ findi ngs-acll 747

---

Published as a conference paper at ICLR 2026
Konstantin Dobler and Gerard de Melo. FOCUS: Effective Embedding Initialization for Mono- lingual Specialization of Multilingual Models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference Empirical Methods in Natural Language Pro- cessing, pp. 13440-13454, Singapore, December 2023. Association for Computational Linggss- tics. doi: 1118866///12222.mmmmmmmm emr.p maimmmnmmin... 8297
URL httpp::://allaannoooogggo///////
C. M. Downey, Terra Blevins, Nora Goldfine, and Shane Steinert-Threlkeld. Embedding structure matters: Comparing methods to adapt multilingual vocabularies to new languages, October 2023. URL httttttpp/aaiii/ org/ /209 04679 arXiv:2309.0467. [cs].
Ali Elfilali, Hamza Alobeidli, Clémentine Fourrier, Basma EI Amel Boussaha, Ruxandra Cojocaru, Nathan Habib, and Hakim Hacid. Open Arabic LLM Leaderboard.
Co/
Li Ieeerbbbara 2024.
Nelson Elhage, Tristan Hume, Catherine Olsson, Neel Nanda, Tom Henighan, Scott Johnston, Sheer HIShowk, Nicholas Joseph, Nova DasSarma, Ben Mann, Danny Hernandez, Amanda Askell, Kamal Ndousse, Andy Jones, Dawn Drain, Anna Chen, Yuntao Bai, Deep Ganguli, Liane Lovitt, Zac Hadde Jackson Kernion, Tom Conerly, Shauna Kravec, Stanislav Fort, Saurav Kadavath, Josh Jacobson, Eli Tran-Johnson Jared Kaplan, Jack Clark, Tom Brown, Sam McCandlish, Dario Amodei, and Christopher Olah. Softmax linear umits. Transformer Circuits Thread, 2022. htmm.
https:/traasfforer-cciiiiss wwt.2002/sohu///m.....mtm hwtp://eeee..rooiisooo////////ee
Manuel Faysse, Patrick Fernandes, Nuno M. Guerreiro. António Loison, Duarte M. Alves, Caio Corro, Nicolas Boizard, João Alves, Ricardo Rei, Pedro H. Martins, Antoni Bigata Casademunt, François Yvon, André F. T. Martins, Gautier Viaud, Céline Hudelot, and Pierre Colombo. CroissantL.LM: A truly bilingual French-Fnglish language model, 2025. URL https:: settps://arxii.or///a/ ors/
2402. 00786
Leonidas Gee, Andrea Zugarini, Leonardo Rigutini, and Paolo Torroni. Fast Vocabulary Trans- fer for Language Model Compression. In Yunyao Li and Angeliki Lazaridou (eds.), Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track, pp. 409-404444444 416, Abu Dhabi, UAE December 2022. Association for Computational Linguis- tics. doi: 10. 10.11653/v1/2022.210
URL pttp:ssss//aalaattoollo...ore/ee
emmr innusiinttt....m 417]
Leonidas Gee, Leonardo Rigutini, Marco Ernandes, and Andrea Zugarini. Multi-word Tokenization for Sequence Compression. In Mingxuan Wang and Imed Zitouni (eds.), Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing: Industry Track, pp. 612- 621, Singapore, December 2023. Association for Computational Linguistics. doi: 18.13////
2023. www.emmmlliiinurreee URL http:: httptpps://ccc.aaaooooooooo. www.hhhoolll....//2///e http:/222...mmm1nnnnm--aa--eeeee 58/
Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, and Mor Geva. Patchscopes: A unifying framework for inspecting hidden representations of language models, 2024. URL
06102
Charles Goddard and Fernando Fernandes Neto. Training-fres tokenizer tranpaantion via orthogo- nal matching pursuit, 2025. URL httLtttts//arxitti www.oos///o...ooooooooo........a htrggaab//506..00 2206. 06607
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Farcia-Oland Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron,
12

---

Published as a conference paper at ICLR 2026
Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock. Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Çelcbi, Patrick Alrassy, Pengchuan Zhang. Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly. Ross Taylor, Ruan Silva, Rui Hou, Rui Wang. Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang. Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman. Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do. Vish Vogeti, Vítor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpiere Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisemman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer. Cynthia Gao, Damon Civin, Dana Beaty. Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Due Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuae Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern Grant Herman. Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan. Hakan Inan, Hamid Shojanazeri. Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Elena Veliche Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Japn-Bap Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang. Joe Cummings, Jon Carvill Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev. Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White,
13

---

Published as a conference paper at ICLR 2026
Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner. Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Raf Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang. Sneha Agarwal, Soji Sajuyighe, Soumith Chintala, Stephanie Max. Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury. Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kochler, Thomas Robinson, Tianhe Li. Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihaileseu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaojian Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia. Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang. Yossi Adi, Youngjin Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary De Vito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. The Llama 3 Herd of Models, November 2024. URL http::/ http://axxvv.rr//babbe /200 21783 arXiv:2407.21783 [es].
David Ha, Andrew M. Dai, and Quoe V. Le. HyperNetworks. February 2017. URL https
7/ //openreview..o net/ httpa//vwweeet./ffomoallooooooo. "tte:/oooommiidrxaccce.
Abul Hasan, Jinge Wu, Quang Ngoc Nguyen, Salomé Andres, Imane Guellil, Huayu Zhang, Arlene Casey, Beatrice Alex, Bruce Guthrie, and Honghan Wu. Infusing clinical knowledge into tokenisers for language models. 2024. doi: 10.48550/ARXIV.4400 143.112.....3.. URL https://arxiv.. /ttp///xxivv.r//////
abs 1 4312 Publisher: arXiv Version Number: 1.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring Massive Multitask Language Understanding, January 2021. URL httpp 03300 arXiv:2009.033000 [cs].
//axxiv.orr/abs//20000
John Hewitt. Initializing New Word Embeddings for Pretrained Language Models, 2021. URL html Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the Knowledge in a Neural Network, March 2015. URL/httt://aaxxi http://rrggr/aab//reeeeeee
abs abs//103. 055111 arXiv:1503.0253) [stat].
Hongzhi Huang, Defa Zhu, Banggu Wu, Yutao Zeng, Ya Wang, Qiyang Min, and Xun Zhou. Transformer: Vocabulary is Generally Worth Scaling, January 2025. URL
http://www...///////50...011 16975 arXiv:2501.1693 16975 [cs].
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux. Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7B, October 2023. URL httpp 06255 arXiv:2310.66R2. [cs].
Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams, September 2020. URL http::/aaxxi..org///e/ arXiv:2009.10 1308 [cs].
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W. Cohen, and Xinghua Lu. PubMedQA: Dataset for Biomedical Research Question Answering, September 2019. URLhtpp://aax org/aba bog/abs/19033 06146 arXiv:1909.06110 [cs].
Guy Kaplan, Matanel Oren, Yuval Reif, and Roy Schwartz. From tokens to words: On the inner lexicon of L.MMs. In /3th International Conference on Learning Representations, ICLR 2025, 13th
14

---

Published as a conference paper at ICLR 2026
International Conference on Learning Representations, ICLR 2025, pp. 5401-5425. International Conference on Learning Representations, ICLR, 2025.
Diederik P. Kingma and Jimmy Ba. Adam: Method for Stochastic Optimization, January 2017.
URL http:/ttta/aitt
org/ abs/ 4412. 6980 1IXiv:1412.660 [cs].
Fajri Koto, Jey Han Lau, and Timothy Baldwin. IndoBERTweet: A Pretrained Language Model for Indonesian Twitter with Effective Domain- -opeii-Specifie Vocabulary Initialization. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pp. 10660-10668.
Online and Punta Cana, Dominican Republic, 2021. Association for Computational Linguis- tics. doi: 10.1800..00 18653/v 1/w011211111ee 1e2222..emllle. www...emeenlaa......
URL https: https: https////aaaaatoooy.oooooo
emn.p-ma in 833
Kullback and R. A. Leibler. On Information and Sufficiency. The Annals of Mathematiccal Statistics, 22(1):79-86, 1951. ISSN 0003-4851. URLhttps: /www. jstor wttto..orr//taalee/22200. Publisher: Institute Mathematical Statistics.
Vedang Lad, Wes Gurnee, and Max Tegmark. The Remarkable Robustness of LLMs: Stages of Inference?, June 2024. [es].
1 19384 arXiv:2406.199 19384
Andrew K. Lampinen and James McClelland. One-shot and few- learning of word embeddings. January 2018. http::/aaxxl...rrraab/////11 10280 arXiv:1710. 10280 [cs].
Seungyoon Lee, Scongtae Hong, Hyeonseok Moon, and Heuiseok Lim. Semantic aware linear transfer by recycling prented language models for cross-lingual transfer, 2025. URL https
//arxivv wwwpor/g/ab.//oooooo/e/e.// 2505. 1...11000 10945
Pietro Lesci, Clara Meister, Thomas Hofmann, Andreas Vlachos, and Tiago Pimentel. Causal 03149
estimation of tokenisation bias, 2025. URL/atps.//azi.v.0o/////
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Guirajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. AlpacaEval: An Automatic Evaluator of Instruction-folloviin Models, May 2023. URL hnttps::/giittbb..oom// date: 0023-055255T09:3:2225
lab/a aabbaappa aplcapaaaeea _val original-
Alisa Liu, Jonathan Hayase, Valentin Hofmann, Sewoong Oh, Noah A. Smith, and Yejin Choi. SuperBPE: Space Travel for Language Models, April 2025. URL htRl/tttp://arx..orr/////// 2503.1342. arXiv:2503.1 13423 [es].
Yihong Liu, Peigin Lin, Mingyang Wang, and Hinrich Schütze. OFA: A Framework of Initializing Unseen Subword Embeddings for Efficient Large-scale Multilingual Continued Pretraining, March 2024.
wwtp:/vvvvrvvaoooooo abs /2311 C8849 arXiv:231 1, 1.88499 [cs].
Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization, January 2019. URL arXiv:171 1.05101 [es].
http::/arxx org/ hwtp://b///11/1
Kelly Marchisio, Patrick Lewis, Yihong Chen, and Mikel Artetxe. Mini-Model Adaptation: Efficiently Extending Pretrained Models to New Languages via Aligned Shallow Training. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: 2023. pp. 5474-5490, Toronto, Canada, July 2023. Association for Computational org/
Linguistics. doi: 10.18653/w1/202 1/203 wwt.2222...nnnss--.oi. findinngs-acl.33 ac1.338. URL https: http:://ccaantthhhc
2023. f findings-adeec .388
Benjamin Minixhofer, Fabian Paischer, and Navid Rekabsaz. WECHSEL: Effective initializa- tion of subword embeddings for cross-lingual transfer of monolingual language models. In Marine Carpuat, Marie- Cathie-Cateeineeeee de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Proceed- ings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pp. 3992-4006, Seattle, United States, July 2022, Association for Computational Linguistics. doi: 10.18653/v www.022n...annnnnmmmmmm 2933. URL https::/aclanthhooogg ww..org/////..... 2022. 2 naacl ma i...in.2220 293/
Benjamin Minixhofer, Edoardo Maria Ponti, and Ivan Vulić. Zero-Shot Tokenizer Transfer, May 2024. URL httttttp//axiiv wewpiiiorrrrr//ee org.orggaaaaa /2405. 240020080088333 arXiv:2405.07883 [cs].
15

---

Published as a conference paper at ICLR 2026
Vladislav Mosin, Igor Samenko, Alexey Tikhonov, Borislav Kozlovskii, and Ivan P. Yamshchikov. Fine-Tuning Transformers: Vocabulary Transfer. Artificial Intelligence, 317:103860,, April 2023.
ISSN 00043702. doi: 10.1016/...... aat.....ttt.2223.033333 14569 1rXiv:2112.1 14569 [es].
httt:: wwtp://w.rgg./a..oeooooo
Nandini Mundra, Aditya Nanda Kishore, Raj Dabre, Ratish Puduppully, Anoop Kunchukuttan, and Mitesh M. Khapra. An Empirical Comparison of Vocabulary Expansion and Initialization Approaches for Language Models, October 2024. URL http::://axxii.orr//bbb...
05841 arXiv:2407.0S5. .05841 [cs].
Itay Nakash, Nitay Calderon, Eyal Ben David, Elad Hoffer, and Roi Reichart. AdaptiVocaby En- hancing LLM Efficiency in Focused Domains through Lightweight Vocabulary Adaptation, March 2025. URL httt:://arxta www.wworrraa..so...e.. /2033. 19693] arXiv:2503. 19693 [cs].
Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo. Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark. Pradcep Dasigi, Nouha Dziri, Michal Guerquin, Hamish Ivison, Pang Wei Koh, Jiacheng Liu, Saumya Malik, William Merrill, Lester James V. Miranda, Jacob Morrison, Tyler Murray, Crystal Nam, Valentina Pyatkin, Aman Rangapur, Michael Schmitz, Sam Skjonsberg, David Wadden, Christopher Wilhelm, Michael Wilson, Luke Zettlemoyer, Ali Farhadi, Noah A. Smith, and Hannaneh Hajishirzi. 2 OLMo 00656 arXiv:2501.0063
Furious, January 2025. URL htllhttp//aaaee wwtexxvv.rrg/ee [es].
Malte Ostendorff and Georg Rehm. Efficient Language Model Training through Cross-Lingual and Progressive Transfer Learning, January 2023, htttatp.//aaxii...rraa////222110 09626 arXiv:2301.0963 [cs].
Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. MedMCQA: : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering, March 2022. URL
Distp:/arrDin htrg/abs/2240 2203. 14371 AAIv:2203.11377 [cs].
Ankit Pal. Pasquale Minervini. Andreas Geert Motzfeldt, and Beatrice Alex. open- https://uggiiggaacee co/ co/spaces/ openlliescciencess//ooo mediccaleediaalllll im_ Leaderboard 2024.
liffeciienceaiooeeneeeeee meeiaiioooppeemimaaaaaaeaaa 1lm leaderboard.
Yuval Pinter, Robert Guthrie, and Jacob Eisenstein. Mimicking Word Embeddings using Subword RNNs. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel (eds.), Proceedings of the 2017 Con- ference on Empirical Methods in Natural Language Processing, pp. 102- 112, Copenhagen, Den- mark, September 2017. Association for Computational Linguistics. doi: 10. 10.18653//1/D17-10000
URL htattppss/acaanntooooo.ooo
Nina Poerner, Ulli Waltinger, and Hinrich Schütze. Inexpensive Domain Adaptation of Pretrained Language Models: Case Studies Biomedical NER and Covid-19 QA. In Trevor Cohn, Yulan He, and Yang Liu (eds.), Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1482-1490, Online, November 2020. Association for Computational Linguistics. doi: 10. 1e111865//1/2222/ http://///....nnnnnnnnnee/mee/e/ findings-ennlp.132. URL httpss/////.aaahhooogoy.oo
Hindings-emnlpp 134/
François Remy, Pieter Delobelle, Hayastan Avetisyan, Alfiya Khabibullina, Miryam de Lhoneux, and Thomas Demeester. Trans- Tokenizaiinz and Cross-lingual Vocabulary Transfers: Language Adaptation of LLMs for Resource NLP, August 2024. URL http:tttp//axivv.rrg///o an arXiv:2408.04303 [cs].
2408. 04303
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A Primer in BERTology: What we know about how BERT works. November 2020. URL arXiv:2002.12327 [cs].
Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference
16

---

Published as a conference paper at ICLR 2026
Online, August Natural Language Processing (Volume 1: Long Papers), pp. l.ll-long.223 URL 2021. Association for Computational Linguistics. doi: 10. 10.18653/11/2211 a011.ncl-
https://accaanthooooryy
Vin Sachidananda, Jason Kessler, and Yi-An Lai. Efficient Domain Adaptation Language Models via Adaptive Tokenization. In Proceedings of the Second Workshop Simple and Efficient Natural Language Processing, pp. 155-165, Virtual, 2021. Association for Computational Lin- guistics. doi: 10211653/ sustainip-1.. 1.16
1.16. ResLLattss//aalaataaoooo..o
Timo Schick and Hinrich Schütze. Attentive Mimicking: Better Word Embeddings by Attending to Informative Contexts. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers). pp. 411444994444 494, Minneapolis, Minnesota. June 2019. Association for Computational Linguistics. doi: 10.18653/v
URL http://aaaaanttoooo.ooo www./19ggniii90 1048/
Timo Schick and Hinrich Schütze. BERTRAM: Improved Word Embeddings Have Big Impact on Contextualized Model Performance. In Dan Jurafísk,, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pp. 3996 4007, Online, July 2020. Association for Computational Linguistics. doi: URL http:tttppp///aaaaaccooooo..... http://laaacooooloor/o//22222220 acl.mmii...laainn
12663//1/20222allmmaaa 3687
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural Machine Translation of Rare Words with Subword Units, June 2016. URL httt:://arxiw.orr/abb///SS arXiv:1508.07909 [cs].
Pranaydeep Singh, Eneko Agirre, Gorka Azkune, Orphee De Clereq, and Els Lefever. EnerGIZAr: Leveraging GIZA++ for effective tokenizer initialization. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Findings of the Association for Com- putational Linguistics: ACL 2025, pp. 2124-2137, Vienna, Austria, July 2025. Association for doi: 10. (0.18/3//1/2025. http://ww//.....nnngge-ell/// fin. URL htt.ttss/aaaaaaahoaiaaaaeooo org org/ 2025. findings-acl....... act.
Computational Linguistics. ISBN 979-8- 76-25566-5556556
Priyanka Singh, Vladislav D. Mosin, and Ivan P. Yamshchikov. Vocabulary Transfer for Biomedical Texts: Add Tokens You Can Not Add Data, November 2024. URL /trtpp///rxxv.og//
abs/ 2208. 02554 urXiv:2208.0255: [es] version: 3.
Charlie Snell, Dan Klein, and Ruigi Zhong. Learning by Distilling Context, September 2022. URL
http:://axit v. htgg/abs//2000 15189 arXiv:2209.15: 15189 [es].
Ryan Teehan, Brenden Lake, and Mengye Ren. CoLLEGe: Concept Embedding Generation for 15332
Large Language Models, October 2024. URL
13X2v [cs].
Ke Tran. From English To Foreign Languages: Transferring Pre-trained Language Models, April 2020. UtL:ttp://aaaa org/ altp 07306 arXiv:2002.07306 [cs].
Danae Sánchez Villegas, Ingo Ziegler, and Desmond Elliott. ImageChain: Advancing Sequential Image-to- Text Reasoning in Multimodal Large Language Models, February 2025. http
l/arxiv.orge wwtp:ii.org/bb///55000 1 9409 arXiv:2502. 19409 [cs].
Ivan Vulić, Edoardo Maria Ponti, Robert Litschko, Goran Glavaš, and Anna Korhonen. Probing Pretrained Language Models for Lexical Semantics. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7222-7240, Online, November 2020. Association for Computational Linguistics. doi: 10. 1 1/20:///../2200000mmmm0mmmiiiii 2020. emn lamnlpmman ma n. 5867
URL ht potpps: / http::/:////aaaaaaooo.o...////// org/
Hai Wang, Dian Yu, Kai Sun. Jianshu Chen, and Dong Yu. Improving Pre- Trained Multilingual Model with Vocabulary Expansion. In Mohit Bansal and Aline Villavicencio (eds.), Proceedings
17

---

Published as a conference paper at ICLR 2026
of the 23rd Conference Computational Natural Language Learning (CoNLL), pp. 316-327 Hong Kong, China, November 2019. Association for Computatioonl Linguistics. doi: 10.18653/ v1/K19. 19-10.0 UBLLhtpp:///ccaan.ooooo....
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Perric Cistac, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-AAt Natu- ral Language Processing, October 2020. URL httts::::: /www ac web. org/ anthology 2020. emnlp-mnmpmmmmm demos...o..momoooooommm 6
Pages: 38-45 original-date: 2018- 10-1109911:::5::0022
Atsuki Yamaguchi, Aline Villavicencio, and Nikolaos Aletras. An Empirical Study on Cross- lingual Vocabulary Adaptation for Efficient Language Model Inference, September 2024a. URL
http://arxiy/ www.org////o..o ab////b//2/2// 10712 arXiv:2402.10711 12 [es].
Atsuki Yamaguchi, Aline Villavicencio, and Nikolaos Aletras. How Can We Effectively Expand the Vocabulary of LLMs with 0.01GB of Target Language Text?, September 2024b. URL http: 1477 11477 [cs].
7/arxiv. org. abs/aaasaaa 2406. 1..5..1111 11477 arXiv:2406.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang. Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao. Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang. Xinyu Zhang. Xuancheng Ren, Yang Fan, Yang Su. Yichang Zhang, Yinger Zhang, Yu Wan, Yugiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL
abwprg/abb/255555 09388
Qingcheng Zeng, Lucas Garay, Peilin Zhou, Dading Chong, Yining Hua, Jiageng Wu, Yikang Pan, Han Zhou, Rob Voigt, and Jie Yang. GreenPLM: Cross-Lingual Transfer of Monolingual Pre- Trained Language Models at Almost No Cost, May 2023, URL http:://aaxt:iii wwreiiivorrg/aee org/abs/ abs 2211.06993 arXiv:2211.000 1.06993 [cs].
A
LIMITATIONS
We have already highlighted and analyzed various choices and potential weaknesses of our method in Section 5.1 and [Section 5.3) Nevertheless, we find that our method outperforms even very strong baselines. However, now want explicitly summarize and expand the limitations of method.
Unknown tokens. Since the motivation for the distillation- based objective match the original model's behavior, our method is generally not applicable if the original model cannot meaningfully process new token. Note that this specifically limits the applicability of our method for adapting a model's vocabulary to completely unseen language, although in current pretraining regimes, many languages do end up being seen to a certain extent due to imperfect filtering mechanisms or deliberate inclusion.
Learning rate. Since our method involves gradient descent, we require setting learning rate. However, can be seen in Figuure 2 our method seems to afford wide range of viable learning rates that achieve good results, rendering the tuning of this hyperparameter less critical.
Only input embeddings. As discussed in /Seciion 3 our method only addresses the task of inducing imput embeddings, since our distillation objective is not applicable to output embeddings. Our method can be freely combined with any other method for obtaining valid output embeddings. Alternatively, new tokens can be added only in the input embeddings, since the Transformer architecture does allow for input tokens which do not occur in the output vocabulary (although this assumption is made in some popular implementations).
Short subword tokens. We find that our method does not work as well for lexically ambiguous tokens, i.e. tokens with many different meanings in different contexts. These often short subwords like <ed> or <fer> that occur in many words. This also means that our method is not ideal to initialize embeddings for entirely new vocabulary that contains such subwords. However, these tokens
18

---

Published as a conference paper at ICLR 2026
could be initialized using other methods (e.g. Minixhofer et al. 2024), while our method is applied to the rest.
Impact of chosen reference snippets. The reference corpus for retrieving contexts for training with the distillation objective can bias the resulting learned embeddings. Consider the word "tender", which in general means gentle soft, but in the financial domain describes a specific type of buy offer. Note however that this can also be beneficial, as the resulting representations are now specialized to the target domain. Additionally, if such an effect is undesired, we can use our proposed data generation scheme via prompting the model to generate contexts containing the new word. This can be interpreted as distillation training on samples from the original model's learned data distribution of samples containing the new word.
B ADDITIONAL RESULTS
B.I DIFFERENT DISTILLATION OBJECTIVES
We report results for using different distillation objective variants in Table 9 hidden state logit-based distillation well the differences between MSE and KI. based distillation is not significant. However, our chosen approach of hidden state distillation using MSE has the additional advantage of allowing the use of earlier hidden states, offering significant speedups (see Figure 2 and Figure 3).
The difference between
Mistral 70
Llamad SD
Llama3 8D
Llama), 1 3D
Llama3. I 8D
Llama3 2 3D
Llama3 2 30
OLMs2 7B
Avg.
* Token Disiiliatin (Logits)
63.0en.4
67.2a03
07.5e0.3
67.3a0.2
70.7a0.5
57.4e03
62 Ne0.
00.2e0.
64.5
* Tikken Dictillaton- - NTP
63.0-1
67.2eu
66 7e1
67. 2-133
20.6-114
57 660.3
62. 240.3
59 Sell
64.3
"Token Distillation GNTP
62.8e0
67.6pn44
07.8
67.429.4
70.9
57.9e8.1
02.5al
61.2e0.2
64.7
* Token Distillation
62 5
67.3002
67.3
71.0em1
56.2e1.9
63.1e9.
21
64.5
Token Disuillaion (KL)) NIP
62.3el
66.7an2
66-469.3
66.7811
70.120.4
57.3e02
62 1a06
19.7a03
63.9
Token Disiiliation (KL) oNTP
66.261
70.5a0.
57.5005
02.5e0.2
00.5el1
64.3
"Tiken Diviilation (KL.)
63.0.02
67.3.02
07.200.2
67.1.0.2
70.8.0.5
57.52044
62 720.0
60.5188
64.5
Table Comparison of different variations of our distillation objective with results on biomedical benchmarks. We bold the best result in each column and underline all results that are not significantly worse (one-sided Welch's I-lest with Bonferroni correction, p < 0.05). *: Our method(s).
B.2 RESULTS FOR "TOKENS TO WORDS" (KAPLAN ET AL..
2025
We report results for the method from "Tokens to Words" Kaplan et al. 2025) in Table 10 and Table IT alongside our method and other baselines. Note that the "Tokens to Words" method only initializes embeddings for tokens where a PatchScopes (Ghandeharioun et al. 2024) extraction using special prompt identify hidden layers is successful. It is difficult apply their method if the extraction is not successful. To give the fairest assessment of the method, we do not add these "unsuccessful" tokens the vocabulary when using "Tokens to Words". However, this likely
Mistral 7B
Liama3 8D
Liama3 3B
Liama..1 8B
Llana3.1 I SD-1
Llama3.2 3D
Liama3-2-50 2:38+
GLMc2 7B-1
Ang.
Original Inkonizaiion
64.5
69.8
20.6
69.2
72.3
60.1
64.4
61.3
66.5
Random
Hewin 2021
57.0e116
58 8a1.5
60
29.3200.
62 620
51 31.fzt14
55.5g032
53.920.6
57.5
NIP loo mar.erraaoottt...
55.72115
03 kc1.)
63.8a11
60.
51.5a115
#5.5g0.5
58.060.3
5938
Mean
Gie eca 1021
58.3
63.8
63.4
53.7
66.9
54.2
49.4
58.0
6088
Toben 25 Rondi 15705n
1025
(61.7)
(64.3)
(65.0)
(83.9) 9)
106.0 6)
(52.3)
(61.5)
(50.5)
(62.5)
NTP
Lumpiacs
61.2a044
66.060.5
92n
61.36055
58.7
Tokel Dismunn
66.7116
70.6
57.6am3
Token Disillation
3
50.2110
6122633
Token Disillation
57.
Table 10: Benchmark results on biomedical domain adaptation for different initialization methods. We report a -vverage + standard deviation over five random seeds on the tasks in the Open Medical-LLM leaderboard (see (Section 4.1). The best initialization result for each model is given in boldface, while all results that are not significantly worse (one-sided Welch's 's (-test with Bonferroni correction, < 0.05) are underlined. Methods without *x.x are deterministic. -: We only report results for 7eTT where pretrained hyper-networks are available. (...): Tokens to Words able to extract representations for only 60%-90% of added biomedical tokens. We still report results for completeness but note that this comparison overestimates the method's performance compared to the other methods shown. x: Our method(s).
19

---

Published as a conference paper at ICLR 2026
Llama3-88
Qwent-88
Avg.
Original tokenization
69.5
69.1
72.1
81.7
73.2
Random Hewini 2021
$1.0
46.1
51.5
62.1
52.7
Tokett
10 Worth (hapaan et 2025
(64.8)
(62.6)
(66.6)
(82.2)
(69 1)
NTP Lampinee
2015
61.7
67.0
70.1
81.2
70.8
*Takeh Dianarria
68.5
68.9
72.9
81.5
72,9
Table 11: Performance on FrenchBench multiple-choice tasks. 1 For ZeTT, only Mistral-7B has multilingual-tunel hyper-network, while Llama3-8B(-i) uses the general English+Code hyper- network. (...): Tokens to Words is able to extract representations for only 25%-50% of added French tokens. We still report results for completeness but note that this comparison overestimates the method's performance compared to the other methods shown.
overestimates the method's performance, since the model now uses more the original embeddings during testing, which have been tuned during massive pretraining. On average our biomedical domain adaptation results, roughly 10-20% of the total tokens did not have a successful PatchScopes extraction, varying across models (up 35% in the of Llama3.2-3B and 40% in the case of OLMo2-7B-i). On our French language adaptation experiments, this is even more severe, ranging from 50% 75% of unsuccessful extractions.
B.3 DIFFERENT TARGET LAYERS FOR TOKEN DISTILLATION
repeat the analysis of best target layer for Token Distillation from Section 5.3 in Fig ure 3 for Llama3. 1-1...11111 IB-Instruct instead of Llama3-88 Instruct. Our analysis for Llama -emmaa--BBIMMA Ins-IInttttt ruct in Section 5.3 applics.
71
Avg. Performance (%6
70
69
68
Learning
16:05
67
3e
5e.05 05
7e-05
66
9e:05
1e-04
3e-01
65
10
26
29.301
Target Layer
Figure 3: Analysis of varying the target layer for the Token Distillation objective. We report the average performance on the biomedical benchmarks for Llama3. 1----IIssrrucss
B.4 WALL-CLOCK SPEEDUPS FROM REDUCED OVER-TOKENIZATION
A note on downstream computational speedups. Over-tokenization is not only associated with reduced downstream task performance (Rust et al.) 2021
al. 2024) but also with increased computational cost due to longer sequence lengths (Ahia al. 2023 Yamaguchi al. 2024a). This computational cost is reduced when reducing over-tokenization by adding new (commonly occurring) tokens to the vocabulary. Importantly, for input embeddings, the reduction in computational cost completely independent of the actual quality of the embeddings for the newly added tokens (for given fixed sequence of text). The same computational speedups are achieved by using Random initialization and much better methods such as ZeTT (Minixhofer et al. (0224) or our proposed Token Distillation. Of course, the actual quality of any predictions made conditioned on sequence
20

---

Published as a conference paper at ICLR 2026
containing such new tokens does heavily depend the embedding quality. Therefore, in this work, we primarily focus on direct evaluation the quality of produced embeddings. However, we further provide empirical wall-clock validation of the efficiency gains next.
Empirical speedups. We empirically validate that reduced token counts translate into concrete wall-clock speedups. We compare batch processing of documents from the arb Arab split of / fineeb--2 between (i) the original Mistral-7B model and tokenizer and (ii) adapted model using Token Distillation with 1, 1,000 Arabic tokens selected via Adapti Vocab (Nakash) Vocat et al. 2025). We repeat the benchmark on both Nvidia H100 GPUs and Nvidia B200 GPUs. We use single GPU, process 8192 documents per run, and truncate documents 4k characters. For B200 GPUs, we use a batch size of 4 and for H100 GPUs, we use batch size of to avoid OOM. We report both inference-style forward-only runtimes and training-style forward+backward runtimes. Note how the compute savings can exceed the token-count reduction due to the quadratic attention cost in sequence length.
Accelerator
A tokens
A compute (fwd)
A compute (fwd+bwd)
A tok/sec (fwd)
A tok/sec (fwd+bwd)
H100 B200
-35%
-41.7%
-00.8%
+9.5%
+7.9%
-35%
-49.7%
-0.55%
+24.6%
+26.2%
Table 12: Measured throughput and wall-clock compute savings from reduced over-tokenization on Arabic documents.
C IMPLEMENTATION DETAILS
C.I TOKEN SELECTION FOR DOMAIN ADAPTATION
When using benchmarks to evaluate the quality of new token embeddings, it is crucial to ensure that the new tokens actually occur frequently enough to affect the performance if the new embeddings poor. See, e.g., the "Random" baseline for randomly initialized embeddings, which still performs better than random chance the benchmarks in Table
Therefore, we select all whole words from the chosen benchmarks that are not already existing tokens in the original vocabulary. We exclude tokens that include digits any of the characters in the following string in order to reduce noise:
(() ()<> ... . 1 "*t_"/\""
Additionally, to exclude very rare words that would not be commonly used as new tokens, we apply the following two filters: (1) We exclude words that appear fewer than five times across all benchmarks, and (2) we exclude words that occur fewer than times in a sample of 768,000 documents from domain-specific reference corpus (we ncbi /pubmed for biomedical benchmarks). This amounts to roughly 2,600 new tokens (varying per model). The full list of new words is available in our GitHub repository. For illustrative purposes, we provide a random sample of 100 added tokens here:
Igangrene, emphysema, Alkaline, pleural, psychoanalysis, paracervical, glancular, acidosis, reticular, two-factor, hyperplasia, bicarbonate, atopy, reactant, Riboflavin, lipoprotein, long- Chediak Chediak-Higashi, attenuated, pruritus, immuchistochenisttee
calcifications, staphylococcal, aspirin, bromide, mechanistic, cytosol, insufficiency,
arousc, thyroidectomy, Fallopian, primum, retrograde, febrile,
Cytomegalovirus, molar, cryptogenic, inorganic, deceleration,
Amikacin, gluconate, inhaler, Cushing, discordant, Rpstein-Barr,
intraccular, VIDDI, tomography, supraclavicular,, glucocorticoids, pyruvate, occluded, pigmented, neutrophil, irradiation, pharmaceuticals, cementum, LOI-cholesterol, postpartum, contractile, prostatectomy, deafness, septic, analgesics, tonsils, side-to-side, Prostaglandin, hoarseness, acuity, vena, dislocated, inciting, haloperidol, eczema, remnant, innervation, heartburn, elicited, yellowish, amylase, antihypertensive, ossification, redness, Giardia, placenta, trigeminal, Spearman, squamous, perioperative, abdominis, photosynthetie, transcribed, thynoma, condylar, furosemide, exudate, kJ, afebrile, non expooxde,,, unrestrained]
21

---

Published as a conference paper at ICLR 2026
C.2 TOKEN SELECTION FOR FRENCH LANGUAGE ADAPTATION
(TABLE 3
We use a similar approach as described in Appendix C.I but use BegggingFaceFW. finwweb-2 as domain-specific reference corpus for filtering rare words and add words that appear our chosen benchmarks from FrenchBench (Faysse et al., 2025). This amounts to roughly 11,600 new tokens (varying per model). We also provide random sample of 100 added tokens for illustrative purposes here:
lexcitation pelvienne, piblique, préserver, ratisse, coiffants, narine, mutuellement, sentira, ventilateurs, Réalisez, Accédez, relaient, inclinée, trébuche, structurer, pisseniit, contenants, débutant, annulation, mucus, accueillir, grossiers, sérums, æuf, Enfin, victoire, recevre?, balayer, vendez, officiel, agression, Fenser, suspendue, versez, socialement, scientifiques, chirurgical, Voulez-vous, doués, poursuivi, glacière, boitier, purifier, rocheuse, profitant, connaissent, partenaires, cuisent, paresseux, considérations, mèche, substitut, relaxante, célébrité, culpabilité, visuel, convulsions, correctours, en-dessous, alcooliséc, développeront, métres, fabriquer, imperméables, vapeur, granuleux, licr, nuage, creusez, opératicns, entourent, ressentant, extéricurs, intenter, métropolitaine, chutes, arrière-plan miettes, flaque, renforcera, entonnoir, r, acheté, brillant, courtier, kiwi, texturée, brouette, diminvera, débutants, plaire, quitté, appétit, abimer, équilibrée, avancez, pratiquent, inconvénient, restants, offriront]
C.3 MULTI-WORD TOKEN GENERATION
We prompted gpt--4-mini-hiie (as of April 7, 2025) with the following prompt:
Provide a list of entities, places, people, concepts, phrases or phrasings, idioms, or other terms that span multiple words, at least two and maximum five. Provide them in JSON format grouped by category and at least 100 per category. Ensure that you do not shorten coherent names just to fit them into the five word limit. "nm
We refined this prompt exclude common failure modes through trial-and-error. We include the list of resulting multi-word tokens in our GitHub repository.
C.4 OPEN MEDICAL-LLM LEADERBOARD EXPERIMENTS
For evaluation, we use the in-evaluationn -on-aaaiittttttn on-harness (Biderman et al. 2024) library at the version 0. We use 5-shot evaluation single Nvidia H100 GPU using ofloatle 16 precision. Specifically, we use the following command:
1m_eval model hf
model_args tasks
medncqa, mtt__ttttiimimmmtuaaattaaaao mmlu mmll_clinical Ltlniical_koooeddeee,
mmlu_college biology, www._moollgeeeeemeeooomomooolele www.imddiinneee iee. mmllumedicall cendiical_eeeetccs mullu_rrofessiona lasiaalmmmiiiieeee pubmedga
mmlu_colleeeebbolloyy,
num
gru__fewshot,
device cuda: 0
-batchhsize
This evaluates the task of the Open Medical-LLM Leaderboard (Pal et al. 2024 Jin et al. 2019 2020) Hendrycks et al., 202T; Pal al. 2022). All code and lock-file with specific versions are available in our GitHub repository.
C.5 FRENCHBENCH EXPERIMENTS
As for the Open Medical-LLM, Leaderboard Experiments, we the 1m-emmmlllle evaluationn harness (Biderman et al. 2024) library at the version 0 0.4.7 and use 5-shot evaluation on a single Nvidia H100 GPU with b bflooa fflooale 1 precision. We evaluate the multiple-choice tasks from FrenchBench (Faysse et al. 2025) and use letters (e.g., A, B, C or D) as the answer choices. We publish the task configuration our GitHub repository. Specifically,
the following command:
22

---

Published as a conference paper at ICLR 2026
1m eval
model hf
model_ pretrained-$KODEL
dtype="bfloat1"" ott
include_pati
tasks haanchh_benchhhbclgeecco french_bench, bench_ sarich_vocab_abaee
+
abcd, _te.ch bre.nct.enncaattuuuv22222
french srench__ench__topic_ooo nli abe,
french seacc___eenh__raiinaaape
abod
num fewshot
device cuda:0
batch_size 8
C.6 ARABIC EXPERIMENTS
We use the 1m-eval uat wwt.eevllaiiioonooooooeooooooeie hon-neeesesss (Biderman et al. 2024) library at the version 0.4.7 and use 5-shot evaluation on single Nvidia HI00 GPU with bflcat16 precision. We evaluate on multiple-choice tasks from the arabic lealerradooo letters (e.g. A, B, C or D) as the answer choices. We publish the task configuration in our GitHub repository. Specifically,
iight (Elfilali et al. (2024) and use
the following command:
1m_eval -modeel hf
-modelodaaaar args gretrained-SMODEL PATHEOCEEEE_LE,,E__C_EEE "type==bfloatt1s
include_path
httpa//aaeaaeeeeeereeeeeaaaaaaaa light_
tasks
num_fewshot
device cuda: \
-batchssiee 8
C.7 LLM-AS-A-JUDCH EXPERIMENTS
For generating definitions for particular new token, we use the prompt "The word <new token> defined as" following Teehan et al. (2024), where <new <new_token>> is replaced by the string repre- sentation of the new token (with whitespace stripped from the left side). We use greedy decoding. Our evaluation considers the model checkpoint corresponding to the best learning rate from the experiments described in Appendix fo4 each method.
LLM-as-a-Judde evaluations are not perfect see for example in Table 2 the similarity scores (Sim) of the target model (measuring similarity with the original tokemization), which should obviously always be 100%. but turn out to be 99.4% / 99.88 in the case of 2LMMo2-BB- and Mist ratl7B respectively. Nevertheless, these results (which we report as a sanity check on the judge quality) very close correct, which is encouraging.
We use met.a-ll ama / /ww.lama...... ama 3- 1-70B-Inst.rs.. as the judge model and run using bfloat 16 6 fuyer-paralle inference over two H100 GPUs using HuggingFace transformers We use prompts adapted from Li et al. (2023) and Villegas et al. (2025) for evaluating the general correctness of generated definitions well as the similarity with the original model.
For correctness evaluation, we use the following prompt:
E" <"**<|start. stt headeeeeeeer____ id/>>ydssssns end header id
You are pattern-follovtte
assistant that can only answer with "Yes" or "No", Your goal is to determine whether provided definition for
given word is correct. The definition should be on topic and speeific but does not need to be exhaustive. ill
Remember to answer with word either "Yes"
"No".
IH Instruction:
Determine the provided definition for the word 1strip() is correct.
### Definition (token. 1strip()
(linel "moodel_definition s)srrip(()
23

---

Published as a conference paper at ICLR 2026
### Is the provided definition correct, specific, and on tepic (Yes hatpa/attt//ooceeeeeedddaa/aaaaa
No)
For similarity with the original model's definition, we use this prompt:
E" <
hataasyysttteeeeeneeeeeeee end__hederr_id
You are pattern-follovine assistant that can only answer with "Yes" or
"No". Your goal is to determine whether predicted definition
conveys a similar enough meaning the ground truth definition
provided for a given
start_header id |a
Remember to
with word either "Yes" "No".
### Instruction:
Determine if the predicted definition conveys a similar meaning to the ground truth definition. The word is "(token.1strip()
### Ground truth definition:
(line "linel"targen "target_deiiintion" strriip())
### Precicted definition: mmooee_deriiiition"]]
IFI Does the predicted definition cenvey similar meaning to the ground truth definition (Yes
No)
C.8 GENERATION OF RELEVANT CONTEXTS
When generating contexts that contain a new token, we prompt the model with the beginning-of sequence (BOS) token followed by the textual representation of the new token. Using the new token <palatable> as an example, our prompt is then: "
 palatable". Note that add prefix space after the BOS token 
. We then sample N new sequences of length 7,, where N and I, are the same in the settings we use for context retrieval from an existing corpus (in this work we use N 25 and L 50). We sample using the standard sampling settings of each model from the trans formersrers (Wolf et al. 2020 library.
C.9 ISOLATED PER-TOKEN OPTIMIZATION JOINT OPTIMIZATION
When implementing Token Distillation, we have the choice of performing the optimization process for each new input embedding separately or simply allowing updates all new input embeddings the same time.
For the main experiments in this paper, utilize the joint optimization approach. However, also implemented the isolated optimization of each new embedding for initial experiments. In this setting, gradients are only propagated to single input embedding (corresponding to the target token of the retrieved snippet), even if other new target tokens also occur in that snippet.
In our exploratory experiments, we found that the joint approach yields slightly better results and conjecture that this because we are able to have more total gradient updates per target token due to co-occurrence in particular snippet. As an added benefit, the joint approach has less implementation complexity. However, when most tokens are actually new tokens (such as in our experiment with adaptation of eestrallB-vo 1 Arabic in (Section 5.2), the separate approach should be preferred because the student model mostly sees "noisy" new embeddings instead of just a single one per sequence.
C.10 HYPERPARAMETERS
We use Adam Kingma Ba 2017 Loshchilov Hutter 2019) optimization for all trainable parameters (the new token embeddings) with a batch size of 16, which maximizes throughput our used Nvidia H100 80GB GPUs. We do not incorporate any weight decay and maintain a constant learning rate schedule with linear warmup for the first half of the training steps. For fair comparison, we sweep for the best learning rate for each method that requires a learning rate. We use the same data for all training-based methods, including baselines and variations of our method.
24

---

Published as a conference paper at ICLR 2026
C.11 CONTINUED TRAINING HYPERPARAMETER FOR ARABIC VOCABULARY ADAPTATION
For the continued- -tainning setting in Table based on Adapti Vocab (Nakash et al. 2025 We train the embedding matrices as well as the first and last Transformer layers. We use an effective batch size of 128 sequences of max length 2,048 and train for 1,000 steps (about 260M training tokens). We use Adam W with learning rate of 5 10-5, 100 warmup steps, and cosine decay.
we follow a lightweight vocabulary adaptation protocol
C.12 MODELS
We use the models Mistral-7B-ye (Jiang al. 2023), 0LMo2-7B-1124-Ins---
(OLMo et al. 2025). )
Llama3. 18..1-8B-11e Instttuttt L-am Llama --aa-------nssutcc
1lama3.1-885
Llama3. 2-3B, Llama3 .aaaa..2----1a-.. I-BB-Inttttt (Grattafion)
et al. 2024) and Qwen3 8---888-BBs Base (Yang et al. 2025).
D COMPUTE BUDGET & RUNTIME
We report the average training time for Token Distillation well as NTP for initializing the new biomedical domain tokens per model in Table 13 We also report the number of new tokens per model. These differ slightly, as we build initial fixed candidate list, which is then filtered against the existing vocabularies each model.
Additional computational budget. Since Token Distillation includes an additional forward pass through the model for generating "teacher" hidden states, do see this overhead reflected in the times. A forward pass can be approximated as taking one third the computational cost of a complete forward-backwa pass which roughly matches our measured demonstrated in [Section 5.3 - Token Distillation can be significantly sped up while potentially even improving results by choosing an earlier target layer than the last one, yielding even faster run times than NTP.
times. Note however that as
However, our main version of Token Distillation is slightly more expensive than NTP- even though both have the same data budget. Thus, we also run the baseline NTP with an additional epoch to compensate for the additional teacher forward pass that Token Distillation uses. Note that this fact significantly overcompensates for the additional budget Token Distillation uses. We report the results in Table 14 NTP is not able to take any major advantage of an additional epoch, yielding similar results to using single epoch. Investigating the loss curves, find that the training converges towards the beginning of the second epoch F
We believe that this is at least partly due to the fundamental limitation of next-token prediction for our task of learning single new embedding for an already pretrained model. Instead of matching model behavior between seeing the single new and the multiple original subtokens, NTP simply adjusts weights to maximize the likelihood of the given sequences. Consider <new_tokens is football player", where we want <new token> represent <Messi>: NTP will only be able to learn generic embedding from this sequence that will capture semantics from many different (American football and soccer) players. Our proposed distillation objective Token Distillation however is able to learn more specific representation, as it has access to hidden states from the sequence "Messi is football player", which capture more specific details necessary for a good representation.
<new_token>
To demonstrate, we also run Token Distillation for additional epoch. Token Distillation is in fact able to take advantage of second pass over the data, in some case with major improvements in the benchmark results. This illustrates the richer training signal that our distillation-basel objective provides. Remarkably, for OIMo2-7B-1, Token Distillation now is actually able to match the original model with the original tokenization while instead using the new tokenization.
Token Distillation versus hyper-networks. Comparing the computational budget with ZeTT Miinihoofer et al. (024)) ZeTT sees 3.2 billion tokens during hyper-network pretraining, while Token Distillation sees only 3.2 million tokens (for initializing 2,600 new tokens), less than 1 1% of ZeTT's budget - still Token Distillation outperforms ZeTT. In return, ZeTT is faster at inference time (in our experiments on single H100 80GB GPU, ZeTT took less than minute). In future work, we believe that investigating an Token Distillation-styl objective for hyper-network pretraining would
Note that do anneal the learning rate zero, so this not an artifact of the learning rate schedule.
25

---

Published as a conference paper at ICLR 2026
08:52
Token Distilation
tokens
2589
2589 2589 2589 2589 2589 2637
2591
Table 13: Minimum training times (mm:ss) on the domain adaptation token initialization experiments for each model and initialization method measured over three runs. We report the minimum, run experiments on a shared cluster. We use single H100 80GB GPU. We also include the number of tokens for each model type.
Ang:
Orisisal ulcmisiion
64.5
69.8
706
60.2
72.3
64.4
$1.3
66.5
spech)
3406
Dotilatioo epecl)
NIP spech)
61.8
66.1
65.4
65.9
58.7
56.
61.5
58,9
63.
* Tolkn Dialibation (2: 2:rpoch)
6X2
68.5
669
67.
718
A8.4
61A
61.1
65.4
Table 14: Comparison of different initialization methods for domain adaptation. We report macro- average the tasks the Open Medical-LLM leaderboard (see (Section 4.1p. The best initialization result for each model is given in boldface. We only single seed for the epoch variants.
be beneficial, as experiments show that this significantly outperforms next-token prediction for learning new embeddings (ZeTT uses next-token prediction).
E USE OF LLMS
LLMs were used for ILM-as-a-Judee style evaluation, as discussed in the paper. LLMs were also used for formatting of Section 3 (gpt-ooo mommiminii- ni-high) intermittently while writing the code for experimentation (VSCode Copilot autocomplete), as well as for the creation of scripts that aggregate the final results into tables and figures. All LLM-generated code has been checked for correctness.
26


Reviews:
[{"review_id": "jKGkwnNEPp", "reviewer": "Reviewer_h7PH", "summary": "This paper proposes a novel embedding initialization method for new tokens, motivated by the fact that conventional methods only consider interactions within embeddings. The proposed method instead considers the output hidden representations of the new and the corresponding base tokens and tries to make them close to each other by using an MSE loss function, while only tuning the new token representations. The experiments show that the proposed method works as the best starting point for adaptation among existing baselines, including a sophisticated method like ZeTT (which uses a hypernetwork). The analysis covers a wide range of aspects related to the design and implementation of the proposed method, showing the robustness of the method.", "questions": "1. How does the proposed method ensure each training sample only contains a single new token? If there are two or more new tokens in a single sample, does Eq. (1) still work?\n\n2. On Weakness 1, I would suggest adding a few experiments that conduct continual pre-training with baselines (including the source model) and the proposed approach, given the same training budget.\n\n3. On Weakness 2, I would suggest incorporating an analysis of different vocabulary sizes.\n\n4. On Weakness 3, I would suggest adding an analysis of inference efficiency. This will further strengthen the contribution of the paper if the proposed method achieves better efficiency.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper is well-motivated. The paper clearly identifies the problem with conventional methods for embedding initialization of new tokens (i.e., not accounting for higher-layer model dynamics) and the necessity of training an auxiliary model for sophisticated methods like ZeTT. The proposed method clearly addresses these issues and employs a lightweight training-based approach that aims to make the representations of a new token and its corresponding source tokens similar using an MSE loss.\n\n2. The experiments well support the advantage of the proposed method against a series of baselines, showing it can provide a good starting point for adaptation.\n\n3. The analysis is quite extensive, covering different aspects of the proposed method, including training data, embedding initialization methods for new tokens, training objectives, and choice of a target layer.", "weaknesses": "1. The major limitation of this paper is a lack of full continual pre-training results. While the proposed method can provide a better starting point as “initialization”, it does not always guarantee a better performance after continual pre-training on target data. Any gains seen at the starting point might not hold up when we conduct further tuning on the target data. Given that almost all methods exhibit worse performance after embedding adaptation, they inevitably require continual pre-training before actual use. Otherwise, I do not see any point in using these methods.\n\n2. The choice of 2,500 new tokens (L202) warrants justification. This number should substantially affect the resulting performance, both downstream and inference efficiency. The current paper does not have an analysis of different sizes of new tokens. Also, the paper lacks a description of how it generates (defines) new tokens.\n\n3. The paper does not involve any analysis related to inference efficiency.  Given that the motivation of this line of work is to improve suboptimal tokenization, thereby achieving better downstream performance and better inference efficiency, the paper would need to demonstrate both.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Fu6aEBNvvo", "reviewer": "Reviewer_KJ3L", "summary": "This paper introduces \"Token Distillation,\" a novel method for initializing input embeddings for new tokens in a pretrained language model. The method addresses the inefficiency and potential performance issues caused by the \"over-tokenization\" of domain-specific terms into multiple subtokens. The authors propose a distillation objective: to optimize a new single token's embedding such that it replicates the model's internal hidden states when processing the original sequence of subtokens. The core contribution is this efficient initialization technique that leverages the full model's knowledge, not just the embedding layer.", "questions": "1. My primary concern, which currently limits my rating, is about the practical necessity of the efficiency-performance trade-off this paper proposes. If the authors can convincingly address this, I am open to raising my score.\nFor maximum performance, my default assumption is that one would always prefer the original tokenization, accepting its computational cost. The significance of your paper hinges on proving that the efficiency gain is not just a marginal benefit but a critical enabler for certain applications. Could you provide a concrete, quantitative analysis of these efficiency benefits? For instance(Do not to do all, just example):\n* Can you demonstrate how the reduced sequence length translates to significant cost savings (e.g., in dollars or GPU hours) for large-scale batch processing of a corpus?\n* Can you showcase a task involving long documents where the original tokenization would force truncation and lead to failure, while your method would succeed by fitting within the context window?\n* Are there real-time applications with strict latency budgets where your method is viable but the original is not?\nProviding evidence for a scenario where the original tokenization is practically infeasible or prohibitively expensive would directly address this concern and strongly bolster the paper's contribution.\n\n2. Table 3 shows a surprising case where Token Distillation outperforms the original tokenization (Llama-3-8B-Instruct on French). Could you elaborate on this? Is it a statistical anomaly, or does it suggest that for some out-of-domain languages, your distillation process might be correcting for certain suboptimalities in how the base model processes subtoken compositions?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Novel and Effective: The paper proposes a clever and elegant method, \"Token Distillation,\" which distills the model's internal behavior (hidden states) rather than simply aggregating embedding vectors. Experiments robustly show it outperforms strong baselines.\n2. Thorough Experimental Validation: The claims are supported by rigorous experiments across a diverse set of models, tasks (domain and language adaptation), and a comprehensive suite of baselines, demonstrating the method's reliability.", "weaknesses": "1. Insufficient Justification for Practical Significance: The paper's primary weakness is the lack of compelling evidence for why the proposed efficiency-performance trade-off is critical. In many high-stakes domains, even a small performance drop is unacceptable, and the paper fails to demonstrate scenarios where the efficiency gain from token compression is a mission-critical requirement rather than a minor convenience.\n2. Misaligned Motivation: There is a narrative disconnect. The motivation suggests over-tokenization harms performance, yet the experiments consistently show the original (over-tokenized) method as the performance ceiling. This weakens the argument and shifts the method's entire value proposition to an efficiency gain that is not sufficiently justified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "96FZhRf4HS", "reviewer": "Reviewer_3UEU", "summary": "This paper proposes a method to train new input embedding parameters for added tokens using hidden state distillation on the last layer. Specifically, the authors align the hidden state of added tokens t* with the one of original sub-tokens $\\tau (t^*)$ in related corpus and distill the hidden state with mean-squared error loss function. Experimental results on 8 decoder models show that the proposed method outperforms other vocabulary adaptation methods, and performs better with weighted next token prediction objective. Further analyses illustrate that better embedding initialization method can brings further improvement.", "questions": "1) How about the results of your method on encoder Transformer models like BERT?\n\n2) If all target tokens like Arabic or Chinese words are significantly different to the source tokens like English words and original subtokens $t_i$ are UTF-8 characters, how does your method perform? Can your method improved by initialization method based on semantic alignment like Focus and TokAlign in this setting?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- It is interesting to incorporate the Transformer layer into the initialization of new input embedding parameters. \n- The experimental results are promising for Transformer decoder models in the vocabulary adaptation task.", "weaknesses": "- Experiments are conducted on the Transformer decoder models. It is unclear for its performance on the vocabulary adaptation of Transformers encoder models.\n\n- Missing details that the number of tokens added in the experiments. The performance of this method may be affected by the lexical similarity between the target token and source token. For example, if all target tokens like Arabic or Chinese words are significantly different to the source tokens like English words and original subtokens $t_i$ are UTF-8 characters, how does your method perform? Can your method improved by initialization method based on semantic alignment like Focus[1] and TokAlign[2] in this setting?\n\n- Typo at the Line 359: \"NTP outperforms NTP\" --> \"NTP outperforms NTP (tune all embeddings)\"\n\n**References**\n\n[1] Konstantin Dobler and Gerard de Melo. FOCUS: Effective Embedding Initialization for Monolingual Specialization of Multilingual Models. EMNLP 2023.\n\n[2] Chong Li, Jiajun Zhang, and Chengqing Zong. TokAlign: Efficient Vocabulary Adaptation via Token Alignment. ACL 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)