Universal Cross-Tokenizer Distillation via Approximate Likelihood Matching
Authors: Benjamin Minixhofer
Ivan Vulić
Edoardo Ponti
Venue: NeurIPS 2025 poster ()
Keywords: distillation
tokenization
modularity
tokenizer transfer
Primary area: deep_learning
TL;DR: We introduce a new method for principled, effective distillation across tokenizers, enabling a number of new applications.

Abstract: Distillation has shown remarkable success in transferring knowledge from a Large Language Model (LLM) teacher to a student LLM. However, current distillation methods require similar tokenizers between the teacher and the student, restricting their applicability to only a small subset of teacher--student pairs. In this work, we develop a principled cross-tokenizer distillation method to solve this crucial deficiency. Our method is the first to enable effective distillation across fundamentally different tokenizers, while also substantially outperforming prior methods in all other cases. We verify the efficacy of our method on three distinct use cases. First, we show that viewing tokenizer transfer as self-distillation enables unprecedentedly effective transfer across tokenizers, including rapid transfer of subword models to the byte-level. Transferring different models to the same tokenizer also enables ensembling to boost performance. Secondly, we distil a large maths-specialised LLM into a small general-purpose model with a different tokenizer, achieving competitive maths problem-solving performance. Thirdly, we use our method to train state-of-the-art embedding prediction hypernetworks for training-free tokenizer transfer. Our results unlock an expanded range of teacher--student pairs for distillation, enabling new ways to adapt and enhance interaction between LLMs.

Universal Cross-Tokenizer Distillation via Approximate Likelihood Matching
Benjamin Minixhofer 0x43
Ivan Vulié 0x43
Edoardo M. Ponti 0x45,0x43
0x43 University of Cambridge Univeriiy of Edinburgh
Abstract
Distillation has shown remarkable success in transferring knowledge from a Large Language Model (LLM) teacher to a student LLM. However, current distillation methods require similar tokenizers between the teacher and the student, restricting their applicability to only a small subset of teacher-student pairs. In this work, we develop principled cross- toss-tooke distillation method solve this crucial defi- ciency. Our method is the first to enable effective distillation across fundamentally different tokenizers, while also substantially outperforming prior methods in all other cases. We verify the efficacy of our method on three distinct use cases. First, we show that viewing tokenizer transfer as self-distillation enables unprecedentedly effective transfer across tokenizers, including rapid transfer of subword models to the byte-level. Transferring different models to the same tokenizer also enables ensembling to boost performance. Secondly, we distil large maths-specialised LLM into a small general-purpose purpose model with a different tokenizer, achieving competitive maths Sublem-solling, performance. Thirdly, we use our method to train state-of-the-arr embedding prediction hypernetworks for training-free tok- enizer transfer. Our results unlock an expanded range of teacher-student pairs for distillation, enabling new ways to adapt and enhance interaction between L.Mss.
Introduction
Even the latest, most powerful Large Language Models (LLMs) still operate on tokens, and therefore require tokenizer manually designed component which turns text into sequence of tokens. Most currently available language models use subword tokenization (Sennrich et al.) [016]] Kudo & Richardson 2018), where each token is a word, or part of a word. Even though subword tokenizers are currently dominant, they are heterogeneously implemented: when a model is released, it often comes with its own tokenizer which has distinct properties such as the actual tokens it comprises (its vocabulary) and which method is used to segment text into these tokens (its tokenization function). Furthermore, there has been recent trend away from subwords toward models using vocabularies consisting of characters [Ta et al. 2022 Nawrot et al. 2023) or bytes (c.g. Xue et al. [0022] Yu ct al. 2023 Pagnoni et al. 2024). This exemplifies the wildly diverse landscape of tokenization.
On the other hand, distillation (Buciluá et al. 2006 Hinton et al., 2015) has emerged as powerful paradigm for creating effective language models by training them based on the signal from another model. However, existing distillation methods require similar tokenizers between the teacher and the student. In particular, most prior methods require the teacher and student represent their input in the same way, i.e., that their tokenizer is equivalent (e.g., Gu et al. 2024 Ko al., 2024). Some recent methods relax this requirement by developing heuristics to incorporate information from the teacher alongside a main objective (e.g., next-token prediction), making progress toward enabling distillation across similar (but not equivalent) tokenizers (Zhang et al.) 2024b. Wan al. 2024: Boizard et al.) 2025). Nonetheless, this requirement still limits their applicability to a rather small subset of the possible teacher-studen pairs. In this work, we develop cross-tokenizer distillation method to solve
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Teacher
mmm J hout Text mmm
Identity
3. Minimise
1. Tokanizz
comparable token chunks
Chunk Likellhoo
Difference
min
Studert
Figure 1:
We propose a cross-tokenizer distillation method which identifies comparable chunks of tokens, then minimizes the differences between their likelihoods (c.f. Section 3
this problem. Our principled objective enables distillation across highly dissimilar tokenizers for the first time (e.g.. distilling from subword tokenizer to byte-level tokenizer), while also unlocking pure distillation - training to purely match the teachers' behaviour, instead of matching the teachers' behaviour as an auxiliary objective, which can lead further empirical improvements.
In a nutshell, given some text, we first tokenize it with the teacher and the student tokenizers and compute the next-token likelihoods of all tokens under both models. We then find aligned chunks of tokens (the chunks encoding the same portions of tex() between the two sequences. Now, to compare the chunk likelihoods of the aligned chunks (e.g., via their KL-divergence), we would need to know the likelihoods of all possible outcomes. This is not an issue in the same-tokenizer distillation case since we can compute the likelihoods of all tokens in the vocabulary in single forward pass. However, in our case, we would have to compute the likelihoods of all multi-token chunks, which is impossible since their number is infinite. We thus approximate the chunk likelihood difference via binarised f f-divergence (Section 3 Together with optional debiasing (Section 3.1 and distillation of the hidden states (Section 3.2 this leads an effective and universal cross-tokenizer distillation method, which we refer to as Approximate Likelihood Matching (ALM), sketched in Figure
We test ALM three use cases. First, we propose a novel viewpoint of tokenizer transfer as cross-tokenizer self-distillation (Use Case 1). This enables transferring subword-based LLMs to a different subword tokenizer while largely retaining performance, and achieves rapid and effective Subword - Byte transfer for the first time. This allows us to create a competitive byte-level model at fraction of the cost of training it from scratch. We further show that transferring different models to the same subword tokenizer enables ensembling their predictions (e.g., via averaging their token-level probabilities) which substantially boosts performance over the individual parts. Secondly, we show that ALM outperforms prior methods at distillation of a large math-specine spechallsedd model into much smaller general-pe general-purpose model with a different tokenizer (Use Case 2}. Thirdly, we plug- in ALM as objective to train embedding predictions hypernetworks as introduced by Minixhofer et al. (2024). achieving a new state-of-the-ae in tokenizer transfer without ever training on the target tokenizer (i.e., Zero-Shot Tokenizer Transfer, Use Case 3). In summary, our work greatly expands the number of possible teacher-student pairs for distillation, improving reusability, composability and transferability of language models. Our code and models are available at atggithub. com/ wtt./biiiniiiereeeeeeeee wtwhofff//ttokkkkoeeeee
2 Preliminaries and Background
2.1 Tokenizers
Tokenizers map the input text to a sequence of tokens. Formally, they operate on finite sequences Σ* over the alphabet Σ. For simplicity and since it is not vital our method, we presume Σ to be the set of bytes {0. 255} and Σ* string represented as bytes via the UTF-8 standard Yergeau 22003). This does not lose generality, since although some tokenizers operate on units built on top of UTF-8 bytes (e.g., Unicode characters: Kudo & Richardson 2018), they can be transformed to operate on bytes instead (Minixhofer et al. 2024). Following prior work (Uzan et al. 2024 Feher al. 2024), we denote the tokenizer as the tuple (V. T) comprising (i) the vocabulary V which determines set of all possible tokens and (ii) the tokenization function T Σ - V which determines the sequence of tokens any given text is mapped to.
Definition.

---

Tokenization functions fuse bytes into larger (e.g., subword) tokens without loss of information Consider the tokens {t,, 12....,,t,,
T(x). Then ti @ t2 O ... @ ta æ (where C is the concatenation operator). This has multiple useful implications. All elements in the vocabulary V are sequences of bytes E. Fl Furthermore, T is injective so there exists left inverse D such that (a)) Ya. We call D the detokenization function. Importantly, D is only a left inverse, i.e., there may exist a token sequence such that t + (()()) since is not necessarily bijective.
Tokenization Bias. Subword tokenizers suffer from what has been termed tokenization bias (Phan et al. 2024p: sequences of tokens can implicitly leak information about the future content of the text they tokenize. For example, assuming a vocabulary of English words, the token sequence {_Hello,_Woo leaks that there is no 'ld' after '_Worl', otherwise the text would have been tokenized as [_eello_Worrdd instead (see also Vieira et al. 2024).
2.2 Generative Language Models
Generative language models are auto-regressive next-token predictors. Given a sequence of tokens, a generative language model defines a probability distribution pe (f:]t.. F over the next token. To compute the probability of a multi-token sequence conditioned on t,, we can compute pe (t., [t.,) = pe((:]t.e).... .... pa ( 11) To auto-regressin rugo-eegressvvely sample from the model, we continuously sample Ps (u(t.:) and append the sampled u to the sequence. However, the token level is not an intuitive interface to language models. To obtain a text level interface, we need to wrap the language model with tokenization and detokenization. We compute the probability of text y conditioned on the text x via pe (T [T)))))) We can auto-regressively generate a text conditioned on x by sampling u4 (u, (T u.)) for i E {O...., n), then detokenizing to obtain y == Duo.. Un
However, for tokenization and detokenization not to introduce any bias, T(x) @ T(y) = T(x &
must hold. This is not always the case due to tokenization bias (see above).
2.3 Distillation
Pure Distillation vs. Hybrid Distillation. The general idea behind distillation is to transfer knowledge from a teacher model o to student model 6, where the student model has some desirable properties (c.g.. being more efficient; Buciluă al. 2006; Hinton al. 2015 We denote the teacher likelihoods pr == Ps and the student likelihoods ps = Po- A key distinction can be drawn between pure distillation, where the primary objective is for the student predictions ( (z) to match the teachers' predictions Pr (y)x) and hybrid distillation, where the main task is to model some ground-truth p(y| (z) and a distillation objective used in addition to the main objective (e.g.. next-token prediction) to increase performance on the main task. Hybrid distillation can be useful to avoid potential issues arising from the pure setup, e.g. teacher hacking (Tiapkin et al.] 2025) and a detrimental -teche capacity gap (Busbridge et al., 2025). On the other hand, pure distillation can be highly effective at preserving the student's 's knowledge when it has the same backbone as the teacher (self-distillation)): setting where hybrid distillation can be destructive (c.f. Section 4
Distillation Objectives. To encourage the student predictions to match the teachers', we need some measure of distance between ( xz) and pr (y(z). The typical choice is the Kullback-Leibler diver-
gence (PT || ps) = Et.ev PT pr (t.]t..) log ps(t, E.I) (Kullback & Leibler
(r.(t:t..)
1951). However, other
objectives have been argued be more suitable for distillation (Gu at. 2024
[o et al.
2024); many
of them are instances of f-divergences of the form Dr(rr || ps) L,v
[pr (4
t. Ips (((())
where
= ps (x) (r(()) and is convex and non non-negative (Rényi
1961 5
'This is not the case for tokenizers which can emit <unk> due out-of-vocabule characters (such as the original BERT tokenizer; [evlin et al. 2019). However, tokenizer with this property again be converted to byte-level and the vocabulary minimally extended to cover bytes to restore the byte fusion property.
FFor ease of notation, we do not treat special tokens (such as an <eos> token marking the end of the text) separately. Special tokens can be considered part of the encoding Σ, e.g.. range from {256... ... 256 + k). .... 'We use Python-style indexing where t., indicates the elements in t starting from i up to but excluding j. and tú indicates the elements from the beginning (index 0) up to but excluding j.
"We refer Xu al. (2024) for an overview on distillation in the context of LLMs. 'Note that while the divergence induced by is typically defined Dy(pllg) E. ((x)f(t ((t)
detine it as Dy (pllg) Σ. (()()))l((z)) to retain clarity in absence of the f-divergence viewpoint.
3

---

Distillation. Recent work has begun investigating ways to distill across different tokenizers. proposing ULD (Boizard et al. 2025), MinED (Wan al. 2024) and DSKD Zhang et al. 2024b). We provide detailed overview of these prior cross-tokenizer distillation methods in Appendix G They have in common that they heuristically incorporate some information from the teacher. These heuristics are imperfect; prior methods thus strictly operate in the hybrid distillation setup, lacking the properties necessary to form a pure distillation objective (e.g., the loss being minimised iff ps(y) pr(y(æ Væ, y). Our key contribution is introducing principled cross- tokenizer distillation objective which enables distillation across fundamentally different tokenizers (e.g., Subword + Byte) for the first time, while also unlocking pure-cross tokenizer distillation.
3
Methodology
Our key goal is for the student likelihood ps (s((s())Tss()) to be equal the teacher likelihood pr (TT(T=))) (z)((())))) for every comparable prefix y Σ" and continuation E D" To this end, we start by computing the next-token probabilities yr (777)))) and ps (<)) (z)))) This can be done in single forward pass over both language models. We can now enumerate the starting and ending positions of the aligned token chunks between the teacher and the student sequences
= y. = D(Ts( 2, ki
Ax(x) e
((Tr()))
(Alignment Indices)
holds
Here, c(i. j. ki, 1) can optionally constrain the alignment to chunks which are amenable to comparison (more on this later in Section 3.I.. For now, we define the chunk-level probabilities p(ar, i:j) as
p i:j) = (
(Chunk-Level Probability)
This lets us define our final objective minimising a divergence between the chunk-level probabilities:
CALM (x)
((r((, ps(r + f(1 pr(x || - (psssrr (m)
Σ
(ALM Objective)
where 7 a temperature scaling hyperparameter and f a function inducing an f-divergence (Rényi 1961). Typically, with identical tokenizers, the sum of the f f-divergence would range over all tokens the vocabulary; however, when comparing chunk-level probabilities there an infinite amount of possible outcomes since there is an infinite amount of possible byte sequences, whose probability cannot be estimated in finite time. We thus resort to computing the f-divergence over the binarised possibilities {p(æ, j), 1 p(æ, i }]} given sample æ D. This constitutes upper bound to the true f-divergence and preserves its crucial properties (c.g., being minimal iff ps = pri see Appendix B for further analysis).
3.1 Outcome Chunk Debiasing
Our formulation for the Chunk-Level Probability is only approximate since the conditions T(x):; i and the outcomes T(x)); y. could exhibit different tokenization biases across the teacher and the student. We can avoid the bias stemming from the outcome chunk by modifying the chunk probability to
debiased (x, (e.i:)) ((T()) Σ ((p(eT(())a)) e E V, starts with any b E B} (Outcome-Debiased Chunk- Probability)
Chunks of tokens are strictly only comparable if the difference hetween their tokenization biases is zero since ps (Ts(((2) (z) = (T(TT())(T<(() Vy, E' does not imply that the two language models are equivalent: if (z) exhibits different tokenization bias from Fr(z), the two models implicitly condition on different evidence. We formalize this notion in Appendix [F] where we also experiment with explicitly choosing chunks which have sufficiently low differences in tokenization bias.
We compute greedy alignments, ie., we do not consider partially overlapping alignments.

---

70%
.Wolld
with Outcome
No Debiasing
Chunk Debiasing
Hello
_Wor
f(1.00e-2 ||
6.93e-1
f(1 20e-3 3 || 1.39e-3)
H
W
0,15
Figure 2: Outcome chunk debiasing removes tokenization bias. For example, the low probability of the subword token _Wor would be matched to the high-pro high-probability byte sequence {_. o, in naive subword byte transfer. We can debias by multiplying by the marginal probability of pretoken-bounet boundary byte occurring after the chunk. In this example, {nn, ! S B and s & B, 1 $ B where B is the of shared peeooen-bboundayy bytes across the teacher and the student.
Here, B is the set of -buudaenybuuddrr bytes; these are bytes which never occur within a token. In practice, this can be e.g. whitespace such as the bytes corresponding to the \n, \t and the space character. Going back to our example from Section 2.1 let = { [_elll,__wol p(t) . (tt) + (p((Na)(t) + .... we explicitly exclude the possibility that t is followed by the letters 'ld', which, assuming every chosen ending token starts with a byte in B, removes tokenization bias (otherwise, the sequence would still be biased since there could exist single token encoding Dtt @ {e])). This debiasing has been first introduced by Pimentel Meister (2024) in the context of correctly computing the conditional probability of a word under an autoregressive language model. See for outcome chunks since the required probabilities are computable 'for free" within a single forward pass. There is, however, one remaining complication: if the debiasing probability multiplication with p (()())) (T()).)) will 'erase" the chunk probability to extremely low value, making meaningful comparison between chunk probabilities difficult. To address this issue, we set the alignment constraint (c.f. Equation Alignment Indices):
By computing
illustration of this example. Debiasing only feasibly applicable the
very low, the
ct.j k, = ((pr()(r(()),)) |e Vr, starts with any b E B)2
We ablate the impact of outcome chunk debiasing in Appendix A.3 finding that it substantially improves performance, with a slight further improvement by choosing nonzero threshold ?,
3.2 Distilling Hidden States across Tokenizers
limitation of our method is that, since we match 32-bit floating point chunk probabilities. the teacher signal has |Ael x 32 bit per text, while it typically has O(VV) more at [T(x) x [V] x 32 bit when the teacher and the student have the same tokenizer (i.c., KL divergence between the teacher and the student next-token distributions). To make up for this sparsity of the signal, we optionally add an auxiliary loss maximizing the similarity of aligned hidden states between the two models
SST
x
Σ
Σ A.j.A.EEA.))
[(r((T(())) proj(Hs(Ts((*))
tr.seLm
(Hidden State Alignment Objective)
where H, H.....) ) are the language models' hidden states at layer and index i in the sequence and LT,S E 2² are layer-wise alignments between the teacher and the student (e.g., indicating that the last
TThis is inspired by same-tor tokenizer distillation methods which align hidden states in Sanh al. 2020
5

---

Table I: Results of transferring the Gemma2 2B IT and Llama3.2 2 3B IT LLMS to the Qwen2 tokenizer and to byte-level tokenization. original denotes the original model without transfer. ARC-C refers to Arc- Are-Challenge. AGI-EN and AGI-ZH refer to the English and Chinese splits of AGIEval.
Benchmark
Model
Tokenizer Method
Avg.
PIQA
ARC-C
BoolQ
MMLU
AGIEN
AGI-ZH
IFEval
original
79.6
50.4
83.8
56.9
42.1
30.7
62.5
58.0
SFT
75.8
43.5
77.7
49.8
31.4
28.7
54.2
51.6
DSKD
74.0
41.4
79.7
50.4
33.0
28.9
53.6
51.6
+ Qwen2 MinED
76.7
44.3
79.6
51.8
33.0
28.8
57.1
53.0
ALM SFT
77.0
48.0
82.5
53.4
36.4
31.4
55.7
34.9
ALM
76.8
49.0
82.7
53.6
38,9
31.6
53.2
55.1
Gemma2 2B IT
SFT
70.7
34.7
67.9
43.1
27.6
30.0
51.5
46.5
DSKD
70.5
35.6
70.2
42.3
27.5
30.4
55.0
47.4
Byte
MinED
69.4
35.8
72.8
42.9
28.7
29.9
50.2
47.1
ALM SFT
71.5
38.2
80.5
51.0
35.6
30.4
51.9
51.3
ALM
72.0
40.6
81.1
51.0
36.0
29.3
44.3
50.6
original
76.9
43.9
78.8
62.4
36.6
40.2
76.6
59.3
SFT
76.4
44.0
80.0
60.7
33.9
29.6
65.6
55.7
DSKD
72.0
37.2
78.3
45,9
32.5
30.9
60.7
51.1
Qwen2
MinED
77.1
44.2
82.4
60.9
35.8
29.4
71.0
57.3
ALM SFT
77.0
44.4
79.9
61.8
37.1
32.0
74.1
58.0
ALM
77.3
45.6
79.0
61.6
37.1
33.3
76.3
58.6
Llama3.2 3B
SFT
75.2
39.8
76.8
51.5
31.5
32.6
60.8
52.6
DSKD
71.1
36.0
65.8
48.0
32.0
30.3
57.9
48.7
+ Byte
MinED
73.2
38.7
78.6
51.1
33.1
59.6
52.4
ALM SFT
73.6
39.8
76.6
57.0
35.7
33.3
58.8
53.5
ALM
73.7
40.1
76.0
55.9
35.7
33.2
49.2
52.0
layer hidden states of the two models should be aligned). proj(*): RRds + IRdr is a learnt projection function mapping the student hidden states the teachers' dimensionality. We use greedy aligmments (without any constraint c) as the hidden state alignments A(x). The auxiliary loss enriches the signal from the teacher by |A] dr x bit. We find it to be particularly useful in some cases when the teacher and the student have the same underlying architecture (c.f. Section 4
3.3 Combining Loss Components
CALM (æ) and Chiskle We can also As per above, we have two distillation loss components S
combine the distillation loss with a next-token prediction objective (making it hybrid distillation; c.f. Section 2.3). The question, then, how combine these losses into one? Prior work ZZang jt al. 2024b; Wan et al. 2024 among others) aggregates losses via their arithmetic sum, where the contribution of each loss is optionally modulated by factors B1, f2, -2,,-.... 9n. Although this allows choosing /3 such that the losses have approximately equal magnitudes, this does not mean that they contribute equally: they only contribute equally if their gradients have equal magnitudes. These two quantities are not necessarily related: for example, forward KL-divergence and cross-entropy have different values but equivalent gradients w.r.t. the predicted probabilities. To solve this problem, GradNorm (Chen et al. 2018) introduces an auxiliary loss to learn task weights such that the per-task gradients Gyyy of the last layer weights W have similar magnitudes across tasks (enumerated by i). We take simpler approach: We compute task gradients for the last layer, then set the weight for the task i l/ |l (normalised to sum to one). Instead of optimizing a loss toward similar last-layer gradient magnitudes, we thus directly solve for equal last-layer gradient magnitudes, choosing only the layer gradients to maintain minimal overhead. This method (which we refer as GradMag) performs par or better than GradNorm while being casier to implement (see Appendix A.4).

---

4
Experiments
We experimentally verify our method on three distinct use cases. First, we observe that self-distillation across tokenizers enables unprecedentedly effective transfer of pre-trained language model to different tokenizer, which we explore in Use Case I Secondly, we find that ALM improves over prior cross-tokenizer distillation objectives at distilling a large maths- spenissspecaliseed model into a much smaller model (Use Case 2). Thirdly, we plug in ALM as replacement for the next-token prediction objective Minixhofer et al. (2024)'s 's hypernetwork training procedure, achieving a new state-of-the -te--artt in Zero-Shot Tokenizer Transfer across a range of tasks (Use Case 3).
Use Case 1: Tokenizer Transfer via Seff-Distillation
Tokenizer transfer is the problem of transferring pretrained language model to different tokenizer which typically has some desirable properties (c.f. Rust et al. 2021). We observe that tokenizer transfer can be treated as cross-tokenizer self-distillation: the teacher is the language model with the original tokenizer, and the student is the same language model with the new tokenizer. We investigate two particular applications of tokenizer transfer via self-distillation. First, we transfer multiple language models to the same (subword) tokenizer; this enables token-level ensembling, improving their performance. Second, we transfer subword-basee language models to byte-level tokenization. fundamentally changing their token granularity. This may be an attractive way of creating byte-level models instead of expensively training from scratch.
Models. We use the instruction-tunn Gemma2 2B IT (Gemma Team et al. 2024) and the instruction- tuned Llama 3.2 3B IT (Grattafiori et al. 2024) the models for self- distillation. We transfer these models to the Qwen2 tokenizer; this enables ensembling the models with models of the Qwen series, for which we choose Qwen2.5 1.5B IT (Qwen et al. 2025). We examine the choice of the pivot and the related subtle decision whether to keep or to transfer the special tokens in Appendix H
Baseline Methods. We compare against initialising the new token embeddings using heuristic (e.g. Tran 2020, Minixhoter et al.] 2022, Gee et al. [022] Dobler & de Melo 2023). then training on next-token prediction the current standard methodology for tokenizer transfer. We refer this baseline as SFT. We use FVT (Gee et al. 2022) as the initialisation heuristic following Minixhofer
et al. 22024). We choose DSKD Zhang et al. (2024b) and MinED (Wan al.) 2024) as cross-tokenizer
distillation baselines P DSKD and MinED add cross-tokemizer distillation to the next-token prediction objective, so they need train jointly on SFT and distillation; this is not the case for ALM. We thus also experiment with our objective plus next-token prediction (ALM + SFT).
Training, We train on the Tulu3 instruction-tune totiionntttinnnnnon dataset (Lambert et al. ctt al. 2022), sweeping over the learning rate per method (see Appendix I and using GradMag to combine loss components (c.f. Section see Appendix for training details. For transfer to bytes, we make additional adjustments to account for this fundamental change discussed in Appendix C
2025 with LoRA (Hu
for
Evaluation & Transfer Results. We evaluate on a standard set of natural language benchmarks consisting of PiQA (Bisk et al.) (020). ARC-Challenge (Clark et al. (018). BoolQ (Clark al. 2019), MMLU (Hendrycks et al. 2021). AGIEval (Zhong et al. 2023 1U and IFEval (Zhou et al. 2023). We use Im-eval (Gao et al. 2024) for all evaluations. Table shows tokenizer transfer results. On transfer to the subword-level Qwen2 tokenizer, MinED leads to consistent improvements over SFT, and ALM and ALM+SFT consistently improve over MinED. Here, pure ALM (without SFT) performs better than ALM+SFT, which may be the case due to better preserving the original model's ' behaviour. The byte-transfer case is substantially harder: in general, we expect the difficulty of cross-tokenizer distillation to correlate with the difference in vocabulary size between the teacher and the student. The vocabulary size difference is extreme for transfer to bytes. In this case, prior cross-tokenizer distillation methods do not achieve consistent improvements over SFT. In constrast, ALM and ALM+SFT achieve substantial and consistent improvements. We hypothesize ALM+SFT outperforms ALM here since naïve transfer bytes is necessarily strongly destructive since the model has to adapt to a vastly different granularity, so preserving the original model's behaviour
WW forgo comparison against ULD BBoizard al. 2025) since DSKD and MinED have been shown perform consistently better by Wan et al. 2024) and Zhang et al. (2024b).
For AGIEval, we report the macro- average over the English and Chinsse subsets excluding GaoKao Math Cloze and AGIEval Math since they are not designed for instruction-turs tuned models in their published form.
7

---

Table 2: Results of ensembling (transferred) Gemma2 and Llama3 with a Qwen2.5 model. p and logp denote taking the arithmetic mean of the probabilities and the log probabilities, respectively. MinED denotes the ensembling baseline based on heuristically aligning token probabilities.
Benchmark
Size Model
Avg.
PIQA
ARC-C
BoolQ
MMLU
AGL-EN
AGI-ZH
IFEval
2.6B G
Gemma2 2B IT
79.6
50.4
83.8
56.9
42.1
30.7
62.5
58.0
Original
3.2B C
Llama3.2 3B IT
76.9
43.9
78.8
62.4
36.6
40.2
76.6
59.3
Models
1.5 C Qwen2.5 1.5B IT
76.3
43.2
77.9
60.1
45.0
54.2
46.3
57.6
2.4B g' g Q Tok.
76.8
49.0
82.7
53.6
38.9
31.6
53.2
55.1
Transfers
3.3B L C Q Tok.
77.3
45.6
79.0
61.6
37.1
33.3
76.3
58.6
7.3B
Q+G++C (MinED-logp)
75.7
46.8
84.6
62.6
42.0
34.5
67.2
59.1
7.3B
@+ (MinED-P)
76.9
46.2
83.2
64.1
40.6
39.4
65.9
59.5
Ensembles
7.2B
2+-'' L' (logp)
77.7
48.4
83.9
62.2
44.2
46.8
61.1
60.6
7.2B
Q+9' + (p)
77.5
48.0
83.4
63.0
43.6
45.7
64.3
60.8
Larger Models
7.6B
Llama3.1 8B IT
81.6
53.9
85.4
68.4
47.3
47.1
81.7
66.5
8.0B
Qwen2.5 7B IT
80.4
54.3
86.4
71.7
57.6
72.6
76.6
71.4
is less crucial. For transfer to bytes, gap with respect to the original model remains in all cases. There are multiple avenues toward closing this gap: may be beneficial add distillation phase on pretraining data, to add more layers to close the gap in effective parameter count, to retrofit the model into an hourglass architecture with token merging as in DTP (Nawrot et al. 2023) or BLT (Pagnoni et al. 2024), and/or to add multi-byte prediction (Gloeckle et al. 2024).
Ensembling Results. Table 2 shows the results of ensembling Qwen with the transferred Gemma and Llama models (Q + g' + L'). Following Huang et al. (2024). we compare against a MinED ensemble: this applies the same heuristic as the corresponding distillation method to map tokens in the respective vocabularies to the pivot vocabulary (in this case, Qwen) [1 The ensemble exhibits two notable characteristics: (i) on tasks where constituents perform at a similar level, the ensemble improves over the performance of each individual model (e.g.. PiQA, BoolQ, MMLU) and (ii) on tasks were either one of the constituents performs exceptionally well, the ensemble moves the needle toward this level of performance, although it doesn't 't reach it (c.g.. ARC-C, AGIEval, IFEval). These improvements stem from naively averaging the models' predictions. We believe they open the door toward more sophisticated inter-model combination via improved ensembling (e.g.. Ormazabal et al. 2023), routing (e.g.,SShen et al. 2024) and/or merging (e.g., Sharma et al., 2024).
Cost of Distillation. We report the FLOPS and memory requirements as well as the task performance across methods in Figure DSKD requires substantially more TFLOPs than the other methods since there is an extra cross-attention step between the teacher and the student sequences which can incur substantial costs. This is further exacerbated with longer contexts (e.g., when transferring to bytes). MinED needs substantially more memory since the [T(x) x [V| matrix of logits needs to be aligned with the student along the sequence and vocabulary dimensions. ALM suffers from neither of these constraints and reduces the gap to the teacher by an additional 34% over the best prior method.
Use Case 2: Large-to-Small Distillation Across Tokenizers
To investigate the efficacy of our method in a large-to- tangelto-small distillation setting, we scale up from the evaluation setups used in prior work on cross-tokenizer distillation to realistic larger-scale setup: distilling the maths-specialised OpenMath2-LLlamall 1-8B (Toshniwal et al.) 2024) into Gemma2 2B.
As in Use Case
1
we compare our method against training with SFT, DSKD, and MinED.
Training. We use the Open dataset (which the teacher has been trained on: Toshniwal et al.) 2024) and follow the training setup from Use Case I apart from increasing the sequence length 1024 allow for longer chain-of-lee chain-of-thought traces, doubling the batch size 64, and accordingly reducing the training steps to 5k to train for an equivalent total amount of tokens. We also found it beneficial to train the full model and reduce the learning rate 5e-6 in preliminary experiments.
"It is worth noting additional methods to ensemble models with different tokenizers: DeePEn (Huang et al. 2024) and (ii) Pack of LLMs (Mavromatis et al. 2024). The required memory of (i) scales quadratically with the vocabulary size, making it infeasible to apply to models with >100k tokens and (ii) is based on combining the sequence-level likelihoods, which makes generation impossible: thus compare against MinED.
8

---

Roquired FLOPs
Required Memory
Task Performance
2x SPT
SFT
58.0 (Teachet)
-70%
20%
-44%
IXSPT
(SPT)
DSKD
MiaED
ALM ALM-SFT
MaED
ALM ALM-SFT
DSKD
MisED
ALM ALMISFT
Figure 3: Efficiency and task performance metrics of cross-tokenizer distillation methods, measured via worst-case performance across transfer of Gemma2 to Qwen2 and byte-level tokenizers. SFT denotes the required FLOPs and memory well the task performance of the SFT baseline.
Table 3:
Results of cross-tokenize distilling the large math-specialized Open 1-8B into the small Gemma2 2B language model. All results are zero-shot CoT.
Model
Method
GSMSK
MATH
Avg.
OpenMath2- Gemma2 2B IT
88.9
20.86
60.2 10.69
74.6 10.55
6.1 20.66
11.3 10,45
8.7
10.40
SFT
67.2 +1229
36.2
4-068
51.7 4:0.3
DSKD
65.7 +1.31
34.9
4-0.77
50.3 40074
Gemma2 2B
MinED
64.9 +1.31
34.6
+0007
49.8 4074
ALM + SFT
70.2 1126
36.4
10.68
20.72
ALM
68.5 1.28
36.7
1:0.88
52.6
40072
Evaluation & Results. We report zero-shot accuracy GSM8K (Cobbe et al. 2021 and the MATH benchmark (Hendrycks et al. 2021) in Table Notably, the SFT baseline is competitive, surpassing both MinED and DSKD. However, ALM and ALM SFT do consistently outperform SFT, leading to an improvement of up to 3% points while greatly surpassing the general-purpost Gemma2 2B IT and achieving 53.3 / 74.6 of the teachers' average performance 4x fewer parameters.
Use Case 3: Improving Zero-Shot Tokenizer Transfer Hypernetworks via Seff-Disiillation
As final use case, we apply ALM to Zero-Shot Tokenizer Transfer (ZcTT) hypernetworks (Minix hofer al. 2024). In a nutshell, the idea is training hypernetwork on a distribution of randomly sampled tokenizers such that, after training, the hypernetwork can predict embeddings for any arbitrary tokenizer. The tokenizer transfer effort thus shifted to hypernetwork training stage, making subsequent tokenizer transfer easier (or even possible in zero-shot). |Mnixhofer et al. (2024) train the hypemettwrks via a next-token prediction objective, i.e., to predict embeddings which minimize the cross-entropy of the next-token prediction. We instead plug in the ALM loss, training hypernetwork to produce embeddings which mimick the behaviour the original model, instead of merely predicting embeddings which perform well at next-token prediction! 12 Results are shown in T ALM outperforms SFT for hypernetwork training (e-g. 67.5 - 74.0 on BoolQ for zero-shot transfer the Mistral v2 tokenizer). In general, expect direct tokenizer transfer self-distillation via ALM as in Use Case to be the most efficient the goal is transferring to a single target tokenizer. However, if the goal is to potentially transfer up to N different tokenizers, there is some threshold of where we expect would become more efficient to train a ZeTT hypernetwork once, then have brief additional training phase per tokenizer, instead of separately transferring to the N tokenizers individually. ZeTT hypernetwork training also demonstrates another advantage of our method: while MinED needs expensive pre-computation step compute the Levenshtein distance between every pair of tokens, ALM does not require any pre-computation: it can operate in online settings such as hypernetwork training where tokenizer is sampled every training step.
We also streamline the hypernetwork architecture to train in single stage for both methods (Appendix 14 "Here, we substitute IFEval for ARC-E (Clark et al (0188) and HellaSwag (Zellers et al. 2019 to report performance superset of the benchmarks used by Minixhofer (204))

---

Table 4: Performance of a Gemma2 2B hypernetwork trained with SFT vs. ALM on zero-shot
transfer to the GPT2 (Radford al. 2019). Mistral v2 (Mistral Al 2025) and Llama3 tokenizers
Benchmark BoolQ
Tokenizer
Method
Avg.
PQQ
ARC-E.
AGLEN
AGL-ZH
HS
original
79.6
78.2
50.4
83.8
56.9
42.1
30.7
72.6
61.8
H-lhet
GPT2
SFT
74.6
63.1
40.1
73.2
40.8
26.7
27.8
68.9
51.9
ALM
76.1
67.2
41.4
71.6
45.0
26.1
27.2
69.7
53.0
SFT
73.6
61.6
37.7
67.5
37.7
25.2
29.3
64.5
49.6
Mistral v2
ALM
75.2
66.3
40.5
74.0
38.5
26.3
28.6
66.0
51.9
SFT
74.2
60.9
38.4
68.0
37.6
24.9
29.2
64.3
49.7
Llama3
ALM
75.5
66.0
40.3
74.3
38.6
25.6
28.8
65.9
51.9
Table 5: Results of transferring Gemma3 12B IT to the Qwen2 tokenizer (c.f. Table
Benchmark
Model
Method
Avg.
PIQA
ARC-C
BoolQ
MMLU
AGLEN
AGI-ZH
IFEval
original
78.2
60.1
87.5
71.5
60.0
57.4
83.3
71.2
Gemma3 12B IT Qwen2 Tok. SFT
81.4
56.7
83.7
67.2
46.2
43.5
73.4
64.6
ALM
78.8
59.9
86.9
70.7
57.6
53.6
76.7
69.2
4.1 Scaling ALM to Larger Models
We ran additional experiments on tokenizer transfer of Gemma3 12B (Gemma Team et al. 2025 to the Qwen2 tokenizer (also used by Qwen3) via ALM self-distillation verify that our method scales to larger models. Results are shown in Table Crucially, the transfer-original gap reduction due to ALM scales favourably with model size both in terms of absolute numbers and relative improvements over SFT (3.5/6.4 /6.4 & 55% reduction for Gemma2 2B Qwen2 Tok. vs. 4.6/6.6 ss 70% reduction for Gemma3 12B > Qwen2 Tok.) in this experiment. We leave scaling truly large models to future work, but note that we expect no fundamental obstacles in doing so (c.f. also Appendix on an efficient implementation of ALM).
5 Conclusion
We have introduced ALM, a principled cross-tokenizer distillation method which outperforms prior heuristic methods across diverse set of use cases while also enabling distillation across fundamentally different tokenizers for the first time, as evidenced by effective distillation subword-bbat based models to the byte-level (Use Case I We have also shown that ALM is effective at conventional large-to-sman distillation (Use Case 2) and enables state-of-the-
zero-shot tokenizer transfer (Use Case 3). Overall, our work vastly expands the number of possible teacher-student pairs for distillation. In addition to enabling range of new applications, this provides basis for further enhancements such as more sophisticated combinations of tokenizer-transferre models via enhanced gnsembling/rrutin//errr. and using ALM to retrofit subword-basee models into dedicated byte-level architectures.
Limitations
Due to computational constraints, we have trained on comparably few tokens (+= 0.6B). It not yet clear how ALM compares to SFT and other distillation methods in regimes with larger amounts of training tokens. Furthermore, we only explore one instantiation of cross-tokenizer transfer via minimising a chunk-level f-divergence, leaving other instantiations (e.g., categorical instead of binary divergence) subsequent explorations. Along the same lines, is not clear whether our chunk selection strategy is optimal and the space of possible chunk selection strategies is large, including overlapping or non-consecutive chunks. Finally, we have restricted ourselves to the seminal Supervised KD setup (Hinton et al.) 2015), leaving applications of ALM to more intricate distillation using e.g. Rejection Sampling (as in Touvron et al. 2023) or On-Policy Distillation (as in Gu et 2024
Agarwal et al. 2024) to future work.
10

---

Acknowledgments and Disclosure of Funding
This work supported by the ERC Starting Grant AToM-FM (101222956) awarded to Edoardo M. Ponti and the Royal Society University Research Fellowship 'Inclusive and Sustainable Language Technology for a Truly Multilingual World' (no 221137; 2022-) awarded to Ivan Vulić. Research supported by the Google Cloud Research Credits program with the award GCP329647813 Research supported with Cloud TPUs from Google's TPU Research Cloud (TRC), We would like to thank Andreas Grivas, Giwon Hong and Hannah Sterz for their helpful feedback on the paper draft.
References
Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk. Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self generrede mistakes. In The Twelfih International Conference on Learning Representations, 2024. URL httLlhttts////oo httpps://pperrevieee net/
Forum?id=3zktaqtue
Federico Barbero, Álvaro Arroyo, Xiangming Gu, Christos Perivolaropoulos. Michael Bronstein, Petar Veličković, and Razvan Pascanu. Why do Ilms attend to the first token?, 2025. URL https://arxiv
brg/aba/2504.07//
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: Reasoning about Physical Commonsense Natural Language. In Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020.
Nicolas Boizard, Kevin El Haddad, Celine Hudelot, and Pierre Colombo. Towards cross-tokenizer distillation: the universal logit distillation loss for LLMs. Transactions on Machine Learning Research, 2025. ISSN 2835-8856. URL httphttt//pppeeeeaiaii
net//forummdd-bv<ztiiio
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, KDD '06, pp. 535-541, New York, NY, USA, August 2006. Association for Computing Machinery. ISBN 978- 1-59993-39999333 doi:
10.112 1554644. URL wwtphhttt///lttt acm. www.a.a.....................aaaa
org/ doi /111//1//////111 1150464
Dan Bushridge, Amitis Shidani, Floris Weers, Jason Ramapuram, Etai I Littwin and Russ Webh. Distillation Scaling Laws, February 2025. URL httpttt::///.ii..r///eeeeee wwtp:///ww......././././..... awwpaaaa.//aae//e/ 08606 arXiv:2502.0666 [es].
Zhao Chen, Vijay Badrinarayanan, Chen- Yu Lec, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing deep multitask networks. In International conference on machine learning, pp.
794 803. PMLR, 2018.
Christopher Clark, Kenton Lee. Ming-Wor Wei Chang, Tom Kwiatkossii, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume I (Long and Short Papers), pp. 2924-222220 2936, Minncapolis, Minnesota, June 2019. Association for Computational Linguisties. doi: wrg./19-- 1300
118865/31/111/ 1300 URL hattps://aclattaaaat
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge. 1rXiv:/8030055574 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. /4168, 2021.
Jacob Devlin, Ming- Wei Chang, Kenton Lec, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805 (es), May 2019. URL http://///// abs/1810. 04800
arXiv: 1810.04805.
Konstantin Dobler and Gerard de Melo. FOCUS: Effective Embedding Initialization for Monolingual Spe- cialization of Multilingual Models. Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 13440-13454, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/w1/0223.com www.p-mmmmppp.p....... main.82.., URL https://accaathhllo org/ www.e-naaaa...e.e maan.man... 829
Darius Feher, Ivan Vulić, and Benjamin Minixhofer. Retrofitting large language models with dynamic tokeniza- tion, 2024. http:: org/ 18553

---

Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal. and Tushar Khot. Specializing smaller language models towards multi-step reasoning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho. Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning. volume 202 of Proceedings of Machine Learning Research, pp. 10421-10430, PMLR, 23-29 Jul 2023. URL https::/ pttps://proceeddngs mlr. press/ gress/vv022) 222// u23d. html
Leo Gao. Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac"h, h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024. URL htttttttt///eeoode org/ rttpr//eoorrrro/eeeee 12608602
Leonidas Gee, Andrea Zugarini, Leonardo Rigutini, and Paolo Torroni. Fast Vocabulary Transfer for Language Model Compression. Yunyao Li and Angeliki Lazaridou (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural language Processing Industry Track, pp. 409-416, Ahu Dhabi, UAF, December 2022. Association for Computational Linguistics. doi: 10. 1118:////20222..mm--nnnaaas///io lcttps://aclanthooogyy org/
URL
org/2022 2022. commp-innduseee .11
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, et al. Gemma 2: Improving open language models practical size, 2024. URL URL https:t//atat....eeee
wwtps///rr/aba///////////// ht/////b///////// 001188
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, al. Gemma 3 technical report, 2025. URL littps://axivy org/ abaaabaaaaaaa 2203. 19786
Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Roziere, David Lopez-Paz, and Gabriel Synnaeve. Better & faster large language models via multi-token prediction. In Forty-first International Conference on Machine Learning. 2024. htLLLitttps//pppereiieeeeee// wtt.://wwww...o.rrroeeeeeeeeeeee
Aaron Grattafiori, Abhimanyu Abbinaa Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, et al. The llama herd of models, 2024. htt.:/atts//aaai..rrr///aaa
21783
Dirk Groeneveld, Beltagy, Evan Walsh, Akshita Bhagia, OLMo: Accelerating the science of language models. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (cds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 15889, Bangkok, Thailand, August 2024. Association for Computational Linguisties. doi: (011865531112222 acl-ooog.cl--o Utt.ttts:: httpp:://aclaathol..gg/oo/o/////
org/2024. acl-long. 841g
Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. MiniLLM: Knowledge distillation of large language models. In The Twelfth International Conference on Learning Reprexentations, 2024. URL https:://openreevee tett/forumiid-hhgqfff
Dan Hendrycks. Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang. Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Thirty-fitth fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL //operevvie net ntt/teeummmddeeeeoo 2yyoorrtmmmmddddyyyyyy2m22e
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the Knowledge in a Neural Network, March 2015. URL http:://aaxvv.org/abb///ta www.ooggg//ooooooooooooooee 1503. 02531 arXiv:1503.022 [stat].
Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL
Yichong Huang, Xiaocheng Feng, Baohang Li, Yang Xiang, Hui Wang. Ting Liu, and Bing Qin. Ensemble
learning for heterogeneous large language models with deep parallel collaboration. In The Thirty-iighth Annual Conference on Neural Information Processing Systems, 2024. URL |RL/ttts:://///o sttps://openreviers #oru=?id=77rAADDKG)
Leonid V Kantorovich. Mathematical methods of organizing and planning production. Management science, (4):366 422, 1960.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. Yoshua Bengio and Yann LeCun (eds.). 3rd International Conference on Learning Representationss, ICLR 2015. San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http:tttp///rriiooo www/////////aae//e/ee/oeeeeeeees 1482.69800....1
Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se- Young Yun. DistiLLM: Towards streamlined distillation for large language models. In Forty-first International Conference on Machine Learning. 2024. URL https://openreviev.u hwt.://oooooodddddo/lllll
12

---

Taku Kudo and John Richardson, SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing. In Eduardo Blanco and Wei Lu (eds.), Proceedings of the 2018 Conference on Empirical Methods Natural Language Processing: System Demonstrations, pp. 66-71. Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10.18653/w1/D18-222 2012. URL httpttt///aalaanhooolll
org/D18- 2012
Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathematical statistics. 22(1):79-88) 86, 1951.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, et al. Tulu 3: Pushing fronticrs in open language model post-training, 2025. URL https://arxiv, org/ attp//bss//11.... 15124
VI Levenshtein. Binary codes capable correcting deletions, insertions, and reversals. Proceedings of the Soviet physics doklady, 1966.
Costas Mavromatis, Petros Karypis, and George Karypis. Pack of Ilms: Model fusion at test-time via perplexity optimization, 2024. URL http:hhttts//tttee
baaprg/abs/2440 2404. 11531
Benjamin Minixhofer, Fabian Paischer, and Navid Rekabsaz. WECHSEL: Effective initialization of subword embeddings for cross- lingual transfer of monolingual language models. Technical Report arXiv:21 12.06998. arXiv, May 2022. htt::// /ttp:v///rvvvvv..///a//e. org/ abs/ 06598 arXiv:2112.065988 [es] type: article.
Benjamin Minixhofer, Edoardo Maria Ponti, and Ivan Vulić. Zero-shot tokenizer transfer, 2024. URL https 7/arxiv. owwaiiioog//ioe hwwpg//bb/2445.. 07883
Mistral AL. Mistral small 3.1, March 2025. URL
Ac-
cessed: 3025-05-08.
Piotr Nawrot, Jan Chorowski, Adrian Lancueki, and Edoardo Maria Ponti. Efficient Transformers with Dynamic
Token Pooling, In Anna Rogers, Jordan Boyd-Graber and Naoaki Okazaki (eds.), Proceedings the 61st Anmual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 6403 6417, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 1188666///222..... acl- act-ooo..-- URL https://aclanthholooo
org/2023. acl llong. 353
Saul B Needleman and Christian D Wunsch. A general method applicable to the search for similarities in the amino acid sequence of two proteins. Journal of molecular biology, 48(3):443-453, 1970.
Aitor Ormazabal, Mikel Artetxe, and Encko Agirre. CombLM: Adapting black-box language models through small fine-tuned models. Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 2961-2974, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 1/6633/11202..emm//l/ ma..:...mmmmm-m..18.0..... 180. URL https
//aclanthology. org/ wwg.oogg//202/ 2023. emnlp-mamee 180/
Artidoro Pagnoni, Ram Pasunuru, Pedro Rodriguez, John Nguyen, Benjamin Muller, Margaret Li, Chunting Zhou, Lili Yu, Jason Weston, Luke Zettlemoyer, Gargi Ghosh, Mike Lewis, Ari Holtzman, and Srinivasan Iyer. Byte Latent Transformer: Patches Scale Better Than Tokens, December 2024. URL htt/ttp::/ http:///ax/iii org/ abs/ 2bb//241...mm 09877 arXiv:2412.09871 [cs].
Buu Phan, Brandon Amos, Itai Gat, Marton Havasi, Matthew Muckley, and Karen Ullrich. Exact Byte-Level Probabilities from Tokenized Language Models for FIM-Tasks and Model Ensembles, October 2024. URL httt:://axiiv org/ absaaas//aa /2100 09303 arXiv:2410.09303 [cs].
Tiago Pimentel and Clara Meister. How compute the probability of word. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.). Proceedings of the 2024 Conference on Empirical Methods Natural Language Processing, pp. 18358-18375, Miami, Florida, USA, November 2024. Association for Computa- tional Linguistics. doi: (0.11655/v1/2024.coo emnlp- main. 1020/
main. 1020. URL Rttl:tttts::/ccattttt
org/ 2024
Qwen, :, An Yang. Baosong Yang. Beichen Zhang, et al. Qwen2.5 technical report, 2025. URL https abs 200.../211....0
//arxiv. perxiv.orgge wwt.://..g////////e/e/oo
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language Models are Unsupervised Multitask Learners. 2019.
Alfréd Rényi. On measures of entropy and information. In Proceedings of the fourth Berkeley symposium on mathematical statistics and probability, volume I: contributions to the theory of statistics, volume 4. pp. 547-562. University of Califoria Press, 1961.
13

---

Phillip Rust, Jonas Pfeiffer. Ivan Vulić. Sebastian Ruder. and Iryna Curevych. How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models. arXiv:2012.15613 [cs]. June 2021. URL http://arziv. org/ abs/ 2012. 15613
arviv: 2012.15613.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, laster, cheaper and lighter, 2020, htlLhhtps://aaatii org/ abs 1tt///9900001100
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural Machine Translation of Rare Words with Subword Units. In Katrin Erk and Noah A. Smith (eds.), Proceedings of the 54th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1715-1725, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10. N..18653/v1/116 11111//1111122221 162. StRpsttts://ant aathoolc org/P16-118
Ekansh Sharma, Daniel M. Roy, and Gintare Karolina Dziugaite. The non-local model merging problem: Permutation symmetries and variance collapse, 2024, URL URL htttps://axxiv.. hwtpe/ivw.rrg/bba/421233 12766
Zejiang Shen, Hunter Lang, Bailin Wang. Yoon Kim, and David Sontag. Learning to decode collaboratively with multiple language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 12974-12990, Bangkok, Thailand, August 2024, Association for Computational Linguistics. doi: 10. 1/623////002.aac/ooooo 701. URL htt.::ttp::s/eet httte::://aclannhoooggg.... org/ acl 1on...0117 701/
Yi Tay, Vinh Q. Tran, Sebastian Ruder, Jai Gupta, Hyung Won Chung, Dara Bahri, Zhen Qin, Simon Baumgartner, Cong Yu, and Donald Metzler. Charformer: Fast Character Transformers via Gradient-based Subword Tokenization. In International Conference on Learning Representations, 2022. URL httptttsps//opperiiiaaaa Dst:/forum2iidJJBBBr/ll
Daniil Tiapkin, Daniele Calandriello, Johan Ferret, Sarah Perrin, Nino Vieillard, Alexandre Ramé, and Mathieu Blondel. On Teacher Hacking in Language Model Distillation, February 2025. URL /ttp:/ttt:///axii org/ abs/ 02671 1IXv:2502.00261 [es].
Shubham Toshniwal, Wei Du, Ivan Moshkov, Branislav Kisacanin, Alexan Ayrapctyan, and Igor Gitman. Openmathinstruct-22 Accelorating ai for math with massive open-source instruction data, arXiv preprint arXiv:2410.01560, 2024.
Hugo Touvron, Louis Martin, Kevin Stone, et al. Llama 2: Open Foundation and Fine-Tuned Chat Models, 2023. _eprint: 2307 3907809288.
Ke Tran. From English To Foreign Languages: Transferring Pre-trained I Language Models. Technical Report 2xXiv20002007306 arXiv, April 2020. URL http/http///axiii.oo org/ http:/aaa/2200000/ 2000000000000 arXiv:2002.073366 [cs] type: article.
Omri Uzan, Craig W. Schmidt, Chris Tanner, and Yuval Pinter. Greed is All You Need: An Evaluation of Tokenizer Inference Methods, 2024. _eprint: 2403.01289.
Tim Vieira, Ben LeBrun, Mario Giulianelli, Juan Luis Gastaldi, Brian DuSell, John Terilla, Timothy J. O' O'Donnell, and Ryan Cotterell. From Language Models over Tokens to Language Models over Characters, December 2024. URL httt:://axxiii.. wtw.aaawwi.....raaaooaaaa hbwggg/bb////// 2412. 03719 arXiv:2412.03719 [cs].
Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, and Shuming Shi. Knowledge fusion of large language models. In The Twelfth International Conference on Learning Representations, 2024. URL
https://openrevierr net. www./eeewrrmineeeeeeeeee ddd.smii2d
Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. A survey on knowledge distillation of large language models, 2024. /ttpp://///x
org/ htt//abb24422...0 13116
Linting Xue, Aditya Barua. Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. ByT5: Towards a free future with trained byte-to-byte byte models. 1056 [cs], March 2022. URL http::/ http://awivv.rraaa////aaaaa abs 2205 13626
arXiv: 2105.13626.
François Yergeau. Utf-8, a transformation format of iso 10646. Technical report, 2003.
Lili Yu, Daniel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettlemoyer, and Mike Lewis. MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers. In Thirty-seventhh Conference on Neural Information Processing Systems, 2023. URL httphhtttt///eeeeeeeeee www.:eee//tooeeeo wttp://wwwww.eeeeetoorrr/ddddddd wtw.o//mmm...m0m0002222m
14

---

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can machine really finish your sentence? In Anna Korhonen, David Traum, and Lluis Mårquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, pp. 4791-4800, Florence. Italy, July 2019, Association for Computational Linguisties. doi: 10.18653/v 1472. URL https? aaclanthology.org//a.. wttp///-////77/111
Peiyuan Zhang, Guangtao Zeng. Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model, 2024a.
Songming Zhang, Xue Zhang, Zengkui Sun, Yufeng Chen, and Jinan Xu. Dual-Spact Stal-Space Knowledge Distillation for Large Language Models. In Yaser Al-Onaizan, Mohit Bansal, and Yun- Yun-Nung Chen (eds.), Proceedings of the
2024 Conference Empirical Methods in Natural Language Processing, pp. 18164-18188 18181, Miami, Florida. mamnn..p.iin..... main. 1010. USA, November 2024b. Association for Computational Linguisties. doi: 10.1 111855/v122224444 h/t0://emllpmm...m...
URL htttpp::/aaaaannoollggg.. wrg/ emnlp- main. 1010/
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang. Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark evaluating foundation models, 2023.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-fonn folloowing evaluation for large language models, arXiv preprint 1.07911 2023.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The claims made in the abstract and Section D match the results in Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss limitations Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
16

---

Justification:
We provide proof' around the binarised f f-divergence in Appendix
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Appendix B states our common experiment setup, and various sections through- out the paper state how we adjust this setup for any particular experiment. Together, this provides the full details to reproduce our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
17

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: Code is not submitted alongside this paper but will be provided upon publica- tion.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We provide training details Appendix and details on hyperparameters and how they were chosen in Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report the standard error of the mean in the experiments on Use Case 2 For Use Case aadd Use Case we perceive quantifying statistical significance not be necessary due to the magnitude and consistency of improvements over prior methods.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
18

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Compute resources and execution times are specified in Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We adhere to all applicable points of the Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss our methods' positive impact on enhancing reusability and compos- ability of LLMs in Section [ We do not believe our method leads to any particular negative societal impact beyond the potential negative impact of increased LLM adoption.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
19

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
We do not release any standalone models; released fine-tuned models are bound the model checkpoints there were fine-tuned from, including being bound the safeguards put on the fine-tuning starting point.
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Asset creators are credited, were applicable, throughout the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
20

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com/ com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: We will release fine-tuned model checkpoints created via training with our method upon publication alongside our code. These assets are not vital our method itself, and we thus do not make them part of this submission.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemmodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
There are no human subjects involved in the experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: There are no human subjects involved in the experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
21

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions NeurIPS Code of Ethics and the
and locations, and we expect authors to adhere guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: Our work does not involve applying LLMs in any way.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, nta-standard components.
Please refer L.LM policy (https://neurips) httppp::/naarrpp..cc/ooooeeeeeee for what should or should not be described.
Ablations & Sensitivity Analyses
A.1
Sensitivity to the Learning Rate
We analyse the impact of different choices for the learning rate across all methods in Table The different methods generally robust settings for the learning rate within the range of 2 to 5e-5, although we occasionally observed loss spikes when choosing high learning rates within this range in preliminary experiments. Accordingly, set the learning rate to le- for all methods in our main experiments.
Table 6: Sensitivity of different methods the learning rate on transfer Gemma 2B the Qwen2 Tokenizer by training for 5k steps (and otherwise matching the experiments in Use Case I).
Benchmark
Method Learning Rate
Avg.
PIQA
ARC.C
BoolQ
MMLU
AGLEN
AGL-ZII
IFEal
2e-6 6
76.1
41.8
81.1
50.4
34.5
29.9
46.8
51.5
SFT
le-5
76.3
42.2
80.3
50.2
33.4
29.8
51.8
52.0
42.7
78.5
50.5
28.5
54.9
51.8
6
75.8
41.7
75.6
49.5
33.4
30.5
42.7
49.9
DSKD
1e-5
74.6
40.9
80.0
49.0
33.1
29.8
52.2
51.4
56-5
74.0
41.4
80.3
51.6
32.3
29.4
54,8
52.0
2e 6
76.0
41.1
78.1
52.0
35.9
30.0
52.6
52.2
MinED
le-5
76.6
44.0
81.9
52.2
36.3
29.7
57.5
54.0
5e-5
76.3
43.8
80.1
51.7
32.8
28.5
58.1
53.1
2 6
75.7
46.0
83.4
53.7
40.1
32.9
57.4
55.6
ALM
1e-5
76.8
48.0
83.1
53.8
39.3
32.0
55.5
55.5
5 5
76.8
49.1
81.9
53.8
38.9
30.9
54.8
55.2
A.2
Impact the Choice for the Distance Function. and the Temperature
The distance function f and the temperature two crucial hyperparameters of our method. We analyse two choices of f. the KL-Divergence fKI. (p' = p /og ς and the total variation distance (Tv
1//
Ip g²/r| across range of temperatures. Figures and show the gradients with respect log q the binarised outcomes of fKL and frvD. respectively. For
22

---

loga (precictinn)
longer duanks
longer chanks
long
longar
-2000
longer dhunks legp (trgett
longer dhunks
longer chunks
longer chunks
Figure 4: The KL-Divergence gradients
8 olog
1
over T.
r=1
T= 2
T=
T 40
loge (prediciona)
longar chunkss
Iongar chunks
longer
longuer
-2000
20.0
long diunks logo (targeti
0.0
longer chunis
longer chunes
chunes
longer thunks
Figure 5: The Total Variation Distance gradients ffrvn [1 1/
frvm
1/5)
1 (
blug
over 7.
small temperatures (e.g.. = 1), the gradient vanishes as log or log decreases. In general, this expected behaviour: mismatches in higher p or q should generally result in a higher contribution to the loss gradients than mismatches when and are both low. However, importantly, our method operates on chunks tokens and the likelihood a chunk multiplicatively decreases as the length of the chunk increases (since the likelihood of any token is always less than I); this means that the contribution of longer chunks to the loss vanishes the temperature is low. The value of T thus likelihood chunks. Increasing leads increased focus on lower-liked likelihood/longer chunks [5] We also observe notable behaviour of the KL-Divergence and Total Variation Distance - oc, and that the gradients
defines trade-off: low leads to focusing high-likelihood
can be computed in closed g1/T) C(llgg log q) log(p) log( log + fivD(1 - p p /17) = logp log g as + oc. To show this, we will use the fact that exp(x) = 1 + z for small enough and let log logp//r = log(p)/ and b = log //r log(g)/T, both of which are close to zero since r - co. We have
form. Specifically, fKL. (p' [T) fka. (1-p1/
and
p1/r
fKL. (11///// = 1T log
91/T
log exp(b))
exp(a)(a-1
= (1+ a1+ -b)
since 7 +
a a² - b 3--ad ab
log(p)/T log(p) 2/72 log(g)/r - llg/p) log(g)/7² (log(p) log(g))/T
since 1/7 >>>>>>>>>>>>>/ 1/72
C(log - log q)
where the constant C' = 1/T
and
DDsentangllng the loss contribution by longer chunks compared to shorter lower- likelikkeeeddd chunks could be avenue for future work.
23

---

(1 pl/7)
fKL (01-p//111 p1/1 g = - (1-pt/) log
(1 - (1/))
- cxxp(aa (log(1 exp(a)) - log((1 cxp(b)))
~ -a(log(-a) log(-b)) since 7 + 00
log(() log log( (llo-llo)///
log
C log(p) log
where the constant C' 1/7
logp
Analogously, for the Total Variation Distance have
) fivo(1 (00(11pp/1111-000 p-/1111-9t/1 -1
=
=lpl/T 9²/T1
= -/exp(a) - cxp(b)
(1++a a) (1 + b)l
since 7
-aa-b.
|log(p)/T log(g)/rl log(p) - log(4)
where the constant = 1/T
Empirically, both KL-divergence and Total Variation Distance perform well. and is beneficial (Table 7 We opt for the Kl.-divergence for our main experiments and set the temperature 7 100 16
Table 7: Sensitivity to the distance function f and temperature on transfer of Gemma 2B IT to the Qwen2 Tokenizer by training for 5k steps (and otherwise matching the experiments in Use Case 1).
Benchmark
Distance function
Temperature
Avg.
PIQA
ARC-C
BoolQ
MMLU
AGI-EN
AGI-ZH
IFEval
76.4
47.2
53.6
40.0
31.4
61.1
56.
76.5
47.6
83.3
53.7
39.7
32.3
60.7
56.3
== fkt.
100
76.8
48.0
83.1
53.8
39.3
32.0
55.5
55.5
т + 0c
76.3
47.7
83.2
53.6
39.0
32.4
54.4
55.2
1
75.8
46.8
83,5
53.9
41.5
32.0
58.9
56.1
75.6
47.9
53.7
40.0
32.1
61.0
56.2
f fIVD
100
76.3
48.9
82.9
53.6
39.2
32.2
55.6
55.5
76.1
48.4
82,9
53.7
38.6
32.9
52.9
55.1
A.3 Ablating Outcome Chunk Debiasing, the threshold 7 and Hidden States Distillation
We ablate the impact of Outcome Chunk Debiasing (Section 3.1 well the associated threshold and the Hidden State Distillation Loss (Section 3.2) Table g Notably, the hidden state loss is slightly detrimental to Subword transfer. Outcome Chunk Debiasing is consistently highly beneficial, and can marginally further improved via threshold 7 > O.
Subword transfer, but highly beneficial to Subword Byte
A.4 Comparing Loss Combination Strategies: Fixed (3 vs. GradNorm vs. GradMag
We compare the different loss combination strategies discussed in Section 3.3 in Table GradMag performs on par or slightly better than GradNorm across methods, and both perform vastly better than different fixed combination coefficients 8, while exhibiting only small overhead from the additional per-task proppppppaattionn through the last layer. Notably, GradNorm requires an extra optimizer (and tuning the associated hyperparameters) which method does not.
16,- 5 performs better and might be a better choice for future experiments. We chose 7 100 for our main experiments by optimizing for our initial method (see Appendix (F: higher temperatures might have been more important for the initial version of method since it exhibited longer tail of low tow-rrobayility chunks.

---

Table 8: Effect of varying the bias threshold on transfer of Gemma 2B IT to the Qwen2 Tokenizer and Llama 3B IT byte-level tokenization, training for 5k steps and otherwise matching the experiments in Use Case D denotes the addition of Outcome Chunk Debiasing (Section 3.1 and H denotes the addition of the Hidden State Loss (Section 3.T).
Benchmark
Model Tokenizer ALM Setting
Avg.
PIQA
ARC-C
BoolQ
MMLU
AGI-EN
AGI-ZH
IFEval
7 0.10, (default)
76.8
48.3
83.1
53.6
38.9
32.2
56.0
55.6
0.99, D
77.1
48.0
83.0
53.7
39.3
32.6
53.7
55.4
0.50, D
76.6
47.9
83.4
53.5
39.8
33.2
49.6
54.9
Gemma2 - Qwen2
Y = 0.00,
76.4
47.2
83.1
53.6
39.0
31.4
58.0
55.5
Y 0.00, H
76.6
48.0
82.8
54.0
36.6
31.5
55.8
55.0
7 0.00
76.2
48.0
83.0
53.6
39.0
32.4
46.7
54.1
0.10, D, H (default)
73.2
38.9
77.6
55.8
34.0
32.7
18.0
47.2
= 0.99, D, H
73.1
39.8
78.2
56.1
35.3
31,8
17.3
47.4
? = 0.50, D, H
73.1
38.8
76.3
55.9
33.4
31.6
17.7
46.7
Llama3 -> Byte
7 0.00, D, H
73.5
38.8
77.1
56.5
34.4
30.9
17.7
47.0
0.00,
73.2
40.3
75.1
52.5
30.0
29.3
16.9
45.3
0.00
73.4
41.2
74.0
52.5
30.0
31,0
0.2
43.2
Table 9: Analysis of the efficacy of (i) different fixed coefficients / (ii) GradNorm (Chen et al. 2018)
and (iii) GradMag (Section 3.3 at transfer of Gemma 2 2B IT to the Qwen2 Tokenizer by training for 5k steps and otherwise matching the experiments in Use Case
Benchmark
Method
Distill. Coef.
8
Avg.
PIQA
BoolQ
AGL.7H
IFFval
SFT
76.3
42.2
80.3
50.2
33.4
29.8
51.8
52.0
3 = 1/3
75.5
42.2
79.9
49.1
32.7
31.6
52.8
52.0
3 = 1
75.2
39.8
80.7
47.7
32.3
30.3
49.3
50.8
DSKD
3 3
72.5
35.5
81.3
46.6
31.6
30.9
47.9
49.5
GradNorm
75.3
41.6
79.7
49.0
32.8
30.3
51.3
51.4
GradMag (ours)
74.6
40.9
80.0
49.0
33.1
29.8
52.2
51.4
3 = 1/3
76.5
42.5
80.4
49.8
33.4
29.4
51.8
52.0
8
76.3
42.7
80.3
50.0
33.4
28.4
50.9
51.7
MinED
76.3
43.1
80.5
50.4
33.6
29.0
51.2
52.0
Gram
76.3
44.5
81.2
50.9
36.0
29.6
53.8
53.2
GradMag (ours)
44.0
52.2
29.7
54.0
1/3
42.3
49.8
34.0
29.3
$2.0
52.0
8 1
76.4
42.6
79.5
50.1
34.4
29.4
52.2
52.1
ALM + SFT
3 3 3
76.3
42.2
80.0
50.4
34.7
30.0
52.6
52.3
GradNorm
77.1
46.0
83.2
53.7
38.7
32.2
61.6
56.1
GradMag (ours)
76.7
46.0
82.8
53.4
37.9
32.1
58.8
55.4
B
Training Details
Across experiments, unless specified otherwise, we use a batch size of 64 texts and sequence length of 512 tokens for both student and teacher. We use Adam (Kingma & Ba 2015) without weight decay following the adaptations settings of Groeneveld et al. (2024). We choose a default peak learning rate of le based the findings in Appendix A. training for 20k steps with linear warmup over 2k steps, then linear decay to zero. We use GradMag (c.f. Section 3.3) to balance loss components. We use LoRA (Hu al. 2022) with
64. For ALM, we use threshold ς 0.1, the temperature 7 = 100, and set f to fKL (pr [ps) Pr log Pz to recover the KL-divergence, analysing ps the impact of these choices in Appendix A.2 We conduct all experiments on a cluster of 40 v3 TPU chips and v4 TPU chips. The largest individual experiments run on a pod of 32 v4 TPU chips and take z 24 hours for transfer of Llama3 to byte-level tokenization, ~ 12 hours for transfer of Gemma2 to byte-level tokenization, Rs days for Gemma2 hypernetwork training and as 5 to 10 hours individually per remaining experiment.
25

---

C Adjustments for Transfer to Bytes in Use Case
For the transfer bytes, we start from the hyperparameters described in Use Case I, with the following modifications. We quadruple the student sequence length to 2048| halve the batch size 32 texts, increase the learning rate 3e-5 and train the full model. Adding the hidden state distillation loss leads to consistent improvements (while results are mixed in the subword case, c.f. Appendix A.3), so we add it here and do not add for subword-te wwwwwortto-suwweeee to-subword transfer. We also untie the embedding matrices since sharing the input and output embedding parameters has negligible impact on the parameter count when the vocabulary just consists of the 255 possible bytes. Transfer bytes substantially decreases the parameter count due to shrinking the embedding matrix (which can make up a considerable portion the total parameters). Thus, to make it less destructive, we devise a strategy to re-incorporate the subword embedding parameters: for every byte position, we backtrack to find the longest matching subword token ending at this position. We then add the embedding of this token to the byte embedding. This is akin to BLT's strategy of adding n-gram embeddings Pagnoni itt al. (2024), where in our case the vocabulary is given by the sloword-basee model. This strategy only accounts for the parameter mismatch at the input layer, not at the output layer, might still expect decreased performance.
D Hypernetwork Architecture Modifications for Use Case 3
We simplify the hypernetwork training procedure introduced by Minixhofer et al. (024) to only require a single stage instead of the original two-stage training procedure and remove the necessity of the auxiliary embedding similarity loss. While Minixhofer et al. (2024) predict the target embeddings as _ Ho( (E""), we switch to a residual formulation E E" [O + Hp (im)) where He the hypernetwork and ET is the input embedding sequence. This way, the hypernetwork (i) does not need initially learn mimic the input embeddings for sequences of length (the purpose of the original stage one) and (ii) is less prone to catastrophically drifting in embedding space throughout training (enabling removing the auxiliary loss). We first reproduce Minixhofer al.] (2024)'s results on the LimyL model (Zhang et al. (0244), then test the impact of switching our residual formulation Table 1O Based on these findings, we use residual hypernetwork without auxiliary loss our main experiments.
Table 10: Performance of a TinyLlama hypemnework trained with Minixhofer al. (2024) setting, vs. our residual setting. While removing the auxiliary loss is catastrophic in Minixhofer et al. (2024)'s setting, residual setting allows removing at only minor performance degradation. We exclude BoolQ and MMLJ from this comparison since TanyL.ama performance is not better than random.
Benchmark
Method
Tokenizer
Ayg.
PIQA
ARC-F
ARC-C
AGLEN
AGI-7H
HS
original
73.1
55.2
30.7
23.3
27.8
59.1
44.9
B-Shol
GPT2
70.9
48.8
28.3
22.9
27.5
55.9
42.4
Minixhofer et al. (repraduced)
2024
O-shel
Mistral v2
70.6
50.3
28.8
21.9
27.9
54.0
42.2
Llama3
70.5
49.7
28.2
21.9
27.1
53.8
41.9
Average
70.6
49.6
28.4
22.2
27.5
54.5
42.2
B-chet
GPT2
71.4
50.1
30.4
21.5
26.6
56.4
42.7
SFT (Residual Lcx.)
D-shel
Mistral
70.3
50.3
29.7
21.2
27.1
54.7
42.2
Llama3
71.0
49.8
29.6
20.8
26.5
54.2
42.0
Average
70.9
50.1
29.9
21.2
26.7
55.1
42.3
0-sbot
GPT2
70.4
49.2
28.9
21.8
26.2
55.9
42.1
SFT (Residual)
O-sbot
Mistral
69.4
49.6
29.1
21.5
26.8
53.8
41.7
Llama3
70.0
49.5
28.5
22.1
26.5
53.5
41.7
Average
69.9
49.4
28.8
21.8
26.5
54.4
41.8
The increase in sequence length is necessary since token consists of =z 4 bytes average.
26

---

E Relation of the Binarised f-divergence to the Categorical f-divergence
Recall that we define the divergence induced by f as Dy(pll)) = (((())l((z)) Here, f must be writable as (p((<))la(z)) (())g
pla)
where is convex and non-negative by the definition of
g(x)
f-divergence (Rényi 1961). The binarised f-divergence is Diimarssed (pllg) f(p(x)ll(<)) + f(1 p(=)111 g(x)) for some D. We will show that the binarised f-divergence is lower bound to the categorical f-divergence. We have
Dy(pllg) Σ f(p(x))l((x))
=((p(=))a(())
Σ ((p(a))la(x'))
splitting the divergence into two parts
<'=D(()
Since the first term of the split D, and Diinarised is the same, it remains show that
f(1 - p(x))( {(x)) s Σ
(((a')lla(2'))
<'wD((a)
We expand 1 - p(z) and 1 q(x) and insert ((x)g
p(z)
f(p(=))la(z))
((xr)
EnnDrta p(x')) g ExnDiiaa g(27)
p(a') g(x')) s(x')
g(d's M')
Σ
Σ [>3
I
which is true by Jensen's inequality since is convex.
Alternatively, it follows from the data processing inequality that the binarised f-divergence is lower bound to the categorical f-divergence since post-processing (by merging all but two options) can (
only decrease the divergence. Since Dpinarised (pll4) is an f-divergence, the important properties that Dhinarised (pllg) = O iff p = q and Dyinarised (pll4) > otherwise also hold.
F A Variant of ALM Which Explicitly Takes Tokenization Bias into Account
We initially experimented with a version of ALM which explicitly selects chunks with low approxi- mate differences tokenization bias. This was intented make likelihood comparison more accurate by minimizing spurious differences caused by differing tokenization biases between the teacher and the student sequences. We did not end up choosing this formulation since it requires pre-computation step like MinED, which results in additional complexity and prevents online applications such as Use Case while performing roughly equally well. Nonetheless, we believe this previous formation and the associated formalisms may be useful to the community, and thus provide a description in the following, starting with some additional preliminaries.
Mcasuring Tokenization Bias. We can measure tokenization bias via the notion of cover encod- ings (Phan al., 2024)). The cover encodings
sequence of tokens = T(x)) are sequences of tokens such that (i) decoding and re-encoding the sequence leads to the same token sequence T(D(u)) = u (this has been referred to as validity), (ii) æ is a prefix of D(u) and (iii) æ is not prefix of D (i.c., the last token in covers part of x) 18 We denote the set of all cover encodings as cover(t). Note that t E cover(t). We further defne the set of implied exclu- sions (t) (D(u) u E cover(t), u o 1). (x()) is a concrete measure of the tokenization bias of any particular token sequence t. If we observe the token sequence T(æ)., we know that æ & (t). Otherwise, x would have been tokenized differently. our previous example,
Hellllllll BeelloWWrrld World E
See Phan et al. 2024 for more detailed exposition of cover encodings.
27

---

Choosing Chunks with Low Approximate Tokenization Bias Difference. We would now like to constrain chunk alignment such that the prefix and continuation are biased in the same way, i.e., (T)) = (TT( and (z)s(z)) = (TT((z)). Unfortunately, this requirement is prohibitively stringent in practice: for example, if the student tokenizer is unbiased while the teacher tokenizer is not, it might rarely (if ever) be fulfilled. We thus define relaxation which lets us define the alignment constraint c(i, k. 1) (c.f. Equation Equation/Aligmment Alignment Indices) such that the chunk bias differences are small enough, but not necessarily zero. To this end, we introduce precomputable scalar approximation of the difference in tokenization bias between two sequences. Let us consider arbitrary language models A and B (not necessarily teacher and student). We first define (A' \ Kg) (x) to be the set difference of implied exclusions X' on some text x between the token sequences of the two i.e. (Xa) \ Xx) (x) = (TA (xx) (TT(( We can then quantify scalar difference b ballB (2) in tokenization bias between the two sequences.
(a))
E
(d(Tx(d))
dE (o)(())
If (A' \ (x)(x)) = 9. (xx) is biased to a lesser or equal extent as TB (2:) and A (x) = 0. not, & (xx) will be high if any text d which TA is biased against but Tn is not is assigned
a high probability by the language model A. Instead, if a text d E (Xa xxx((a) is assigned
a low likelihood, it affects the bias difference to a lesser extent. We can use the symmetrised bias (x) = max(bB) (xr), bAllB (a)) as measure of the 'biasedness' of any text. However, computing bnj (xx) is prohibitively expensive since we need to compute the cover encodings of every text a under both tokenization functions, as well as the likelihood assigned by the language model to every text d in the difference between implied exclusions (Aa \ (x)) Therefore, we make two key approximations to estimate bB) (xx) Rs (a)) (and analogously (3)A(()) se P (z)) at nearly zero computational overhead.
Approximating (T)))) via (z(())),) We approximate the implied exclusions of T(x) via the implied exclusions of the last token 7 T(x), in T(x).
(T)))) = (D(T()) ou E ((T(2))))
(Bias Approx. 1)
This makes it possible to precompute the implied exclusions (t) for all t E V.
Approximating p(T())) via Punigram (T()) We approximate the language model likelihood (x)( via the unigram likelihood Punigram (T()).) computed via the token counts of a text corpus.
(TT())) = Ponigram (T(x)),)
(Bias Approx. 2)
This approximation is crude by necessity since it is distinctly intractable to run every token se- quence d (X' \ (x) through large language model. However, show later in Section 4 that performs well in practice. [Bis Approx. I and Bias Approx. 2 together make the bias only depend the last token in the sequence. Going back to our distillation setup, the above approximations allow precomputing the bias difference bs,T(x) for any text by enumerating all possible last token pairs (ts.tr)= Vs x Vr and storing the result in a sparse matrix in RiVs|x(Vr This achieves our goal of defining an efficiently computable scalar metric to measure the differ- ence tokenization bias between two token sequences, and lets us set the alignment constraint to e(i, k. ==
))), (D(((((())))) S 7 where 7 is manually defined bias difference threshold. Together, these approximations give result in a viable alternative formulation of ALM, although one which requires pre- pre-computation See Table IT for a minimal comparison of this variant with our main method.
G Overview of Existing Cross-Tokenizer Distillation Methods
ULD (Boizard et al., 2025) trains the student via next-token prediction plus a Wasserstein distance loss (Kantorovich] 1960) between the teacher logits and the student logits at every step. This ensures that the absolute difference between the sorted teacher logits and sorted student logits at every position
is small.
28

---

Table 11: Comparing variant of ALM which explicitly aims to take tokenization bias into account (Appendix F) with the main ALM version on transfer of Gemma2 2B IT the Qwen tokenizer as in but training for 5k steps instead.
Use Case
Benchmark
ALM Setting
Avg.
PQQA
ARC-C
BoolQ
MMLU
AGLEN
AGI ZII1
IFEval
ALM
76.8
48.3
83.1
53.6
38.9
32.2
56.0
55.6
ALM Appendix
Variant
76.9
46.8
83.1
53.6
39.4
31.5
52.9
54.9
Table 12:
Ensembling results across Qwen and Llama pivots and across keeping/transferingg the special tokens. We adjust our notation as follows: X=3y now denotes transferring to tokenizer ) while preserving the special tokens of (the default Table 2). X-y denotes transferring regular and special tokens to tokenizer V We also explicitly denote the pivot used by the MinED baseline.
Benchmark
Size ID
Model
Avg.
PIQA
ARC-C
BoolQ
MMLU
AGI-N AGI-ZH
IFEval
2.6B g
Gemma2 2B IT
79.6
50.4
83.8
56.9
42.1
30.7
62.5
58.0
Original Models
3.2B
Llama3.2 3B IT
76.9
43.9
78.8
62.4
36.6
40.2
76.6
59.3
1.5B Q
Qwen2.5 1,5B IT
76.3
43.2
77.9
60.1
45.0
54,2
46.3
57.6
2 g Q Tok.
76.8
49.0
82.7
53.6
38.9
31.6
53.2
55.1
G-4 g Q Tok. Special Tok.
68.0
32.0
71.3
39.6
31.7
28.8
53.1
46.4
3.3B L - Q Tok.
77.3
45.6
79.0
61.6
37.1
33.3
76.3
58.6
3.3B C-> Q Tok. Special Tok.
75.5
46.0
78.4
60.1
36.8
32.3
74.0
57.6
Transfers
2.4B g + Tok.
77.1
48.4
83.0
53.7
39.2
32.0
53.8
55.3
2.4BG-c g Tok. Special Tok.
77.1
48.0
82.9
53.7
38.9
31.8
48.5
54.4
1.58 Q Q Tok.
76.3
45.9
78.4
59.6
44.7
41.9
45.9
56.1
1.5 Q- Q L Tok. + Special Tok.
76.1
44.9
77.7
59.3
44.2
45.7
44.7
56.1
7.3B
Q+ (MinED-pp. pivot)
76.9
46.2
83.2
64.1
40.6
39.4
65.9
59.5
7.3B
L+G+Q (MinED-p. C pivot)
76.9
47.2
83.1
64.2
38.5
37.7
54.9
57.5
7.2B
Q+ L. (p)
75.6
45.6
77.3
62.1
42.0
44.9
49.1
56.6
Ensembles
7.2B
C C+G-s+
77.5
48.7
83.7
62.9
43.3
42.5
64.8
60.5
7.2B
@+9 + + (p)
77.5
48.0
83.4
63.0
43.6
45.7
64.3
60.8
7.2B
C+ Q- (p)
77.6
48.0
84.1
63.2
43.6
43.3
62.3
60.3
Larger 7.6B
Llama3.1 8B IT
81.6
53.9
85.4
68.4
47.3
47.1
81.7
66.5
Models
8.0B
Qwen2.5 7B IT
80.4
54.3
86.4
71.7
57.6
72.6
76.6
71.4
MinED (Wan et al., 2024) aligns the student and teacher logits by greedily aligning every token in the students' vocabulary with the teacher token which has the minimal Levenshtein distance (Levenshtein) 1966) to the student token. The main objective is again minimizing cross-entropy of the next-token prediction. The KL-divergence between the student logits and aligned teacher logits is added at every position where there is one-to-one alignment between the elements of the two sequences.
DSKD (Zhang al., 2024b) projects the student representations the teacher sequence and the teacher representations to the student sequence via cross-attention mechanism. then adds the KL- divergence between the teacher predictions and the -tutcce-to-teacher projected predictions (and vice versa) to the next-token prediction loss.
H Analysing the Choice of Ensemble Pivot & Special Tokens
For our main ensembling results (Table 2). we have chosen to transfer to the Qwen tokenizer, i.c., to use Qwen the 'pivot'. The question arises, then, of how the choice of pivot impacts performance. We additionally transfer Gemma2 2B and Qwen2.5 1.5B to the Llama3 tokenizer and ensemble with the (ron-tranfferred) Llama3 3B answer this question. Results are shown in Table 12 We find that, at an average score of 60.8 with Qwen as the pivot and 60.3 with Llama as the pivot, the ensemble of transferred models is robust to the choice of pivot. This is, however, not true for the MinED baseline: here, using Llama as the pivot catastrophically degrades the ensemble to perform on average worse than the individual ensemble constituents.
We use this opportunity to additionally study related subtle choice: whether to keep the special tokens (such <bos>, <eos>, etc.) of the original tokenizer, or use the special tokens of the new
29

---

tokenizer upon transfer. Different special tokens do not pose problem for sequence alignment since they can be easily detected and manually aligned. The same holds true for autoregressive generation. where it is fairly trivial to maintain separate running token sequences for the different models, which each insert their own special tokens as needed when one is generated. Nonetheless, this does result in implementation overhead, and for some use cases it may be convenient to achieve 100% parity with the target tokenizer by transferring the special tokens as well. Alongside the aforementioned ensembling results, Table 12 also shows the performance of models which had their special tokens transferred and ensembles of these models. For the Llama pivot, transferring the special tokens completely preserves performance. However, for the Qwen pivot, transferring the special tokens catastrophically degrades performance (especially the performance of the Gemma2 2B constituent). This is the case since Qwen2 does not use reginning-of-sequenn <bos> token. In models which do use a <bos> token, this token is often heavily attended to, behaving as an "attention sink' (Barbero et al. 2025). The models transferred to the Qwen2 special tokens thus have to relearn to distribute attention from the (now absent) <bos> to different tokens in the sequence. Evidently, Llama3 is able to cope with this situation better than Gemma2; more training of Gemma2 may fix this problem. Based on these findings, we preserve the special tokens in our main experiments since it is in general the safer choice, and the only downside for our purpose is additional implementation complexity.
I Implementing ALM
ALM is simple to implement efficiently by making changes to the data preparation and the loss computation.
Data preparation. Alongside tokenizing all example texts in the current batch, data preparation must include computing aligned chunks of tokens between the teacher and the student. To implement the chunk alignment algorithm, we can take advantage of the fact that both token sequences encode the same bytes. It is thus possible to compute alignments in a single pass via indices (i, k)), where i is increased as long as the token at position i in one sequence ends earlier than the token at position j in the other sequence, and vice versa, and indices are stored both tokens end at the same byte position. This is in contrast to more complex approaches used in prior work (Fu et al. 2023 Wan et al. (024):: they minimize edit distance between alignments via variants of the Needleman- Wunsch algorithm (Needleman & Wunsch| 1970). This is not necessary if the tokenizers have first been converted to the byte level. We can represent the computed chunks via binary matrices the batch size, the student sequence
M {0, 1xxmmx and N E 10,1)xxmx where b, m, length and the teacher sequence length, respectively and k = min (m. n) is the maximum amount of chunks (i.e. the minimum of the student and teacher sequence lengths). M and N have entries one if the i-th token is part of the j-th chunk and zero otherwise. The above can be done in a separate data-loading thread avoid blocking the main process.
Loss computation. Loss computation involves computing the student and teacher next-token log probabilities over input tokens log ps (z) b and log Pr (y(y))))) az) E Réxn where y the input tokens shifted by one as in the causal language modelling objective. We can then compute chunk log- probabilities Cs (log ps (y]
(x) ) and CT (logPT PT (y VV We then apply the chunk-level f-divergence element- to Cs and Cr and take the mean to compute the loss. In the simplest case where f fIvD and 5 + oc, this reduces to CALM (x) above results in negligible overhead compared the forward and backward pass over the Transformer backhone (two gathers, two matrix multiplications, and some element-wise comparisons). Alignment constraints c(i. j, k. 1) (c.f. Equation Equation|Alignment Alignment Indices) and chunk debiasing (c.f. Equation Outcome- Debiased Chunk-Level Probability) add some implementation complexity to the above but are cheap to compute and, in principle, optional.
1 b E,, (Cssi (Cr)uyl. All of the
30


Reviews:
[{"review_id": "LbeYzSQroa", "reviewer": "Reviewer_RibG", "summary": "The authors introduce Approximate Likelihood Matching (ALM), a new objective for cross-tokenizer knowledge distillation. The authors discuss the three use cases to show the effectiveness of the proposed method.", "questions": "1 How does ALM behave when the teacher’s vocabulary is much larger than the student’s?\n2 From the ablation study, it looks like the performance is very sensitive to some setups, like the learning rate and the selection heuristic. How do we determine the best setup when we are performing the proposed ALM?\n4 Authors claimed that the proposed method enables effective distillation across fundamentally different tokenizers. How do we measure the distance between tokenizers? It would be good to see the trend of using the proposed method on tokenizers with increasing distances.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths\n1 The problem definition is clear and the proposed method should be of interests for researchers in this area. \n2 The authors detailed discuss the three use cases to show the effectiveness of the proposed method. \n\nWeaknesses\n1 All experiments are ≤ 8 B params and ~0.6 B tokens. The effectiveness for frontier-scale LLM is unclear. \n2 Considering the small models and tokenizers, and the sensitiveness to the hyper-parameters, the performance improvement is slight.", "was_revised": true, "final_justification": "4: Borderline accept", "rebuttal": "Thank you for your review! We address your concerns below.\n\n> All experiments are ≤ 8 B params and ~0.6 B tokens. The effectiveness for frontier-scale LLM is unclear.\n\nWe favored breadth for our experiments (thorough ablations, sensitivity analysis, etc.) over scaling to large parameter counts. However, we do recognize the importance of showing that our method scales to larger LLMs. To do so, **we have conducted additional experiments transferring a larger LLM, Gemma3-12B, to the Qwen3 tokenizer**. The results are shown below.\n\n|                                  | PiQA | ARC-C | BoolQ | MMLU | AGI-EN | AGI-ZH | IFEval | Avg. |\n|----------------------------------|------|-------|-------|------|--------|--------|--------|------|\n| Gemma3-12B-Instruct                    | 78.2 | 60.1  | 87.5  | 71.5 | 60.0   | 57.4   | 83.3   | 71.2 |\n| $\\rightarrow$ Qwen3 Tok. via SFT | 81.4 | 56.7  | 83.7  | 67.2 | 46.2   | 43.5   | 73.4   | 64.6 |\n| $\\rightarrow$ Qwen3 Tok. via ALM | 78.8 | 59.9  | 86.9  | 70.7 | 57.6   | 53.6   | 76.7   | 69.2 |\n\nThe hyperparameters exactly match those of the subword $\\rightarrow$ subword tokenizer transfer experiments in the paper, and the loss continuously decreases throughout training (no loss spikes) for both methods. Notably, while performance on some benchmarks increases throughout training for the SFT baseline (ARC-C, PiQA, IFEval), performance on others stagnates or even decreases (BoolQ, MMLU, AGI-*). From these results, we can conclude that ALM is superior to SFT for model scales larger than those originally considered in our paper.\n\n> Considering the small models and tokenizers, and the sensitiveness to the hyper-parameters, the performance improvement is slight.\n\nWe beg to differ as, in our view, our experiments show the opposite: across all hyperparameter settings of ALM for the learning rate, temperature, and distance function, our method outperforms the baselines across all baseline hyperparameters (see Tables 5 and 6).\n**The performance improvements close the gap toward the teacher by an additional 34% over the baselines in Use Case 1 (Figure 2), improving by 3% absolute percentage points on GSM8K in Use Case 2 (Table 3), and improving over SFT by up to 6.5% absolute percentage points in Use Case 3 (Table 4)**; we consider these margins to be quite significant. \n\nAnd to answer your questions.\n\n> How does ALM behave when the teacher’s vocabulary is much larger than the student’s?\n\nThis is an interesting question! The extreme case of the teacher’s vocabulary being much larger than the student’s occurs in our experiments on transfer to bytes, where the student’s vocabulary has a size of just 256 tokens, whereas the teacher’s vocabulary has ~100k-200k tokens. We found that, **especially in such cases, ALM performs well while prior methods break down** (see Table 1 and the discussion of Use Case 1).\n\n> From the ablation study, it looks like the performance is very sensitive to some setups, like the learning rate and the selection heuristic.\n> How do we determine the best setup when we are performing the proposed ALM?\n\n**We use a single setting for ALM that works well across all experiments** (temperature=100, distance function=KL, threshold=0.1, LR=1e-5) as detailed in Appendix B. The only exception is transfer to bytes, where we make a couple of adjustments detailed in Appendix C. So, when using ALM, we recommend that practitioners use the hyperparameters we use across all our experiments. The ablations & sensitivity analyses are only intended to provide intuition as to the effect of varying these hyperparameters.\n\n> Authors claimed that the proposed method enables effective distillation across fundamentally different tokenizers. How do we measure the distance between tokenizers? It would be good to see the trend of using the proposed method on tokenizers with increasing distances.\n\nHow to measure the distance between tokenizers is an interesting question and one which does not have a trivial answer. A simple proxy is the vocabulary size, but it might not always be accurate (and would easily break in the setting of duplicating the tokenizer and adding N unused/barely used tokens). We will clarify this in the paper and would hope to see future work on quantifying tokenizer distance.\n**Our claim to enable distillation between fundamentally different tokenizers is based on the fact that ALM allows for distilling between subword tokenizers and byte tokenization which are the two extremes of the spectrum contemporary tokenizers lie on**.  We agree that quantifying the trend of cross-tokenizer distillation method performance across increasingly distant tokenizers would be a very interesting experiment for follow-up work, thanks for the suggestion. **The reason we did not explore this setting is that we constrained our experiments to subword tokenizers which are used in practice by modern LLMs** (these are largely in the ~100k-200k vocabulary range).\n\nWe hope this addresses your concerns, and please let us know if you have any further questions."}, {"review_id": "Q7hWNKLEmr", "reviewer": "Reviewer_G5Di", "summary": "The paper tackles the problem of current distillation methods that require similar tokenizers between teacher and student LLMs. The authors propose a novel cross-tokenizer distillation method ALM, enabling effective knowledge transfer even when the teacher and student use different tokenizers. The approach is evaluated on three distinct use cases, consistently outperforming existing methods.", "questions": "1. Does the teacher and student models have to have similar tokenizers? What if the teacher and student tokenizers have very small overlappings?\n2. Is it possible that for some tokens, they are not covered by chunks from the teacher side, and therefore students could not learn anything on them?\n3. What if the text is very long and the greedy alignments become sub optimal?", "limitations": "The computational complexity of the proposed method could be pretty high. The greedy alignments could be sub optimal when text is long.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The problem studied in the paper is important and critical. The differences between tokenizers are significant blocker between knowledge transferring between models. Having a method to overcome such limitations could directly benefit many applications, such as distilling general, large models for domain-specific use smaller models. \n2. The proposed solution ALM, which aims to model the token level matching probability via chunk likelihoods, makes sense to me. It is a sound methodology.\n3. The paper is easy to follow, especially for readers who have never touched the problem of cross-tokenizer distillation. The authors give sufficient information in the preliminaries and background section.\n\nWeaknesses\n1. From the paper, the complexity of computing loss function on chunk-level probabilities is pretty high. So I am not sure if it is practical to actually deploy the proposed ALM into production use cases. \n2. The first eq in section 3 (Alignment Indices) is not very clear. The authors use a greedy approach to get token chunks. However, when the text is long, there can be multiple valid ways to segment and align text between two different tokenizers. Therefore, greedy strategy might not find the optimal segmentation.\n3. The paper also didn't mention how the student could learn on tokens not covered by chunks from teacher model.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for very much for your review! We believe your concerns mostly stem from a couple of misunderstandings which we hope to clarify here (and will also clarify in the paper). \n\n> From the paper, the complexity of computing loss function on chunk-level probabilities is pretty high. So I am not sure if it is practical to actually deploy the proposed ALM into production use cases. \n\nComplexity (computational and implementation-wise) indeed plays an important role. **Chunk alignments can be computed in O(n) via a simple dynamic programming algorithm** and represented as an $b \\times m \\times k$ and $b \\times n \\times k$ matrix (where $m$ and $n$ are the teacher and student sequence lengths, respectively, $k$ is the maximum amount of chunks, i.e. $\\min(m,n)$ and $b$ is the batch size) with entries 1 if the i-th token is part of the j-th chunk, and zero otherwise (an example is shown below).\n\nExample Matrix 1:\n\n| The | distill | ation | procedure |\n|-----|---------|-------|-----------|\n| 1   |         |       |           |\n|     | 1       | 1     |           |\n|     |         |       | 1         |\n\nExample Matrix 2:\n\n| The | dist | illation | proc | edure |\n|-----|------|----------|------|-------|\n| 1   |      |          |      |       |\n|     | 1    | 1        |      |       |\n|     |      |          | 1    | 1     |\n\nThese matrices can be precomputed in the dataloader. Afterwards, ALM is easy to implement:\n\nCompute the teacher and student log-probabilities ($b \\times m \\times |V_1|$ and $b \\times n \\times |V_2|$).\nSelect the teacher and student log-probabilities along the true next tokens in the data ($b \\times m$ and $b \\times n$, respectively).\nMatrix-multiply the log-probabilities with Matrix 1 and Matrix 2 to compute chunk-level log-probabilities ($b \\times k$ and $b \\times k$).\nCompute the loss as the mean of some distance function between the chunk-level probabilities (for example, the mean of the absolute differences between the chunk-level probabilities).\n\n**All of this is implementable in a couple of lines of code in PyTorch (and will be open-sourced along with the rest of our code)**. We will add an Appendix to the paper to discuss these implementation considerations.\n\n> The first eq in section 3 (Alignment Indices) is not very clear. The authors use a greedy approach to get token chunks. However, when the text is long, there can be multiple valid ways to segment and align text between two different tokenizers. Therefore, greedy strategy might not find the optimal segmentation.\n\nThere are indeed many valid ways to align text between two different tokenizers (exponentially many in the sequence length). However, **there is only one greedy segmentation, the one with the most (and thus shortest possible) chunks**. Our intuition behind selecting this segmentation is that it provides the most signal: for any non-greedy segmentation, we could break down a chunk into two or more chunks which then convey more information about the teacher's predictions of the tokens in this chunk (e.g. if “The distillation” was a single chunk in the example above). It is however a valid empirical question whether non-greedy and/or partially overlapping segmentations could improve ALM; this could be an area for future work. We will clarify this in the paper.\n\n> The paper also didn't mention how the student could learn on tokens not covered by chunks from teacher model.\n\n**The student does not learn from next-token predictions which are not part of any chunk** (i.e. not along the ‘main path’, the true next token logit in the data). As discussed in Section 3.2, this does decrease the density of the signal, which however in practice does not have a large effect on our method's effectiveness and can be alleviated by adding Hidden State Alignment if desired (see Table 7).\n\n> Does the teacher and student models have to have similar tokenizers? What if the teacher and student tokenizers have very small overlappings?\n\nThe Qwen and Llama tokenizers have substantial overlap (intersection over union of tokens in the vocabulary = 64%), the Gemma and Qwen tokenizers have lower overlap (intersection over union of tokens in the vocabulary = 29%). In the extreme case of transfer to bytes (Use Case 1), the overlap is ~0% since the student tokenizer only contains the 256 tokens corresponding to the possible values of a byte. **ALM can effectively cope with this case of essentially zero overlap** (see Table 1).\n\n> Is it possible that for some tokens, they are not covered by chunks from the teacher side, and therefore students could not learn anything on them?\n\nEvery token in both the student’s and teacher’s token sequence is part of a chunk, so the student will learn from all tokens which appear in the input text and it is not possible for chunks not to be covered from the teacher's side (assuming both tokenizers can segment arbitrary text, which is true for virtually all modern tokenizers and can otherwise be made true; see Footnote 1).\n\n> What if the text is very long and the greedy alignments become sub optimal?\n\nAs discussed above, **greedy alignments will not become suboptimal for long texts**. Greedy alignments are always the alignments which result in the chunks with the highest granularity / the lowest average token count per chunk and thus the maximal ability to transmit signal from the teachers’ predictions.\n\nWe would kindly ask you to update your score if your concerns have been addressed, and please let us know if you have any further questions."}, {"review_id": "Tgy1LwWS39", "reviewer": "Reviewer_sqsY", "summary": "This paper introduces Approximate Likelihood Matching (ALM), a novel cross-tokenizer distillation method designed to transfer knowledge between large language models (LLMs) with fundamentally different tokenizers. ALM enables effective distillation across diverse tokenizer types, such as from subword to byte-level tokenizers, by minimizing the differences in chunk likelihoods between the teacher and student models. The method incorporates optional debiasing techniques to mitigate tokenization bias and aligns hidden states to enrich the teacher signal. The authors validate ALM's efficacy through three distinct use cases: tokenizer transfer via self-distillation, large-to-small model distillation, and zero-shot tokenizer transfer hypernetwork training. The results demonstrate ALM's ability to outperform prior methods, significantly expanding the range of teacher-student pairs for distillation and enhancing the adaptability and transferability of LLMs.", "questions": "The questions I expect to ask would be similar to the above section.", "limitations": "The authors have discussed some limitations of their work, such as the computational constraints that limited their experiments to models ranging from 2.4B to 8B parameters and a relatively small number of training tokens (≈0.6B). They also noted that their method has only been tested in the supervised KD setup and its application to more intricate distillation techniques remains future work. However, there are a few additional points that could be further addressed:\n1. Scalability and Larger Models: The experiments were limited to models with up to 8B parameters. The paper does not provide sufficient analysis on how ALM would perform with larger models or more extensive training data, which limits the understanding of its generalizability to more complex and larger-scale applications.\n2. Robustness to Tokenization Bias: While ALM handles tokenizer mismatch effectively, its resilience to residual tokenization bias—especially in low-resource or non-Latin languages—remains underexplored. Evaluating performance under more diverse tokenization schemes would strengthen the generality of the method.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1. Innovative Distillation Method: The paper introduces ALM, a novel cross-tokenizer distillation method that enables effective knowledge transfer between LLMs with different tokenizers.\n2. Empirical Validation: The authors provide comprehensive experiments across three distinct use cases, demonstrating ALM's effectiveness and superiority over existing methods.\n3. Technical Clarity: The methodology is clearly explained, making it accessible and reproducible for other researchers.\n\nWeaknesses:\n1. Limited Scalability: The experiments are restricted to models with up to 8B parameters, limiting the understanding of ALM's performance on larger models.\n2. Comparative Analysis: The paper lacks detailed comparisons with other state-of-the-art methods focusing on similar domain sampling, which could better illustrate ALM's novelty and effectiveness.\n3. Broader Contextualization: The paper does not fully contextualize ALM within the broader landscape of language model distillation, including its relation to other emerging trends and challenges.", "was_revised": true, "final_justification": "Considering the effectiveness and novelty of the method proposed in this article and the opinions of other reviewers, I give the final rating of borderline acceptance.", "rebuttal": "Thank you for your review! We address your concerns below.\n\n> The experiments are restricted to models with up to 8B parameters, limiting the understanding of ALM's performance on larger models.\n\nWe favored breadth for our experiments (thorough ablations, sensitivity analysis, etc.) over scaling to large parameter counts. However, we do recognize the importance of showing that our method scales to larger LLMs. To do so, **we have conducted additional experiments transferring a larger LLM, Gemma3-12B, to the Qwen3 tokenizer**. The results are shown below.\n\n|                                  | PiQA | ARC-C | BoolQ | MMLU | AGI-EN | AGI-ZH | IFEval | Avg. |\n|----------------------------------|------|-------|-------|------|--------|--------|--------|------|\n| Gemma3-12B-Instruct                    | 78.2 | 60.1  | 87.5  | 71.5 | 60.0   | 57.4   | 83.3   | 71.2 |\n| $\\rightarrow$ Qwen3 Tok. via SFT | 81.4 | 56.7  | 83.7  | 67.2 | 46.2   | 43.5   | 73.4   | 64.6 |\n| $\\rightarrow$ Qwen3 Tok. via ALM | 78.8 | 59.9  | 86.9  | 70.7 | 57.6   | 53.6   | 76.7   | 69.2 |\n\nThe hyperparameters exactly match those of the subword $\\rightarrow$ subword tokenizer transfer experiments in the paper, and the loss continuously decreases throughout training (no loss spikes) for both methods. Notably, while performance on some benchmarks increases throughout training for the SFT baseline (ARC-C, PiQA, IFEval), performance on others stagnates or even decreases (BoolQ, MMLU, AGI-*). From these results, we can conclude that ALM is superior to SFT for model scales larger than those originally considered in our paper.\n\n> The paper lacks detailed comparisons with other state-of-the-art methods focusing on similar domain sampling, which could better illustrate ALM's novelty and effectiveness.\n\nCould you give an example of one of the ‘similar domain sampling’ methods you are referring to? Sampling the distillation training data in a way that closely resembles the LLM pretraining data is generally hard since the LLM training data mix is unknown. Furthermore, **ALM is orthogonal to the way the data is sampled; we would expect improvements to the data sampling technique to synergize with the improvements from ALM**.\n\n> The paper does not fully contextualize ALM within the broader landscape of language model distillation, including its relation to other emerging trends and challenges.\n\n**We contextualize ALM within the broader landscape of distillation in Section 2.3, and provide a detailed comparison with prior cross-tokenizer distillation methods in Appendix G**. An exhaustive overview over the ‘emerging trends and challenges’ in language model distillation (besides cross-tokenizer distillation) would have gone outside the scope of our paper. We will add a reference to the survey by Xu et al. [[1]] to Section 2.3 as an additional source of information about the broader language model distillation landscape.\n\n> Robustness to Tokenization Bias: While ALM handles tokenizer mismatch effectively, its resilience to residual tokenization bias—especially in low-resource or non-Latin languages—remains underexplored. Evaluating performance under more diverse tokenization schemes would strengthen the generality of the method.\n\n**Our benchmark results on Chinese (e.g. in Table 2) indicate that ALM can effectively preserve performance in non-Latin scripts (and even in languages without clear word boundaries)**. We believe these results sufficiently support that ALM is not restricted to the Latin script.\n\nWe hope this addresses your concerns, and please let us know if you have any further questions.\n\n[1]: https://arxiv.org/abs/2402.13116"}, {"review_id": "CSjTfZ15fX", "reviewer": "Reviewer_bnAG", "summary": "This paper focuses on a scenario where tokenizers for teacher and student LLMs are different in distillation. To address this challenging scenario, this paper proposes a new approach called approximate likelihood matching (ALM). ALM primarily tries to minimise the divergence between teacher and student at a chunk level, given a sequence. This effectively allows the student's likelihood to be close to the teacher's likelihood, and this chunk-based approach apparently enables cross-tokenizer distillation, which has not been solved in the previous literature. The experiments are divided into three parts: (i) a self-distillation scenario; (ii) large-to-small domain-specialised model distillation; and (iii) training-free tokenizer adaptation. The results show the effectiveness of the proposed ALM approach for each scenario.", "questions": "**Suggested changes and questions**  \n1. I think the description of chunk debiasing and a hidden state alignment objective is quite dense. I would suggest inserting a conceptual figure for better understanding.  \n2. On Weakness 1, I personally would like to see the results of self-distillation and ensembling experiments on a larger scale (> 10B) if resources are available. This is quite useful in that it can help measure the robustness of the proposed approach against models with far better task performances. I sometimes see approaches, which successfully minimize performance degradation in a small-scale experiment but see a larger degradation in large-scale (potentially because larger models might be more sensitive to model modifications.)  \n3. On Weakness 2, it would be nice to have results on non-math-related tasks.  \n4. It seems to me that while ALM and ZeTT are from different paradigms, both can indeed replace the tokenizer of a source model. Could the paper include a discussion on this point? For instance, I assume that training a hypernetwork is costly, so using self-distillation with ALM as in the first use case is a better option in terms of both computational costs and task performance. Is this truly the case?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. This paper proposes a novel approach to allow distillation between different models with different tokenizers using chunk-based approximate log-likelihood matching.  \n2. The experiments are quite extensive and showcase the effectiveness of the proposed approach in three ways: self-distillation/ensembling, strong-to-weak domain-specialised model distillation, and zero-shot tokenizer transfer using a hypernetwork. This cross-tokenizer distillation must be quite a challenging scenario, and thus, observing a good performance across different scenarios suggests further potential of the proposed approach.  \n\n**Weaknesses**\n1. While I acknowledge the effectiveness of the proposed approach, I think the first use case, except for ensembling, is a bit of a waste of space. Specifically, I do not see any practical value of distilling a subword-level tokenizer to a byte-level tokenizer in the first use case, as the latter neither offers a task performance advantage nor better tokenization efficiency. If this is something important, the paper warrants further clarification on this experiment.  \n2. While the large-to-small domain-specialised model distillation in the second use case is fascinating, the paper omits reporting general performance evaluation (e.g. evaluation on MMLU, IFEval, ARC, BoolQ, etc.). This evaluation must be essential to provide a more holistic assessment of the proposed method.", "was_revised": true, "final_justification": "* The authors address the first weakness with sufficient evidence (as long as the final version incorporates this discussion).\n* While the second weakness has not been addressed, this point is a minor weakness, which should not affect the overall rating much.\n* The additional experiment on the model with 10B or larger even seems fascinating and exciting.", "rebuttal": "Thanks a lot for your review! We answer your questions below.\n\n> I do not see any practical value of distilling a subword-level tokenizer to a byte-level tokenizer in the first use case, as the latter neither offers a task performance advantage nor better tokenization efficiency. If this is something important, the paper warrants further clarification on this experiment. \n\nThe experiments on transfer to bytes are important for two reasons: (1) they allow us to show that ALM works well in the extreme case of effectively zero vocabulary overlap between the teacher and the student, and does thus not implicitly rely on large vocabulary overlap / similar source and target tokenizers. Even more importantly, (2) recent work such as BLT [[1]] and H-Net [[2]] pushes byte-level language modelling toward new levels of efficiency, demonstrating that they can achieve better scaling behavior than subword tokenization while avoiding the drawbacks of subword tokenization (such as the vocabulary bottleneck [[3]], tokenization bias [[4]] and imperfect character-level understanding [[5]]). By introducing an effective strategy for subword-to-byte distillation, our work could facilitate the development of byte-level models from pre-existing LLMs and speed up the experimentation with these new architectures and their adoption. We will clarify both these points in the paper.\n\n> While the large-to-small domain-specialised model distillation in the second use case is fascinating, the paper omits reporting general performance evaluation (e.g. evaluation on MMLU, IFEval, ARC, BoolQ, etc.). It would be nice to have results on non-math-related tasks. \n\nSince our goal for the experiments in Use Case 2 was specifically to create a math-specialized model, we believe the performance on math tasks is by far the most important. However, we do agree that evaluation on a larger set of benchmarks would be beneficial, and we will add this to the camera-ready (our resources were occupied by transferring a larger model via ALM so we were not able to run this during the rebuttal period).\n\n> I personally would like to see the results of self-distillation and ensembling experiments on a larger scale (> 10B) if resources are available.\n\nWe agree on the importance of scaling up, and we ran self-distillation experiments on transferring Gemma3 12B to the Qwen3 tokenizer to address this point. The results are shown below.\n\n|                                  | PiQA | ARC-C | BoolQ | MMLU | AGI-EN | AGI-ZH | IFEval | Avg. |\n|----------------------------------|------|-------|-------|------|--------|--------|--------|------|\n| Gemma3-12B-Instruct                    | 78.2 | 60.1  | 87.5  | 71.5 | 60.0   | 57.4   | 83.3   | 71.2 |\n| $\\rightarrow$ Qwen3 Tok. via SFT | 81.4 | 56.7  | 83.7  | 67.2 | 46.2   | 43.5   | 73.4   | 64.6 |\n| $\\rightarrow$ Qwen3 Tok. via ALM | 78.8 | 59.9  | 86.9  | 70.7 | 57.6   | 53.6   | 76.7   | 69.2 |\n\nThe hyperparameters exactly match those of the subword $\\rightarrow$ subword tokenizer transfer experiments in the paper, and the loss continuously decreases throughout training (no loss spikes) for both methods. Notably, while performance on some benchmarks increases throughout training for the SFT baseline (ARC-C, PiQA, IFEval), performance on others stagnates or even decreases (BoolQ, MMLU, AGI-*). From these results, we can conclude that ALM is superior to SFT for model scales larger than those originally considered in our paper.\n\n> I think the description of chunk debiasing and a hidden state alignment objective is quite dense. I would suggest inserting a conceptual figure for better understanding. \n\nThank you for raising this point. We agree that a figure should ease understanding and will add one to the final version of the paper.\n\n> It seems to me that while ALM and ZeTT are from different paradigms, both can indeed replace the tokenizer of a source model. Could the paper include a discussion on this point? For instance, I assume that training a hypernetwork is costly, so using self-distillation with ALM as in the first use case is a better option in terms of both computational costs and task performance. Is this truly the case?\n\nThis is another excellent point. It is correct that if the goal is transferring to a single target tokenizer, direct tokenizer transfer self-distillation via ALM should be the most efficient. However, if the goal is to potentially transfer to up to N different tokenizers, there is some threshold of N where we expect it would become more efficient to train a ZeTT hypernetwork once, then have a brief additional training phase per tokenizer, instead of separately transferring to the N tokenizers individually via ALM self-distillation. We will add this intuition to the section on Use Case 3 in the paper.\n\nPlease let us know if you have any additional questions!\n\n[1]: https://arxiv.org/abs/2412.09871\n[2]: https://arxiv.org/abs/2507.07955\n[3]: https://arxiv.org/abs/2301.10472\n[4]: https://arxiv.org/abs/2506.14123\n[5]: https://arxiv.org/abs/2409.15452"}]

Author rebuttal:

Meta-review: This work develop a principled cross-tokenizer distillation method to solve this crucial deficiency in current distillation method, which typical requires similar tokenizer. All reviewers have positive ratings to this paper, and acknowledge the strengths, such as the problem definition is clear, the problem studied in the paper is important and critical, the paper is easy to follow, and comprehensive experiments across three distinct use cases demonstrating the effectiveness of the method. Although some concerns have been raised by reviewers, such as the experimental model is small scale, the sensitivity of the hyperparameters, concerns on greedy alignment on long sequence, the author address these problems to some extent. Overall, it is a nice paper, and we recommend an acceptance to this paper.
Decision: Accept (poster)