Enhancing Cross-Tokenizer Knowledge Distillation with Contextual Dynamical Mapping
Authors: Yijie Chen, Yijin Liu, Fandong Meng, Yufeng Chen, Jinan Xu, Jie Zhou
Accepted at: acl25

Enhancing Cross-Tokenizer Knowledge Distillation with Contextual Dynamical Mapping

Yijie Chen1,2*, Yijin Liu3, Fandong Meng3, Yufeng Chen1,2, Jinan Xu1,2†, Jie Zhou3
1 Key Laboratory of Big Data & Artificial Intelligence in Transportation (Beijing Jiaotong University), Ministry of Education
2 School of Computer Science and Technology, Beijing Jiaotong University, Beijing, China
3Pattern Recognition Center, WeChat AI, Tencent Inc, China
{22120354, chenyf, jaxu}@bjtu.edu.cn
{yijinliu, fandongmeng, withtomzhou}@tencent.com

Abstract

Knowledge Distillation (KD) has emerged as a prominent technique for model compression. However, conventional KD approaches primarily focus on homogeneous architectures with identical tokenizers, constraining their applicability in cross-architecture scenarios. As for the cross-tokenizer KD, the differences in the tokenizers give rise to two fundamental challenges: (1) sequence misalignment caused by divergent tokenization strategies, and (2) mismatched vocabulary size and composition. While existing probability-matching methods attempt to address these issues, their efficacy remains limited due to suboptimal alignment in both the sequence and vocabulary aspects. To overcome these limitations, we propose Contextual Dynamic Mapping (CDM), a novel cross-tokenizer distillation framework that employs contextual information to enhance sequence alignment precision and dynamically improves vocabulary mapping. We evaluated the effectiveness of our approach across five advanced and widely-used model families (i.e., LLama3, Phi3, Gemma2, OPT and Qwen2), which were configured into three distinct teacher-student pairs. Our method shows significant advantages over existing cross-tokenizer distillation baselines across diverse benchmarks, including instruction-following, code generation and math. Notably, our analysis reveals that combining conventional same-tokenizer distillation and cross-tokenizer distillation through CDM yields further performance improvements.1

1 Introduction

Knowledge distillation (KD) (Hinton, 2015; Wen et al., 2023; Gu et al., 2024; Ko et al., 2024; Guo et al., 2025) has emerged as a promising methodological framework for enhancing the performance of compact models through knowledge transfer from larger, more powerful teacher models. Nevertheless, conventional KD approaches, which aim to minimize the distribution difference between the logits of teachers and students, are still restricted by the requirement for tokenizer consistency between teachers and students. To address this limitation, Cross-Token KD (CTKD) (Fu et al., 2023; Boizard et al., 2024) has emerged as a critical research frontier. As illustrated in Figure 1, the core challenges of CTKD arise from two fundamental aspects: (1) divergent tokenization strategies induce sequence misalignment during text processing, and (2) vocabulary discrepancies create dimensional and semantic mismatches in output probability spaces. Both the sequence and vocabulary level misalignments create significant barriers to effective knowledge transfer between heteroge-

![First, the token sequences of teacher and student models are aligned to spans consisting of similar text.](page_682_370_393_120.png)

(a) First, the token sequences of teacher and student models are aligned to spans consisting of similar text.

![The vocabulary distribution from the teacher and student models should be aligned via token mapping.](page_682_520_393_120.png)

(b) The vocabulary distribution from the teacher and student models should be aligned via token mapping.

Figure 1: The illustration of the alignment process of cross-tokenizer knowledge distillation. A and B mean the tokenizers of the student or teacher models.

* Work done when Yijie was interning at Pattern Recognition Center, WeChat AI, Tencent Inc, China.
† Jinan Xu is the corresponding author.
1The code is available at https://github.com/pppa2019/ContextualDynamicMapping
neous architectures. Current approaches attempt to bridge these gaps through two primary strategies:

• Tokens mapping based on text character similarity (Fu et al., 2023; Wan et al., 2024a,b), which risks semantic misalignment (e.g., confusing "denoted" and "devoted")

• Optimal transport methods (Boizard et al., 2024; Cui et al., 2024) that compute full distribution distances but lack explicit semantic alignment.

To quantify the discrepancies in sentence tokenization and vocabulary among mainstream large language models (LLMs) equipped with different tokenizers, we conducted a comprehensive analysis of alignment rates\(^2\) across five mainstream LLMs. The results demonstrate the wide range of sequence alignment rates (around 30%–90%) and vocabulary alignment rates (around 10%–90%), which collectively highlight the substantial room for improvement in cross-tokenizer knowledge transfer. To further enhance the accuracy of model output alignment, we introduce Contextual Dynamic Mapping (CDM), a novel CTKD framework that introduces:

• **Sequence-level**: Entropy-weighted dynamic programming, which improves the precision of sequential token alignment by dynamically adjusting the alignment process utilizing entropy-based measure of tokens.

• **Vocabulary-level**: Context-aware candidate matching dynamically constructs semantic-preserving token mappings, achieving an improved balance between token-level precision and minimization of unmapped token pairs.

Building on this mapping framework, we further prioritize Top-K tokens based on their contextual significance, which effectively suppresses noise while enhancing reasonable token mapping rates further.

The experiments are based on five encompassing open-source model series, and the training and evaluation contain various tasks, including instruction following, code generation, and math. The experimental results demonstrate that CDM exhibits consistent and substantial superiority over current mainstream cross-tokenizer distillation approaches among different tasks (*e.g.*, for Qwen2-0.5B model, CDM has the improvement of 4.27% on instruction

\(^2\)The detailed statistical process is shown at Section 2

![Table showing sequence matching rate between different tokenizers](page_1016_180_370_246.png)

Figure 2: Matching rate of sequence alignment results.

![Table showing vocabulary matching rate between different tokenizers](page_1016_420_370_246.png)

Figure 3: Matching rate of vocabulary alignment results.

following tasks, 12.19% on code tasks and 3.34% on math tasks) and even surpasses the performance of same-tokenizer distillation in some settings. Our main contributions are as follows:

• We propose the CDM method, which facilitates cross-tokenizer distillation through contextual information to improve the matching accuracy in sequence and vocabulary aspects.

• Extensive experiments are conducted on various backbone models and datasets across three tasks, and the experimental results indicate the effectiveness of CDM consistently.

• We provide a detailed analysis of the alignment rate improvement in CDM to illustrate the correlation between alignment and distillation effectiveness.

• We observe that the model performance can be further improved by combining the conventional same-tokenizer distillation and CDM.

2 Preliminary Analysis

To systematically investigate the fundamental challenges in CTKD, we analyze the alignment rates in the sequence and vocabulary levels separately. On
Figure 4: The architecture of CDM consists of two key components: an entropy-weighted Dynamic Time Warping (DTW) sequence alignment algorithm and a dynamic Top-K vocabulary mapping algorithm. Following the mapping procedure, the output representations from both the teacher and student models are aligned to ensure consistency in both dimensional structure and semantic space.

the sequence level, we calculate the tokenization result of the two selected tokenizers. Concretely, for a certain sentence, we statistically analyze the set of tokens overlap and average on the sample number as final results. The statistic is based on 3000 examples sampled from the training corpus of Dolly-15K (Ouyang et al., 2022) On the vocabulary level, we calculate the ratio of tokens that exactly matches the two selected tokenizers.

The statistical results presented in Figure 2 and Figure 3 highlight two critical limitations: (1) vocabulary-level alignment rates, which reach as low as 9%, and (2) sequence-level alignment rates, with minimum values as low as 31%. These findings underscore the critical need for supplementary mapping mechanisms within existing methodologies. Furthermore, the intersection of these two mismatch dimensions creates compounding effects, namely, sequence alignment errors amplify vocabulary mismatches through error propagation. This issue motivates our proposed contextual dynamic mapping framework, which mitigates alignment errors by utilizing contextual information.

3 Contextual Dynamic Mapping

3.1 Formal Definition
In this section, we take subscripts \( stu \) and \( tea \) to denote the student and teacher models, respectively. The key symbols used in the algorithm are defined in Table 1. Given an input sentence, the tokenization process yields token sequences \( T_{stu} \) and \( T_{tea} \), along with the vocabulary size \( |V_{stu}| \) and \( |V_{tea}| \) (\( |V_{stu}| \neq |V_{tea}| \) and \( |T_{stu}| \neq |T_{tea}| \)).

The model output logits are then expressed as \( O_{stu} \in \mathbb{R}^{|V_{stu}| \times |T_{stu}|} \) and \( O_{tea} \in \mathbb{R}^{|V_{tea}| \times |T_{tea}|} \).

<table>
  <tr>
    <th>Symbol</th>
    <th>Description</th>
  </tr>
  <tr>
    <td>\( T_{stu}, T_{tea} \)</td>
    <td>Token sequences from the student and teacher models</td>
  </tr>
  <tr>
    <td>\( D_{DTW} \)</td>
    <td>Minimized alignment cost in DTW</td>
  </tr>
  <tr>
    <td>\( EditDistance(\cdot) \)</td>
    <td>Edit distance between two tokens</td>
  </tr>
  <tr>
    <td>\( \Pi \)</td>
    <td>Set of all possible alignment paths</td>
  </tr>
  <tr>
    <td>\( H(O) \)</td>
    <td>Position-wise entropy vector</td>
  </tr>
  <tr>
    <td>\( \phi(X) \)</td>
    <td>MinMax normalization function</td>
  </tr>
  <tr>
    <td>\( W(O) \)</td>
    <td>Alignment weight vector</td>
  </tr>
  <tr>
    <td>\( C \)</td>
    <td>Constant controlling the weight range</td>
  </tr>
</table>

Table 1: Symbol definitions for CDM

3.2 Sequence Alignment with Contextual Weight

Fu et al. (2023) introduced a method based on Dynamic Time Warping (DTW) for sequence token alignment, later adopted in (Wan et al., 2024a,b). The DTW is a dynamic programming method that can minimize the cost \( D_{DTW} \) between the two token lists for the same sentence tokenized by different tokenizers. The algorithms minimize:

\[
D_{DTW}(T_{stu}, T_{tea}) = \min_{\pi \in \Pi} \sum_{(i,j) \in \pi} cost(t_i^{stu}, t_j^{tea})
\]

In existing works (Fu et al., 2023; Wan et al., 2024b), the cost function of DTW \( cost(\cdot) \) use edit distance \( EditDistance(t_i^{stu}, t_j^{tea}) \). However, this approach demonstrates suboptimal performance with occasionally generating non-sensical align-
ments, for the edit distance metric introducing misalignment.

To overcome these constraints, we propose an entropy-enhanced DTW to incorporate the context information into the alignment process. The entropy-weight prioritizes the more informative tokens while suppressing noisy alignments, mitigating the sequence token mismatch problem. To illustrate, consider the example in Figure 1. For the original DTW, the edit distance between “e” (from A, denote with italics) and “apple” (from B) is larger than the edit distance between “e” and “a”(from B), which leads to the mistaken matching between the “e” and “a”. However, the entropy in “a” and “e” both have higher entropy compared with “apple” for their ambiguity, so after adding entropy-based weight the distance metric between “e” and “apple” will be lower, leading to a correct mapping (“appl”, “apple”) and (“e”, “apple”).

For a tokenizer with vocabulary size \( V \), let \( m \) denote any model in which both can be a teacher or student, \( O_m \in \mathbb{R}^{[T_m] \times |V_m|} \) represent the output logits where \( |T_m| \) denotes the length of the token sequence and \( o_i \in R^{|V_m|} \) presents the logits vector of the i-th vector in \( O_m \). We first compute position-wise entropy \( H \in \mathbb{R}^{[T_m]} \) through:

\[
H(o_i) = - \sum_{j=1}^{|V_m|} p(o_i^j) \log p(o_i^j)
\]

Subsequently, we apply the MinMax as the normalization function \( \phi(X) = \frac{x - min(x)}{max(x) - min(x)} \) and conduct a linear mapping to obtain alignment weights.

\[
W(O) = [\text{Sigmoid}(\phi(H(O))) \cdot C + C']
\]

The hyperparameter \( C \) controls the weighting range \([C, 2C]\) to ensure both flexibility and computational efficiency in the weighting process. After calculate the weight as Equation 3, let \( cost(t_i^{stu}, t_j^{tea}) = w_i^{stu} \cdot w_j^{tea} \). \( EditDistance(t_i^{stu}, t_j^{tea}) \) be the cost function of DTW. With weighted DTW, we obtain span-level token mapping lists \( T_{stu}^{seq} \) and \( T_{tea}^{seq} \). The original logits \( O_{tea} \) and \( O_{stu} \) are merged according to these mapping lists using mean pooling. After that, the outputs \( O_{stu}^{seq} \in \mathbb{R}^{[T_{align}] \times |V_{stu}|} \) and \( O_{tea}^{seq} \in \mathbb{R}^{[T_{align}] \times |V_{tea}|} \) are obtained and have been aligned in the sequence-level.

Algorithm 1 Algorithm of token alignment

Input: \( O_{stu}^{seq}, O_{tea}^{seq}, F_{EM}, \theta \)
Output: \( O_{stu}^{align}, O_{tea}^{align} \)

1: initialize \( F_{dynamic} := F_{EM} \)
2: gets the tokens \( T_{stu}^{topk}, T_{tea}^{topk} \) with Top-K logits in each position utilizing.
3: for each position \( i \in [1, |T_{align}|] \) do
4:   for each \( tok_a \in T_{stu}^{topk}[i] \) do
5:     if \( tok_a \notin F_{EM} \) then
6:       best = \( \emptyset \); \( min\_dist = \infty \)
7:       for each \( tok_b \in T_{tea}^{topk}[i] \) do
8:         \( d = dist\_func(tok_a, tok_b) \)
9:         if \( d < \theta \) and \( d < min\_dist \) then
10:           best = \( tok_a, min\_dist = d \)
11:         end if
12:       end for
13:       if \( best \neq \emptyset \) then
14:         \( F_{dynamic} := F_{dynamic} \cup \{tok_b \rightarrow best\} \)
15:       end if
16:     end if
17:   end for
18: end for
19: \( O_{stu}^{align} = \text{Mask}(F_{dynamic}(O_{stu}^{topk})) \)
20: \( O_{tea}^{align} = \text{Mask}(O_{tea}^{topk}) \)

Return: \( O_{stu}^{align}, O_{tea}^{align} \)

3.3 Dynamic Vocabulary Mapping with Contextual Candidates

After the sequence alignment, the model logits are still not aligned on the vocabulary dimension, i.e., \( |V_{stu}| \neq |V_{tea}| \). CDM uses contextual information to improve the vocabulary mapping accuracy, and the core process and is described as Algorithm 1. First, tokens that can be exactly matched between the two tokenizers are stored in a mapping dictionary \( F_{EM} \) to facilitate efficient mapping operations. For tokens that remain unmapped, we introduce a dynamic mapping dictionary \( F_{dynamic} \), which is initialized as a copy of \( F_{EM} \) (line 1). The process involves utilizing the aligned model logits \( O_{tea}^{seq} \) and \( O_{stu}^{seq} \). In order to preserve the most relevant information in the context for the training instance and avoid mapping between irrelevant tokens, we select the top k logits at each position according to Equation 4.

\[
\text{Top-}K(o_i) = \text{argsort}\left( \sum_{j=1}^{|V|} o_i^j, \downarrow \right) [:k]
\]
Then, Top-K(\(O_{stu}^{seq}\)), Top-K(\(O_{tea}^{seq}\)) yields both the corresponding logits value \(O_{tea}^{topk} \in \mathbb{R}^{[T_{align}] \times k}, O_{stu}^{topk} \in \mathbb{R}^{[T_{align}] \times k}\) and token sets \(T_{stu}^{topk}, T_{tea}^{topk}\). At each token position \(i \in [0, |T_{align}|)\), there are three possible situations for the tokens during mapping:

• The tokens can be exactly matched keep the exact match mapping results(line 5)

• For tokens in \(T_{tea}^{topk}[i]\) that cannot be exactly matched, search for the most similar token from the tokens in \(T_{stu}^{topk}[i]\). To evaluate the similarity, we employ edit distance with length normalization with a similarity threshold \(\theta\) to mitigate noisy matches (lines 6-15)

• For the tokens that cannot find a similar token to match, their corresponding logits will be masked before distillation. (lines 19-20)

During the training phase, \(F_{dynamic}\) is continuously updated through this iterative process. After the alignment of student output to teacher output, the vocabulary distribution of \(O_{stu}^{seq}\) is projected via \(F_{dynamic}\) to establish a shared probability space with \(O_{tea}^{seq}\). To prevent noise from the mismatch vocabulary, we employ a masking operator Mask(\(\cdot\)) that masks the logits for unmatched positions in the vocabulary dimension. This alignment mechanism yields the refined outputs \(O_{tea}^{align}\) and \(O_{stu}^{align}\).

To enhance the alignment of crucial tokens for student modeling, we implement a reverse-direction alignment from teacher to student. This process generates a reverse mapping dictionary \(F_{reverse}^{dynamic}\) through analogous computational procedures. The alignment yields outputs: \(O_{tea}^{reverse} = F_{reverse}^{dynamic}(O_{stu}^{topk})\) and \(O_{stu}^{reverse} = \text{Mask}(O_{tea}^{topk})\). The final representations are constructed through vector concatenation (denoted by \(\oplus\)), formulated as: \(O_{stu}^f = O_{stu}^{align} \oplus O_{stu}^{reverse}\) and \(O_{tea}^f = O_{tea}^{align} \oplus O_{tea}^{reverse}\). \(O_{stu}^f \in \mathbb{R}^{[T_{align}] \times 2k}\) and \(O_{tea}^f \in \mathbb{R}^{[T_{align}] \times 2k}\) have the same dimension and the same meaning correspond to concrete tokens or spans in sequence and vocabulary dimension.

3.4 Aligned Logits Distillation

After performing contextual alignment in both sequence and vocabulary dimensions, the distribution differences between the teacher and student logits are computed using the KL divergence, as shown in Equation 5:

\[
L_{KL}(O_{stu}^f||O_{tea}^f) = \sum_{i=1}^k O_{stu}^f[i] \log \left( \frac{O_{stu}^f[i]}{O_{tea}^f[i]} \right)
\]

Meanwhile, the language modeling target uses standard cross-entropy loss for a sentence \(T = \{t_0, t_1, \ldots\}\) and is defined as Equation 6.

\[
L_{lm} = - \sum_i^{|T|} \log P(t_i|t_{i-1}, \cdots, t_1)
\]

Integrated with the language modeling loss, weighted by \(\alpha\), the final objective function is formulated in a manner analogous to classical model distillation, as presented in Equation 7.

\[
L = \alpha \cdot L_{KL} + (1 - \alpha) \cdot L_{lm}
\]

4 Experiments

In our experiments, we select five widely used open-source models with different architectures including Llama-3 (8B) (Dubey et al., 2024), OPT (1.3B/6.7B) (Zhang et al., 2022), Gemma-2 (2B/9B) (Team et al., 2024), Phi3 (mini-3.8B) (Abdin et al., 2024) and Qwen2 (0.5B/7B)(Yang et al., 2024). As listed in Table 2, according to our statistic results in Section 2, we select the combinations of the teacher model and student models from the pairs with relatively poor matching rates on vocabulary or sequence level.

<table>
  <tr>
    <th>teacher</th>
    <th>student</th>
    <th>SMR</th>
    <th>VMR</th>
  </tr>
  <tr>
    <td>Llama-3-8B</td>
    <td>Gemma-2-2B</td>
    <td>85.52%</td>
    <td>67.79%</td>
  </tr>
  <tr>
    <td>Llama-3-8B</td>
    <td>OPT-1.3B</td>
    <td>89.03%</td>
    <td>34.46%</td>
  </tr>
  <tr>
    <td>Phi3-mini-3.8B</td>
    <td>Qwen2-0.5B</td>
    <td>31.54%</td>
    <td>65.65%</td>
  </tr>
</table>

Table 2: The settings on teacher and student models, where the SMR means the sequence matching rate and VMR means the vocabulary matching rate.

4.1 Experiment Settings

Baseline Methods In our experiments, the primary baseline constitutes supervised fine-tuning (SFT) applied to both teacher and student models. To provide a comprehensive comparison, our baselines include the following methods for same-tokenizer model distillation (the teacher model maintains an identical architecture to the student model but with scaled-up parameters, e.g. Qwen2-7B serves as the teacher model for distilling knowledge into Qwen2-0.5B):
<table>
  <tr>
    <th>Type</th>
    <th>Name</th>
    <th>Number Train</th>
    <th>Number Test</th>
  </tr>
  <tr>
    <td rowspan="4">Instruction Following</td>
    <td>Dolly</td>
    <td>1100</td>
    <td>500</td>
  </tr>
  <tr>
    <td>Self-Inst</td>
    <td>-</td>
    <td>242</td>
  </tr>
  <tr>
    <td>Vicuna</td>
    <td>-</td>
    <td>80</td>
  </tr>
  <tr>
    <td>S-NI</td>
    <td>-</td>
    <td>1694</td>
  </tr>
  <tr>
    <td>UnNI</td>
    <td>-</td>
    <td>1000</td>
    <td></td>
  </tr>
  <tr>
    <td>Code</td>
    <td>CodeM</td>
    <td>9600</td>
    <td></td>
  </tr>
  <tr>
    <td>Generation</td>
    <td>HumanEval+</td>
    <td>-</td>
    <td>164</td>
  </tr>
  <tr>
    <td>Math</td>
    <td>MBPP</td>
    <td>-</td>
    <td>500</td>
  </tr>
  <tr>
    <td>Math</td>
    <td>GSM-8B</td>
    <td>7473</td>
    <td>1000</td>
  </tr>
</table>

Table 3: Data statistic of training and evaluation data.

• Forward KL divergence (FKL): the standard distillation loss function. Let \( p(x) \) be the distribution of the student model, and \( p(s) \) be the distribution of the teacher model, then the loss function is \( L_{fkl}(p(x)||q(x)) = \mathbb{E}_{q(x)}[log(\frac{q(x)}{p(x)})] \).

• Reverse KL divergence (RKL): reverse the distribution of teacher and student in KL divergence calculation.

And the following methods are for cross-tokenizer distillation:

• Unified Logits Distance (ULD) (Boizard et al., 2024): this cross-tokenizer distillation method leverages Optimal Transport to enable a unified distillation.

• MinED (Wan et al., 2024b): on the sequence level, the DTW uses edit distance as a cost function. On the vocabulary level, it uses exact match first and then uses edit distance to search the most similar token from the full vocabulary to get supplemental mappings.

Training Settings We use the learning rate 2e-5 and batch size 32 for Supervised Fine-Tuning (SFT) to train 10 epochs. For distillation methods, we follow the setting in MiniLLM (Gu et al., 2024), the details of hyperparameters are appended in Section A.1. We evaluate the methods on three types of tasks: instruction-following, code generation, and math. The training datasets include Dolly-15K (Ouyang et al., 2022) for instruction following, CodeM (Zan et al., 2024) for coding tasks, and GSM-8K for math tasks. The training is conducted on 8 Ascend 910Bs and using DeepSpeed3 ZeRO stage 1 for model parallel.

Evaluation Settings For instruction-following task, we follow the existing work (Gu et al., 2024; Zhang et al., 2024) and evaluate Rouge-L (Lin, 2004) on a series of instruction-following test sets including Dolly , Self-Instruction (Wang et al., 2023) (Self-Inst), Vicuna-Evaluation (Chiang et al., 2023) (Vicuna), Super-Natrul Instructions (Wang et al., 2022) (S-NI), and Unnatural Instruction (Honovich et al., 2022) (UnNI). For code generation tasks, the test set contains HumanEval+ (Chen et al., 2021) and MBPP+ (Austin et al., 2021) using EvalPlus (Liu et al., 2024) for an evaluation with more test cases and stricter. The evaluate metric is Pass@1, meaning the ratio of generated code can pass all test cases in one shot, and the decoding setting is greedy search. For math tasks, the test set is GSM-8K (Cobbe et al., 2021). The evaluate metric is Test@1, which has the same meaning as Pass@1 for math problems, and the decoding setting is also greedy search. The data statistic of training and evaluation data can be referred to Table 3.

4.2 Main Results

We conducted the fine-tuning and distillation experiments in the three settings of teacher-student. The main experiments of instruction-following are shown in Table 4, and there are two main findings. First, among all settings, the performance of CDM is significantly higher than other cross-tokenizer distillation baselines (e.g., CDM outperforms ULD by around 0.88 average Rouge-L). Second, compared with the same-tokenizer distillation methods, including FKL and RKL, CDM achieves better performance in OPT and significantly exceeds the related method of cross-tokenizer distillation\footnote{Noting that the performance between cross-tokenizer KD and the same-tokenizer KD methods cannot be compared strictly for the difference in their teacher capacity. However, this finding indicates the potential of the CDM.}.

For code generation and mathematical reasoning tasks, we excluded OPT models due to their inadequate pre-training in these specialized domains, which fundamentally limits knowledge distillation efficacy. Both Gemma and Qwen models achieved varying degrees of performance gains through distillation, with the CDM method consistently demonstrating the most stable and superior effectiveness among cross-tokenizer approaches. Particularly, Qwen2-0.5B delivers notable average improvements of 12.19% on code generation and 3.34% on math tasks. These consistent im-
<table>
  <tr>
    <th rowspan="2">Type</th>
    <th rowspan="2">Setting</th>
    <th colspan="4">Student Model: Gemma-2-2B</th>
    <th colspan="3">#Avg IF</th>
    <th colspan="2">HumanEval+</th>
    <th colspan="2">MBPP+</th>
    <th colspan="2">GSM-8K</th>
  </tr>
  <tr>
    <th>Dolly</th>
    <th>Self-Inst</th>
    <th>Vicuna</th>
    <th>S-NI</th>
    <th>UnNI</th>
    <th>#Avg IF</th>
    <th>HumanEval+</th>
    <th>MBPP+</th>
    <th>GSM-8K</th>
  </tr>
  <tr>
    <td rowspan="3">SFT</td>
    <td>Gemma-2-2B*</td>
    <td>25.72</td>
    <td>14.94</td>
    <td>16.89</td>
    <td>25.29</td>
    <td>30.07</td>
    <td>22.46</td>
    <td>21.34</td>
    <td>21.34</td>
    <td>29.95</td>
  </tr>
  <tr>
    <td>Gemma-2-9B</td>
    <td>26.72</td>
    <td>18.01</td>
    <td>18.85</td>
    <td>27.74</td>
    <td>34.83</td>
    <td>25.23</td>
    <td>24.39</td>
    <td>27.51</td>
    <td>45.34</td>
  </tr>
  <tr>
    <td>Llama-3-8B</td>
    <td>27.01</td>
    <td>21.90</td>
    <td>17.00</td>
    <td>30.66</td>
    <td>35.23</td>
    <td>26.36</td>
    <td>34.76</td>
    <td>50.26</td>
    <td>44.20</td>
  </tr>
  <tr>
    <td rowspan="3">Same Tokenizer KD*</td>
    <td>FKL</td>
    <td>26.51</td>
    <td>14.30</td>
    <td>18.64</td>
    <td>27.61</td>
    <td>32.06</td>
    <td>23.82</td>
    <td>18.90</td>
    <td>23.00</td>
    <td>34.80</td>
  </tr>
  <tr>
    <td>(teacher: Gemma-2-9B)</td>
    <td>RKL</td>
    <td>25.26</td>
    <td>13.80</td>
    <td>18.64</td>
    <td>23.70</td>
    <td>29.79</td>
    <td>22.24</td>
    <td>18.90</td>
    <td>21.42</td>
    <td>27.37</td>
  </tr>
  <tr>
    <td>Cross Tokenizer KD (teacher: Llama-3-8B)</td>
    <td>MinED</td>
    <td>25.83</td>
    <td><b>16.16</b></td>
    <td>16.40</td>
    <td>25.99</td>
    <td>28.60</td>
    <td>22.60</td>
    <td>20.12</td>
    <td><b>22.22</b></td>
    <td>28.43</td>
  </tr>
  <tr>
    <td rowspan="3">Cross Tokenizer KD (teacher: Llama-3-8B)</td>
    <td>ULD</td>
    <td>26.11</td>
    <td>14.58</td>
    <td>17.25</td>
    <td><b>27.69</b></td>
    <td>30.53</td>
    <td>23.23</td>
    <td>20.40</td>
    <td>17.70</td>
    <td>26.38</td>
  </tr>
  <tr>
    <td>CDM</td>
    <td><b>26.13</b></td>
    <td>14.89</td>
    <td><b>18.33</b></td>
    <td>26.40</td>
    <td><b>32.00</b></td>
    <td><b>23.55</b></td>
    <td><b>23.78</b></td>
    <td>21.69</td>
    <td><b>30.40</b></td>
  </tr>
  <tr>
    <td colspan="14">Student Model: OPT-1.3B</td>
  </tr>
  <tr>
    <td rowspan="3">SFT</td>
    <td>OPT-1.3B</td>
    <td>25.48</td>
    <td>14.26</td>
    <td>14.81</td>
    <td>25.88</td>
    <td>31.93</td>
    <td>22.47</td>
    <td>--</td>
    <td>--</td>
    <td>--</td>
  </tr>
  <tr>
    <td>OPT-6.7B</td>
    <td>28.40</td>
    <td>15.71</td>
    <td>15.82</td>
    <td>26.87</td>
    <td>33.56</td>
    <td>24.07</td>
    <td>--</td>
    <td>--</td>
    <td>--</td>
  </tr>
  <tr>
    <td>Llama-3-8B</td>
    <td>27.01</td>
    <td>21.90</td>
    <td>17.00</td>
    <td>30.66</td>
    <td>35.23</td>
    <td>26.36</td>
    <td>--</td>
    <td>--</td>
    <td>--</td>
  </tr>
  <tr>
    <td rowspan="3">Same Tokenizer KD* (OPT-6.7B)</td>
    <td>FKL</td>
    <td>25.36</td>
    <td>15.24</td>
    <td>16.16</td>
    <td>26.47</td>
    <td>31.58</td>
    <td>22.92</td>
    <td>--</td>
    <td>--</td>
    <td>--</td>
  </tr>
  <tr>
    <td>RKL</td>
    <td>25.03</td>
    <td>13.24</td>
    <td>15.42</td>
    <td>23.86</td>
    <td>31.27</td>
    <td>21.77</td>
    <td>--</td>
    <td>--</td>
    <td>--</td>
  </tr>
  <tr>
    <td>Cross Tokenizer KD (teacher: Llama-3-8B)</td>
    <td>MinED</td>
    <td>25.21</td>
    <td>12.60</td>
    <td>15.60</td>
    <td>24.51</td>
    <td>30.52</td>
    <td>21.69</td>
    <td>--</td>
    <td>--</td>
    <td>--</td>
  </tr>
  <tr>
    <td rowspan="3">Cross Tokenizer KD (teacher: Llama-3-8B)</td>
    <td>ULD</td>
    <td>25.45</td>
    <td>13.69</td>
    <td><b>15.88</b></td>
    <td>25.82</td>
    <td>30.07</td>
    <td>22.18</td>
    <td>--</td>
    <td>--</td>
    <td>--</td>
  </tr>
  <tr>
    <td>CDM</td>
    <td><b>26.15</b></td>
    <td>14.39</td>
    <td>15.77</td>
    <td><b>26.33</b></td>
    <td><b>32.33</b></td>
    <td><b>23.00</b></td>
    <td>--</td>
    <td>--</td>
    <td>--</td>
  </tr>
  <tr>
    <td colspan="14">Student Model: Qwen2-0.5B</td>
  </tr>
  <tr>
    <td rowspan="3">SFT</td>
    <td>Qwen2-0.5B</td>
    <td>24.66</td>
    <td>15.17</td>
    <td>15.22</td>
    <td>30.31</td>
    <td>35.00</td>
    <td>24.07</td>
    <td>15.85</td>
    <td>22.32</td>
    <td>27.22</td>
  </tr>
  <tr>
    <td>Qwen2-7B</td>
    <td>29.07</td>
    <td>22.69</td>
    <td>21.42</td>
    <td>37.31</td>
    <td>41.04</td>
    <td>30.31</td>
    <td>39.02</td>
    <td>39.42</td>
    <td>59.14</td>
  </tr>
  <tr>
    <td>Phi3-mini-3.8B</td>
    <td>29.19</td>
    <td>25.39</td>
    <td>21.81</td>
    <td>37.97</td>
    <td>41.07</td>
    <td>31.09</td>
    <td>51.83</td>
    <td>48.68</td>
    <td>64.67</td>
  </tr>
  <tr>
    <td rowspan="3">Same Tokenizer KD* (Qwen2-7B)</td>
    <td>FKL</td>
    <td>27.41</td>
    <td>19.68</td>
    <td>19.24</td>
    <td>32.67</td>
    <td>37.46</td>
    <td>27.29</td>
    <td>17.07</td>
    <td>23.38</td>
    <td>27.67</td>
  </tr>
  <tr>
    <td>RKL</td>
    <td>26.15</td>
    <td>16.15</td>
    <td>16.62</td>
    <td>30.32</td>
    <td>37.53</td>
    <td>25.35</td>
    <td>20.73</td>
    <td>22.75</td>
    <td>26.38</td>
  </tr>
  <tr>
    <td>Cross Tokenizer KD (teacher: Phi3-mini-3.8B)</td>
    <td>MinED</td>
    <td>25.55</td>
    <td>16.56</td>
    <td>15.37</td>
    <td>30.76</td>
    <td>35.69</td>
    <td>24.72</td>
    <td>17.10</td>
    <td>22.56</td>
    <td>24.41</td>
  </tr>
  <tr>
    <td rowspan="3">Cross Tokenizer KD (teacher: Phi3-mini-3.8B)</td>
    <td>ULD</td>
    <td><b>26.43</b></td>
    <td>16.15</td>
    <td>15.34</td>
    <td>30.63</td>
    <td>36.07</td>
    <td>24.93</td>
    <td>17.07</td>
    <td>22.49</td>
    <td>26.38</td>
  </tr>
  <tr>
    <td>CDM</td>
    <td>25.45</td>
    <td><b>16.55</b></td>
    <td><b>16.38</b></td>
    <td><b>30.66</b></td>
    <td><b>36.47</b></td>
    <td><b>25.10</b></td>
    <td><b>18.90</b></td>
    <td><b>23.81</b></td>
    <td><b>28.13</b></td>
  </tr>
</table>

Table 4: Main results of comparing CDM and the baseline models, where "#AVG IF" means the average score of the instruction-following tasks). The bold text means the best performance in comparable cross-tokenizer distillation settings. The table consists of three sections, each labeled with the student models in distillation experiments.

provements across three task categories substantiate CDM’s effectiveness, with comprehensive analyses provided in Section 5.1.

5 Analysis

Our analysis experiments adopts Phi3-mini-3.8B and Qwen2-0.5B as a default configuration, mainly including quantitative measurements of sequence-level and vocabulary-level alignment improvements (Section 5.1), comparative analysis with sequence-level knowledge distillation (Section 5.2), and systematic exploration of dual-teacher distillation paradigms (Section 5.3). We introduce “Average IF” as a composite metric aggregating performance across five instruction-following tasks to streamline result interpretation. Moreover, the supplementary analysis, including time cost analysis (Section A.2), evaluation on general instruction-following testset (Section A.3), ablation studies (Section A.4), sensitivity analyses (Section A.5), and comparative case studies of alignment outcomes (Sections A.6-A.7), are comprehensively documented in the Appendix.

5.1 The Statistic on Alignment Rate

In this section, we analyze the extent of the sequence-level and vocabulary-level alignment improvement separately.

Sequence Level  We define the sequence alignment rate through the following procedure. When aligning two sequences, adjacent tokens may merge into contiguous spans for correspondence mapping. Similar to the statistic in Section 2, the correspondent span pairs that cannot exactly match will be regarded as a mismatch. We sampled 3,000 sentences from the training set of Dolly. The alignment rate of the results in the pure edit distance cost function was 78.31%, which improved to 82.20% after adding entropy weight, demonstrating our method’s efficacy. Detailed case studies supporting these findings are presented in SectionA.6.

Vocabulary Level  On the vocabulary level, CDM optimizes noise suppression and coverage in vocabulary mapping at the same time. Exact Match (EM) provides unambiguous alignment but leads to limited mapping coverage, while similarity-based fuzzy matching inevitably introduces erroneous mappings that negatively impact model distillation
effectiveness. On the mapping coverage, the basic EM between Qwen2 and Phi3 achieves 65% on the coverage rate, and CDM significantly improves to 87%. There are two representative offline vocabulary matching approaches, including mapping using edit distance (ED) (Wan et al., 2024b) and edit distance with prefix constrain(PrefixED) (Wu et al., 2024), and both of them achieve over 99% coverage rates. To ensure rigorous comparison with existing approaches, we conduct controlled experiments maintaining identical experimental conditions except for the vocabulary mapping strategy. As shown in Table 5, outperforms these high-coverage methods by substantial margins, confirming its superior noise suppression capability. Extended case studies with detailed pattern analyses are available in Appendix A.7.

<table>
  <tr>
    <th>Setting</th>
    <th>Average IF</th>
  </tr>
  <tr>
    <td>ED</td>
    <td>24.32</td>
  </tr>
  <tr>
    <td>PrefixED</td>
    <td>24.12</td>
  </tr>
  <tr>
    <td>CDM</td>
    <td><b>25.10</b></td>
  </tr>
</table>

Table 5: Comparison between different vocabulary mapping methods.

5.2 Comparison with Sequence-Level KD

In this section, we provide a comparison between our method and sequence-level KD (Kim and Rush, 2016) (SeqKD). Unlike the probability-based methods discussed in the main experiments, SeqKD uses the generated text of the teacher model to enhance the student model’s performance. In the most advanced models, SeqKD is also applied for cross-tokenizer distillation scenarios (Guo et al., 2025).5 According to Table 6, the results demonstrate that SeqKD and SFT have close performance, which indicates the necessity of logit-based distillation.

<table>
  <tr>
    <th>Setting</th>
    <th>Average IF</th>
  </tr>
  <tr>
    <td>SFT</td>
    <td>24.07</td>
  </tr>
  <tr>
    <td>SeqKD</td>
    <td>24.05</td>
  </tr>
  <tr>
    <td>CDM</td>
    <td><b>25.10</b></td>
  </tr>
</table>

Table 6: The comparison between CDM and SeqKD.

5.3 Dual-teacher Distillation

To investigate the impact of knowledge distillation across different model architectures, we adopted an integrated approach that combines both same-tokenizer and cross-tokenizer knowledge distillation based on OPT-1.3B. Although the results in Table 7 indicate limited improvement over SFT for both FKD and CDM, a significant improvement (10.46% in instruction-following tasks) was observed when simultaneously leveraging the knowledge of two distinct teacher models through an average loss of distillation. The findings in dual-teacher settings imply that different tokenizer strategies may provide complementary information that is effectively utilized to enhance model performance.

<table>
  <tr>
    <th>Settings</th>
    <th>Average IF</th>
  </tr>
  <tr>
    <td>SFT</td>
    <td>22.47</td>
  </tr>
  <tr>
    <td>w/ OPT-6.7B (FKD)</td>
    <td>22.92</td>
  </tr>
  <tr>
    <td>w/ Llama-3-8B (CDM)</td>
    <td>23.00</td>
  </tr>
  <tr>
    <td>Dual Teacher</td>
    <td><b>24.82</b></td>
  </tr>
</table>

Table 7: The comparison for distillation using dual-teacher and single-teacher settings.

6 Related Work

In the context of cross-tokenizer distillation, several methods have been proposed to align the probability distributions of models before performing distillation. This alignment typically involves both sequence and vocabulary dimensions. Fu et al. (2023) aligns sequences through dynamic programming and aligns vocabularies through exact matching. To improve vocabulary alignment, the (Wan et al., 2024a,b) series introduced methods such as MinED and statistical matching for fuzzy matching to supplement vocabulary alignment. Despite these advancements, their effectiveness remains constrained by the prevalence of numerous mismatches. Beyond alignment strategies based on text character similarity, Boizard et al. (2024) proposed the use of optimal transport to quantify the distance between model logits. Furthermore, Cui et al. (2024) refine this approach by optimizing the cost function at both sequence and vocabulary levels, effectively integrating both local and global information to improve overall performance. Additionally, DSKD (Zhang et al., 2024) introduces a dual alignment framework that simultaneously aligns hidden states and model logits. However, both ULD and DSKD methods suffer from ineffi-

5Our implementation employs sampling decoding (Temperature=0.2) and integrates teacher-generated data with original training data, conducting 5-epoch training to maintain equivalent training iterations.
iciencies due to underutilization of the vocabulary.

7 Conclusion and Future Work

In this work, we propose Contextual Dynamic Mapping (CDM), a novel approach to CKTD. CDM enhances the CTKD by improving the alignment of model outputs on both sequence and vocabulary levels through the use of online context information. Specifically, the method incorporates entropy-based weights in the sequential alignment process and employs contextual Top-K token pairs to dynamically map vocabulary probabilities. Through extensive experiments across three groups of teacher-student configurations and three types of tasks (instruction-following, code generation and math), our method demonstrates significant advantages over existing approaches and shows further potential in dual-teacher scenarios. Furthermore, statistical analyses and case studies are presented to demonstrate how the method enhances model alignment accuracy in both sequence and vocabulary. In future work, we plan to further scale the application of CDM by assessing its performance on more diverse training datasets and larger student and teacher models to enhance scalability. In particular, the dual-teacher distillation will be extended to multi-teacher settings for further observation.

8 Limitation

Despite its effectiveness in cross-tokenizer knowledge distillation, our proposed Contextual Dynamic Mapping (CDM) method has certain limitations. First, CDM’s performance depends on the quality and diversity of the training data. Although we evaluated its effectiveness across multiple tasks, its applicability to other domains or data types, such as less structured or noisy text, remains unexplored. The method’s robustness to variations in data distribution and complexity requires further investigation. Second, our experiments are limited to student models ranging from 0.5B to 2B parameters and teacher models from 3.8B to 8B parameters. Due to computational constraints, we did not conduct experiments with larger-scale models using full-parameter fine-tuning, which is left for future research.

Acknowledgments

The research work described in this paper has been supported by the National Nature Science Foundation of China (No. 62376019, 62476023, 61976015, 61976016, 61876198, and 61370130), and the National Key R&D Program of China (2020AAA0108001). The authors would like to thank the anonymous reviewers for their valuable comments and suggestions for improving this paper.

References

Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219.

Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732.

Nicolas Boizard, Kevin El Haddad, Céline Hudelot, and Pierre Colombo. 2024. Towards cross-tokenizer distillation: the universal logit distillation loss for llms. arXiv preprint arXiv:2402.12030.

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna.lmsys.org (accessed 14 April 2023), 2(3):6.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.

Xiao Cui, Mo Zhu, Yulei Qin, Liang Xie, Wengang Zhou, and Houqiang Li. 2024. Multi-level optimal transport for universal cross-tokenizer knowledge distillation on language models. arXiv preprint arXiv:2412.14528.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Yao Fu, Hao Peng, Litou Ou, Ashish Sabharwal, and Tushar Khot. 2023. Specializing smaller language models towards multi-step reasoning. In International Conference on Machine Learning, pages 10421–10430. PMLR.
Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. 2024. Minilm: Knowledge distillation of large language models. In The Twelfth International Conference on Learning Representations.

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihaq Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.

Geoffrey Hinton. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531.

Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. 2022. Unnatural instructions: Tuning language models with (almost) no human labor. arXiv preprint arXiv:2212.09689.

Yoon Kim and Alexander M Rush. 2016. Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1317–1327.

Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se-Young Yun. 2024. Distillm: Towards streamlined distillation for large language models. arXiv preprint arXiv:2402.03898.

Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81.

Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2024. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744.

Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussonet, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118.

Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, and Shuming Shi. 2024a. Knowledge fusion of large language models. In The Twelfth International Conference on Learning Representations.

Fanqi Wan, Longguang Zhong, Ziyi Yang, Ruijun Chen, and Xiaojun Quan. 2024b. Fusechat: Knowledge fusion of chat models. arXiv preprint arXiv:2408.07990.

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13484–13508.

Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. 2022. Benchmarking generalization via in-context instructions on 1,600+ language tasks. arXiv preprint arXiv:2204.07705, 2.

Yuqiao Wen, Zichao Li, Wenyu Du, and Lili Mou. 2023. f-divergence minimization for sequence-level knowledge distillation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10817–10834.

Jiayi Wu, Hao Sun, Hengyi Cai, Lixin Su, Shuaiqiang Wang, Dawei Yin, Xiang Li, and Ming Gao. 2024. Cross-model control: Improving multiple large language models in one-time training. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671.

Daoguang Zan, Ailun Yu, Wei Liu, Bo Shen, Shaoxin Lin, Yongshun Gong, Yafen Yao, Yan Liu, Bei Guan, Weihua Luo, et al. 2024. Codem: Less data yields more versatility via ability matrix. In Findings of the Association for Computational Linguistics ACL 2024, pages 714–729.

Songming Zhang, Xue Zhang, Zengkui Sun, Yufeng Chen, and Jinan Xu. 2024. Dual-space knowledge distillation for large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 18164–18181.

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuhui Chen, Christopher DeWan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068.

Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911.

A Appendix

A.1 Details of Training Settings in Main Experiments

For all SFT baselines, the last checkpoint of 10 epochs is applied to be the final result. For distillation settings, three-epoch SFT is conducted first,
and then seven epochs of continual distillation are applied to keep the total steps the same. The hyperparameters about distillation are as Table 8.

<table>
  <tr>
    <th>Notation</th>
    <th>Value</th>
  </tr>
  <tr>
    <td>\( \theta \)</td>
    <td>0.3</td>
  </tr>
  <tr>
    <td>\( K \)</td>
    <td>100</td>
  </tr>
  <tr>
    <td>\( \alpha \)</td>
    <td>0.5</td>
  </tr>
  <tr>
    <td>\( T \)</td>
    <td>2.0</td>
  </tr>
  <tr>
    <td>\( C \)</td>
    <td>3</td>
  </tr>
</table>

Table 8: Hyperparameters of the distillation methods.

A.2 Time Cost Analysis

In CDM, the vocabulary and sequential alignment components require additional computational overhead, with both operations exhibiting quadratic time complexity relative to sequence length (\( O(n^2) \)). However, since the maximum sequence length configured during training is typically set to 512-2048 tokens in fine-tuning scenarios, this design enables containment of the overall training duration within practical limits.

In Table 9, we compare the time cost of DTW and the two baseline methods:

• ULD: the distillation method only aligns on the final loss, regarded as nearly a minimal time-consuming in cross-tokenizer training.

• MinED: align both on the sequential and vocabulary dimensions.

• CDM: using contextual information to improve the sequential and vocabulary dimension.

The experiment settings in this section are distilling Qwen2-0.5B by Phi-3-mini-3.8B in the code dataset for 10 epochs, and all the hyperparameters are the same as the main experiments.

<table>
  <tr>
    <th>train method</th>
    <th>time(hour:min:s)</th>
  </tr>
  <tr>
    <td>ULD</td>
    <td>6:16:18</td>
  </tr>
  <tr>
    <td>MinED</td>
    <td>7:53:04</td>
  </tr>
  <tr>
    <td>CDM</td>
    <td>8:31:04</td>
  </tr>
</table>

Table 9: Time cost for three CTKD methods.

According to the time cost of the three methods, the time increase of CDM is around 35% for ULD and around 7% for MinED, which is acceptable for the training process.

A.3 Evaluation on General Instruction-following

To strengthen the empirical validation, we augmented our evaluation framework through rigorous experimentation on IFEval(Zhou et al., 2023), a standardized benchmark for assessing instruction-following capabilities. Under the trained settings where Gemma2-2B is employed as the student model and Llama-3-8B as the teacher, with the Dolly training set, we evaluate the IFEval performance for the models trained by different methods in Table 10. The experimental results demonstrate that CDM significantly improves over the baseline methods. Specifically, CDM outperforms conventional distillation approaches by around 2%-7% in the average of accuracies under IFEval’s multi-dimensional criteria, confirming the superior capability of CDM.

A.4 Ablation Study

To verify the effectiveness of the modules of the method independently, we conducted ablation studies comparing three configurations: (1) removing sequence-level dynamic alignment, (2) using exact match directly and disabling dynamic vocabulary matching and (3) eliminating the dual mapping strategy (*i.e.*, cancel the reverse process in Section 3.3, and only calculate distillation loss based on the Top-K logits of the teacher model and its correspondent logits in the student model outputs). As evidenced by Table 11, each component removal adversely affects model convergence. Specifically, disabling sequence-level alignment reduces performance by 1.23 points on average, while removing vocabulary-level matching and dual mapping cause 0.86-point and 1.25-point degradations, respectively. These results quantitatively demonstrate the necessity of simultaneous optimization across both sequence and vocabulary dimensions and dual mapping.

A.5 On the Sensitivity of Hyper-Parameters

This section systematically examines the parameter sensitivity of two critical hyperparameters in our CDM framework: the similarity threshold (\( \theta \)) and \( k \) in the Top-K selection.

On Similarity Threshold  The similarity threshold \( \theta \) is the key to controlling the selection of candidate tokens for mapping, which can be referred to in Algorithm 1. In this section, we conduct a comprehensive sensitivity analysis by evaluating
<table>
  <tr>
    <th>setting</th>
    <th>inst_level_loose_acc</th>
    <th>inst_level_strict_acc</th>
    <th>prompt_level_loose_acc</th>
    <th>prompt_level_strict_acc</th>
    <th>Average</th>
  </tr>
  <tr>
    <td>SFT</td>
    <td>23.74</td>
    <td>22.66</td>
    <td>14.23</td>
    <td>13.49</td>
    <td>18.53</td>
  </tr>
  <tr>
    <td>ULD</td>
    <td>21.82</td>
    <td>21.22</td>
    <td>11.46</td>
    <td>10.91</td>
    <td>16.35</td>
  </tr>
  <tr>
    <td>MinED</td>
    <td>26.98</td>
    <td>25.54</td>
    <td>15.71</td>
    <td>14.60</td>
    <td>20.71</td>
  </tr>
  <tr>
    <td>CDM</td>
    <td><b>28.66</b></td>
    <td><b>27.34</b></td>
    <td><b>18.85</b></td>
    <td><b>17.38</b></td>
    <td><b>23.06</b></td>
  </tr>
</table>

Table 10: The experiment results of IFEval.

<table>
  <tr>
    <th>Setting</th>
    <th>Average IF</th>
  </tr>
  <tr>
    <td>CDM</td>
    <td><b>25.10</b></td>
  </tr>
  <tr>
    <td>- w/o Entropy-based Weight</td>
    <td>23.87</td>
  </tr>
  <tr>
    <td>- w/o Dynamic Vocabularay Mapper</td>
    <td>24.24</td>
  </tr>
  <tr>
    <td>- w/o Dual Mapping</td>
    <td>23.85</td>
  </tr>
  <tr>
    <td>SFT</td>
    <td>24.07</td>
  </tr>
</table>

Table 11: Ablation study on CDM.

four representative similarity thresholds: 0.0, 0.1, 0.3, and 0.5, as detailed in Table 12. To ensure consistent evaluation across tokens of varying lengths, we employed an edit-distance score normalized by token length. This approach ensures that a threshold of 0.0 corresponds to exact string matching, while a threshold of 0.5 permits a broader range of fuzzy matches, albeit at the expense of precision. Our experimental results demonstrate that an intermediate threshold strikes a better balance between accuracy and coverage, with thresholds of 0.1 and 0.3 emerging as particularly effective in this context.

<table>
  <tr>
    <th>\( \theta \)</th>
    <th>Average IF</th>
  </tr>
  <tr>
    <td>0.0</td>
    <td>24.24</td>
  </tr>
  <tr>
    <td>0.1</td>
    <td>24.98</td>
  </tr>
  <tr>
    <td>0.3</td>
    <td><b>25.10</b></td>
  </tr>
  <tr>
    <td>0.5</td>
    <td>24.31</td>
  </tr>
</table>

Table 12: Sensitivity test on hyperparameter similarity threshold, where 0.0 means exact matching, and 0.5 means a relatively low requirement on similarity.

On the selection of K for Top-K  The determination of \( k \) in Top-K sampling constitutes a pivotal design consideration for Cross-Domain Mapping (CDM), as it directly influences the number and diversity of candidate tokens during vocabulary mapping. To systematically evaluate how this hyperparameter affects model convergence, we conducted controlled experiments across multiple \( k \) values. Empirical results demonstrate that while all Top-K configurations outperform supervised fine-tuning baselines, optimal model performance occurs at \( k = 100 \). Conversely, performance degradation emerges at both extremes: undersized candidate pools (K=50) restrict mapping flexibility through excessive token exclusion, while oversized pools (K=200) introduce noise token mappings. This non-monotonic relationship is quantitatively validated in Table 13.

<table>
  <tr>
    <th>k</th>
    <th>Average IF</th>
  </tr>
  <tr>
    <td>50</td>
    <td>24.37</td>
  </tr>
  <tr>
    <td>100</td>
    <td><b>25.10</b></td>
  </tr>
  <tr>
    <td>200</td>
    <td>24.70</td>
  </tr>
  <tr>
    <td>500</td>
    <td>24.66</td>
  </tr>
</table>

Table 13: Sensitivity test on hyperparameter Top-K

A.6 Case Study on Sequence Alignment

The following cases in Table 14 demonstrate the effectiveness of the entropy-based weight in enhancing the DTW alignment algorithm. In the original DTW alignment results, misalignments occur frequently at positions with ambiguous meanings, such as commas or short alphabet spans at the beginning of sentences. In contrast, the entropy-weighted DTW approach in our method (CDM) achieves more accurate span alignment. By assigning higher weights to these ambiguous positions, which carry more information, the underestimation of their importance is reduced, resulting in improved alignment accuracy. This case study demonstrates that context-aware weight calibration substantially improves alignment robustness for linguistically ambiguous elements.

A.7 Case Study on Vocabulary Alignment

Table 15 provides case studies demonstrating how contextual information optimizes vocabulary mapping accuracy in CTKD. Our contextual alignment mechanism successfully resolves lexical ambiguities by integrating character-similarity and semantic dependencies. In contrast, conventional edit-
<table>
  <tr>
    <th>id</th>
    <th>type</th>
    <th>content</th>
  </tr>
  <tr>
    <td rowspan="4">1</td>
    <td>A</td>
    <td>Moon ||| Knight ||| is ||| Marvel ||| , ||| Batman ||| is ||| DC</td>
  </tr>
  <tr>
    <td>B</td>
    <td>Moon ||| Knight ||| is ||| Marvel ||| , ||| Bat ||| man ||| is ||| DC</td>
  </tr>
  <tr>
    <td>A'</td>
    <td>Moon|||Knight|||is|||Marvel|||,|||Batman|||is|||DC</td>
  </tr>
  <tr>
    <td>B'</td>
    <td>Moon ||| Knight ||| is ||| Marvel |||, Bat ||| man ||| is ||| DC</td>
  </tr>
  <tr>
    <td rowspan="2">2</td>
    <td>CDM A'</td>
    <td>Moon|||Knight|||is|||Marvel|||,|||Batman|||is|||DC</td>
  </tr>
  <tr>
    <td>CDM B'</td>
    <td>Moon ||| Knight ||| is ||| Marvel ||| , ||| Batman ||| is ||| DC</td>
  </tr>
  <tr>
    <td rowspan="4">3</td>
    <td>A</td>
    <td>D ||| odge ||| is ||| American ||| , ||| Volkswagen ||| is ||| German</td>
  </tr>
  <tr>
    <td>B</td>
    <td>D ||| odge ||| is ||| American ||| , ||| Volks ||| wagen ||| is ||| German</td>
  </tr>
  <tr>
    <td>A'</td>
    <td>D ||| odge ||| is ||| American |||, ||| Volkswagen ||| is ||| German</td>
  </tr>
  <tr>
    <td>B'</td>
    <td>D ||| odge ||| is ||| American |||, Volks ||| wagen ||| is ||| German</td>
  </tr>
  <tr>
    <td rowspan="2">-</td>
    <td>CDM A'</td>
    <td>D ||| odge ||| is ||| American |||, ||| Volkswagen ||| is ||| German</td>
  </tr>
  <tr>
    <td>CDM B'</td>
    <td>D ||| odge ||| is ||| American ||| , ||| Volkswagen ||| is ||| German</td>
  </tr>
</table>

Table 14: Three representative examples of sequence alignment outcomes demonstrate the impact of incorporating contextual information. Noting that although CDM is also not fully correct on the second case, its misalignment parts do not affect the main token meaning.

distance approaches exhibit fundamental limitations: (1) inability to capture semantic relationships beyond surface-form similarity, and (2) lack of dynamic adaptation to contextual variations. The contrast reveals that context-agnostic methods relying solely on character-level edit operations systematically neglect higher-order semantic associations crucial for knowledge transfer.
<table>
  <tr>
    <th>id</th>
    <th>type</th>
    <th>content</th>
  </tr>
  <tr>
    <td>1</td>
    <td>full sentence</td>
    <td>There isn’t any one bicycle that would be ideal for all people. Bike shops have experts who can advise the right model and size for you and your main uses. You could also look at product results from online bike shops and read reviews to supplement the advice from the shop. A meetup or ride with a local cycling group would be another great source of advice and targeted knowledge for making a decision which bike is right for you.</td>
  </tr>
  <tr>
    <td rowspan="2"></td>
    <td>w/o context</td>
    <td>"_publishers"→"gepubliceerd"</td>
  </tr>
  <tr>
    <td>w/ context</td>
    <td>"_publishers"→∅</td>
  </tr>
  <tr>
    <td>2</td>
    <td>full sentence</td>
    <td>The English army fought for King Harold Godwinson.</td>
  </tr>
  <tr>
    <td rowspan="2"></td>
    <td>w/o context</td>
    <td>"_fights"→"weights"</td>
  </tr>
  <tr>
    <td>w/ context</td>
    <td>"_fights"→"fight"</td>
  </tr>
  <tr>
    <td>3</td>
    <td>full sentence</td>
    <td>Championship rowing races are conducted over 2 kilometers (1.2 miles) with dedicated lanes delineated by bouys.</td>
  </tr>
  <tr>
    <td rowspan="2"></td>
    <td>w/o context</td>
    <td>"kilom"→".iloc"; "denoted"→"_devoted"</td>
  </tr>
  <tr>
    <td>w/ context</td>
    <td>"kilom"→" kilomet"; "denoted"→"_defined"</td>
  </tr>
</table>

Table 15: Three representative examples of token mapping are presented, where the prefix '_' denotes a space. The tokens are from the Top-K tokens, so they may not appear in the original full text.

Reviewer paper summaries: This paper introduces Contextual Dynamic Mapping (CDM), a novel framework for cross-tokenizer knowledge distillation (CTKD), which addresses sequence misalignment and vocabulary mismatch when transferring knowledge between teacher and student models with different tokenization schemes. CDM improves alignment through entropy-weighted Dynamic Time Warping (DTW) for sequence alignment and dynamic top-K token mapping for vocabulary alignment. The method is evaluated across five model families and three task categories (instruction-following, code generation, math), demonstrating consistent improvements over CTKD baselines such as MinED and ULD.

This paper proposes Contextual Dynamic Mapping (CDM), a novel approach to enhance Cross-Tokenizer Knowledge Distillation (CTKD), which suffers from sequence misalignment and vocabulary mismatch due to differing tokenization strategies. CDM improves sequence alignment using an entropy-weighted Dynamic Time Warping (DTW) method and enhances vocabulary mapping by dynamically selecting contextually relevant token correspondences. Experimental results across multiple teacher-student model pairs (e.g., Llama3, Phi3, Qwen2) demonstrate that CDM outperforms existing cross-tokenizer KD methods, achieving higher accuracy in instruction-following, code generation, and math tasks. Furthermore, combining same-tokenizer and cross-tokenizer distillation through CDM yields additional performance improvements. These findings suggest that CDM effectively facilitates knowledge transfer between models with different tokenization schemes, making it a valuable technique for heterogeneous LLM compression.

The paper introduces Contextual Dynamic Mapping (CDM), a novel framework for cross-tokenizer knowledge distillation (CTKD) aimed at addressing the challenges of sequence misalignment and vocabulary mismatch in heterogeneous model architectures. Traditional knowledge distillation (KD) excels with homogeneous tokenizers but struggles across different tokenizers due to divergent tokenization strategies and vocabulary compositions. CDM leverages contextual information to enhance sequence alignment via entropy-weighted dynamic programming and dynamically maps vocabularies using context-aware candidate matching. Evaluated across five model families (LLama3, Phi3, Gemma2, OPT, Qwen2) and three tasks (instruction-following, code generation, and math), CDM outperforms existing CTKD baselines and, in some cases, same-tokenizer KD. The study also explores dual-teacher distillation, combining conventional same-tokenizer KD with CDM-based CTKD, further boosting performance.
Summary of strengths: - **Comprehensive evaluation across diverse benchmarks**
    - CDM is tested on five model families and three distinct NLP tasks, demonstrating its general applicability.
- **Takeaway message from analysis**
    - The necessity of logit-based distillation is well-demonstrated in Section 5.2. This highlights the importance of cross-tokenizer KD through logits rather than direct text outputs.
    - Dual-teacher distillation in Section 5.3 is an insightful contribution, showing that integrating both same-tokenizer and cross-tokenizer KD provides complementary signals, leading to significant performance gains.
- **Thorough ablation**
    - The authors conduct case studies on alignment, ablation studies, and sensitivity tests to validate CDM’s components.

1. CDM effectively addresses sequence misalignment and vocabulary mismatch issues, which are key challenges in CTKD. It improves knowledge transfer between models with different tokenization schemes, making heterogeneous model compression more viable.

2. CDM consistently outperforms prior CTKD methods (ULD, MinED) across instruction-following, code generation, and math tasks.

3. Evaluated across multiple LLM architecture (Llama3, Phi3, Gemma2, OPT, Qwen2), demonstrating its broad applicability. The combination of same-tokenizer and cross-tokenizer distillation further enhances model performance.

S1: The paper provides comprehensive statistical and case study analyses, deepening the understanding of CDM’s impact.

S2: The method is rigorously tested across five diverse model families and multiple tasks, providing strong empirical evidence of its superiority over baselines.

S3: CDM introduces a novel use of contextual information to tackle both sequence and vocabulary alignment, surpassing the limitations of traditional probability-matching methods.
Summary of weaknesses: - **Method novelty and insufficient justification for design choices**
    - The proposed approach introduces a bit technical modifications to existing methods, and the rationale for these modifications is not thoroughly justified.
    - More specifically, the paper does not provide a strong theoretical or empirical justification for using entropy-based weighting. The explanation with Figure 1 (Lines 183-194) is just a single case, and it does not generalize to explain why higher entropy should always lead to weaker mappings. A broader justification or more empirical analysis would strengthen the argument.
- **Marginal gains in instruction-following tasks**
    - While CDM provides substantial improvements in code and math, gains in instruction-following tasks are relatively modest, suggesting a potential limitation in handling free-form natural language. A discussion on why instruction tasks see smaller gains would improve clarity.
- **Lack of experiments in extreme settings**
    - The paper does not test CDM in settings where the matching rate between teacher and student models is extremely low (e.g., teacher: Gemma, student: Phi).
    - Evaluating CDM and baselines in such settings would validate its robustness under severe sequence and vocabulary mismatches.
- **Unfair comparison with same-tokenizer KD**
    - The paper reports cases where CDM outperforms same-tokenizer KD (Lines 385-388), but Table 3 shows that the teachers used in same-tokenizer KD are weaker than those used in cross-tokenizer KD. The performance gap may be attributable to the strength of the teacher models rather than the KD approach itself.
    - To support the claim that CDM is superior to same-tokenizer KD, the authors should use teachers of similar performance.
- **Missing relevant baselines from related work**
    - Some methods [1,2] from related work are not included as baselines, despite their relevance to cross-tokenizer KD.



[1] Cui et al., Multi-Level Optimal Transport for Universal Cross-Tokenizer Knowledge Distillation on Language Models, AAAI 2025

[2] Zhang et al., Dual-Space Knowledge Distillation for Large Language Models, EMNLP 2024

1. It seems DTW might require nearly quadratic time complexity for in sequence length, which might significant increasing of training time. Could you provide the time complexity and analysis for training time of proposed method?

2. There are too many mathematical definitions, such as $cost(\cdot, \cdot)$ or $EditDistnace$, used to define the proposed method. I believe readability could be improved by using notation tables or other clarifying techniques.

W1: Experiments are constrained to smaller models (0.5B-2B for students, 3.8B-8B for teachers), leaving its efficacy on larger models unexplored due to computational limits.

W2: The dual alignment process (sequence and vocabulary) and hyperparameter tuning may increase computational and design complexity.

W3: Evaluations focus on three specific tasks, potentially limiting generalizability. May include some general datasets like IFEval and GPQA.
Comments, suggestions and typos: - **Failure case discussion**
    - Include a brief discussion on potential failure cases, such as extreme vocabulary mismatch scenarios or settings where entropy-based alignment may not perform well.
- **Typographical errors**
    - CKTD → CTKD (Line 518).
    - Appendixn → Appendix (Line 460).
    - blod → bold (Table 3 caption).

**Typos**

- Appendix A.1, "Details of Training Settings in Main Experiemnts" $\rightarrow$ "Details of Training Settings in Main Experiments"
- Appendix A.2, " daul mapping strategy" $\rightarrow$ "dual mapping strategy"
- Table 10, "Sensitivty test on hyperparameter Top-K" $\rightarrow$ "Sensitivity test on hyperparameter Top-K"
- Table 7, "Hyper-paramenters of the distillation methods" $\rightarrow$ "Hyper-parameters of the distillation methods"
- Introduction, "The detail statistic process is shown at Section 2" $\rightarrow$ "The detail statistical process is shown at Section 2"
- Section 5.2, "SeqKD is also applied for cross-tokenizer distillation seneraios" $\rightarrow$ "SeqKD is also applied for cross-tokenizer distillation scenarios"

S1: Tables 4-7 can be better organized in a more concise location. 

S2: The intro is too long to capture the info. It would be better to follow the conventional way, demoposing as intro and related work sections.
Overall assessment scores: 3

3.5

3
Metareview score: 3.5
Metareview: All reviewers acknowledged the positive results from the proposed method, as well as the importance of the studied problem. Some reviewers asked for experiments with additional model configurations, on additional tasks, and comparisons to additional baselines. The authors added one more dataset during the author response, also with positive results, and sufficiently addressed questions regarding computational complexity of the method. 

There was a major concern about an unsupported claim that cross-tokenizer distillation outperforms same-tokenizer distillation, but the authors added more context and promised to rephrase their claim to reflect the inability to perform a strictly controlled comparison for that aspect.

Overall I believe this is a sound and interesting contribution. Because the novelty over prior methods is not very high, and the method adds complexity, I am recommending borderline Conference.
Summary of reasons to publish: * The paper presents a new method for cross-tokenizer distillation, which outperforms appropriate baselines from prior work on three types of tasks, and an additional evaluation (IFEval) added during the author response period
* The experiments are extensive, including teacher-student configurations with five language model families 
* The method is described relatively clearly, with more improvements promised for a revision