Overcoming Vocabulary Mismatch: Vocabulary-agnostic Teacher Guided Language Modeling
Authors: Haebin Shin
Lei Ji
Xiao Liu
Yeyun Gong
Venue: ICML 2025 poster (2025)
Keywords: Knowledge Distillation
Cross-Tokenizer
Vocabulary-agnostic Pretraining
Teacher Guided Language Modeling
Primary area: deep_learning->large_language_models
TL;DR: We propose Vocabulary-agnostic Teacher Guided Language Modeling (VocAgnoLM), enabling guidance from any teacher model during student model pretraining, regardless of vocabulary mismatch.

Abstract: Using large teacher models to guide the training of smaller student models has become the prevailing paradigm for efficient and effective learning. However, vocabulary mismatches between teacher and student language models pose significant challenges in language modeling, resulting in divergent token sequences and output distributions. To overcome these limitations, we propose Vocabulary-agnostic Teacher Guided Language Modeling (VocAgnoLM), a novel approach that bridges the gap caused by vocabulary mismatch through two key methods: (1) Token-level Lexical Alignment, which aligns token sequences across mismatched vocabularies, and (2) Teacher Guided Loss, which leverages the loss of teacher model to guide effective student training. We demonstrate its effectiveness in language modeling with 1B student model using various 7B teacher models with different vocabularies. Notably, with Qwen2.5-Math-Instruct, a teacher model sharing only about 6% of its vocabulary with TinyLlama, VocAgnoLM achieves a 46% performance improvement compared to naive continual pretraining. Furthermore, we demonstrate that VocAgnoLM consistently benefits from stronger teacher models, providing a robust solution to vocabulary mismatches in language modeling.

Overcoming Vocabulary Mismatch:
Vocabulary-agnostic Teacher Guided Language Modeling
Haebin Shin 12
Lei Ji
Xiao Liu
Yeyun Gong'
Abstract
Vocab overlap ratio for each Teacher Model
90
100
Overlap
Using large teacher models guide the train- ing of smaller student models has become the prevailing paradigm for efficient and effective learning. However, vocabulary mismatches be-
80
Overlap in Teacher Vocabs (%)
Teacher Performance
70
75.58%
80
60
60
50
tween teacher and student language models pose significant challenges in language modeling, re- sulting in divergent token sequences and out- put distributions. To overcome these limitations, we propose Vocabulary-agnoot
40
Same Vocebs w/ Studeett
40
30
Different Vocabs Student
20
Teacher Guided Language Modeling MVo novel ap- proach that bridges the gap caused by vocab- ulary mismatch through two key methods: (l) Token-level Lexical Alignment, which aligns to- ken sequences across mismatched vocabularies, and (2) Teacher Guided Loss, which leverages the loss of teacher model guide effective stu- dent training. We demonstrate its effectiveness in language modeling with 1B student model using various 7B teacher models with different vocab- ularies. Notably, with Qwen2.5- -aatnh Instruct, a teacher model sharing only about 6% of its vo- cabulary with TinyLlama, VocAgnoLM achieves 46% performance improvement compared to naive continual pretraining. Furthermore, we demonstrate that VocAgnoLM consistently ben- efits from stronger teacher models, providing robust solution to vocabulary mismatches in lan- guage modeling.
10
6.32%
Liemma (32K)
Mistral Aroral-FroMMath (32K)
DecoSockMach (100K)
Qwenz.5 (150K)
Teacher Model (Num. vocabs)
Figure I: Limitation in Utilizing Better LLMs as Teacher
Models due to Vocabulary Mismatch: Qwen2.5-Math (Yang et al., 2024) outperforms Llemma (Azerbayev et al., 2024) on math evaluation suite, but shares only 6.32% of its vo- cabulary with the student model, TinyLlama (Zhang et al., 2024a).
training. This paradigm has been instrumental in addressing critical challenges, such as compensating for the limited ca- pacity of smaller models during pretraining (Gemma- Team et al., 2024: Meta, 2024; Muralidharan et al., 2024), optimiz- ing language modeling with carefully selected curated data (Gu et al., 2024b; Lin et al., 2024), and providing task-specific on-policy guidance for downstream tasks (Gu et al., 2024a; Agarwal et al., 2024). By inheriting capabili- ties from more advanced models, student LLMs can refine their behaviors, align with specific objectives, and achieve substantial performance gains.
1. Introduction
Despite these advances, critical bottleneck remains: the vocabulary mismatch between teacher and student models. Most current methods assume identical vocabularies, lim- iting the student model's ability to benefit from the latest state-of- the-art or camain--peccillzed teacher models with distinct tokenization schemes. This constraint not only pre- vents student models from leveraging diverse open-source LLMs but also imposes additional costs, such the need train customized teacher models rely exclusively on large models with compatible vocabularies (Gu et al., 2024b; Lin et al., 2024). As illustrated in Figure 1, the vocabu-
Large language models (LLMs) have increasingly adopted guidance from teacher models to enhance student LLM
'Work done during internship at Microsoft Research. 'Microsoft Research "KAIST AL. Correspondence to: Haebin Shin Chinbin.shiife @ kaist.ac.kr>> Lei <leiji miiiiosoooossmmmmm Xiao Liu <xiao. liu. www.m.maaiaaee. micmosoft.com>. Yeyun Gong <yegong microsoft.com..
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Overcoming Vocabulary Mismatch: Yocabulary-agnostie Teacher Guided Language Modeling
Input: wer. Probability, Uncertainty and Quantitative Risk
Qwen
Qwen
aMA
DeepSeck
Qwen
TiyyLaaMA
DeepSeek
Probuhility
Und
ertainty
and
Quani
Fattive
Ris
Pro
Mistral
Probability
ability
Prob
Cy(=ā)
4y(x)
Pro
b
ability
Uni ertain
and
Quant
itarive
ability
Co(xf)
ability
Proh
hiiliry
Ung estain
DeepSeek
TY Land
Quani itarive Ri
crtainty
Un
Une
Cr(x5)
(x(x5)
Ln
2y(x))
vertain
cortain
portain
TiayLLaMA
1y
Pro
ability
Un
and Quant itative
Token Sequence Misalignment
Token-level Lexical Mapping
Teacher loss based Guidance
Figure 2: Overview of Vocabulary-agnosti Teacher Guided Language Modeling. Left: Teacher models (such as Qwen, Mistral, DeepSeek) produce token sequences that differ from those of the student model (TinyLlama), leading to misalign- ment. Middle: To address this, Teven-level Lexical Mapping establishes one-to-many mapping from each student token to corresponding teacher tokens. Right: To overcome logit distribution divergence, the mapped teacher token loss is utilized to guide the training of the student model.
lary overlap ratio between Qwen (Qwen et al., 2024) and TinyLJama (Zhang et al., 2024a) is only 6.32%, despite Qwen's strong performance. This disparity underscores how vocabulary mismatches severely limit the adoption of high-performing teacher models, creating an urgent need for approaches that transcend these restrictions.
We evaluate VocAgnoLM by continual pretraining TinyL- LaMA 1.1B (Zhang et al., 2024a) using 7B teacher models built on a different vocabulary system, such as Mistral (Jiang et al., 2023), DeepSeek (DeepSeek-AT et al., 2024), or Qwen2.5 (Qwen et al., 2024). Notably, using Qwen2.5- Math-Instruce Instruct (Yang et al., 2024), which has only 6% vo- cabulary overlap with TinyLLaMA, VocAgnoLM achieves 46% performance improvement over naive continual pre- training baseline and a 33% improvement over a logit distribution mapping baseline.
Given this mismatch, we analyze the tokenization discrep- ancy between different LLMs to observe two primary lim- itations: (1) token sequence mismatch and (2) logit dis- tribution divergence. As illustrated in Figure 2, different LLMs tokenize the same input phrase " "Probability Uncertainty" into 4, 6, and 7 tokens, respectively. This tokenization variability disrupts sequence alignment, complicating the student model's ability to interpret the teacher model's outputs. Furthermore, even if two models produce a similar number of tokens, their logit distributions can vary significantly due to differences in architecture, training data, or optimization techniques. This makes it challenging to directly use teacher model's logits as guid- ance for student model.
We highlight the effectiveness of vocabulary-agnosti dis- tillation, enabling specialized teacher models to be directly utilized without vocabulary constraints. Our approach not only mitigates the vocabulary mismatch issue but also en- hances the efficiency and performance of domain language modeling tasks.
Our contributions are summarized in three folds:
We tackle the vocabulary mismatch problem in the language modeling domain, which has remained un- derexplored despite its critical impact on leverag- ing teacher guidance. To address this, we intro- duce Vocabulary-agnostic Teacher Guided Lan- guage Modeling VV).Ag.L.MM, novel approach that bridges vocabulary discrepancies, enabling effec- tive cross-vocabulary teacher guidance.
To overcome these challenges, we propose simple yet effective approach, Token-level Lexical Alignment. Our method achieves token-level alignment in a one-to-many manner, enabling student models to receive fine-grained guidance from teacher models without requiring additional training to unify vocabularies. By bridging the gap be- tween differing vocabularies, Teven- Lexical Align- ment allows the efficient integration of emerging teacher models and ensures the student model can effectively learn from teacher models' guidance. Additionally, we introduce Teacher Guided Loss to leverage aligned tokens, allowing the student to benefit from the aligned teacher outputs, even when their logit distributions differ.
We propose Token-level Lexical Alignment ap- proach that enables ine-grained, token-by-token guid- ance to address sequence mismatches between student and teacher tokens. Furthermore, we employ a teacher loss based guidance to address the logit distribution gap, ensuring the student model's effective training across vocabularies mismatch.
2

---

Overcoming Vocabulary Mismatch: aocaoulary-agnostic Teacher Guided Language Modeling
Sequence Misalignment by Granularity
leads to progressive decrease in the coverage of student tokens by teacher chunks, thereby making it increasingly difficult for the student to receive precise guidance from the teacher. This motivates us to design comprehensive alignment algorithm, where the student can receive token- level guidance from fine-grained teacher token alignment, ensuring precise and effective supervision.
1.0
0.9
0.8
Overlap
0.7
0.6
3. Vocabulary-agnostic Teacher Guided Language Modeling
Mistrel
Charlevel los
DeepSrek
Charllevel lou
0.5
Owen
Word-level loU
We introduce Vocabulary-agnostic Teacher Guided Lan- guage Modeling (VocAgnoLM) to address vocabulary mis- match between student model (S) and a teacher model (T). To tackle the token sequence mismatch, and logit distri- bution divergence, we propose Token-level Lexical Align- ment (Section 3.1) for sequence alignment and Teacher Guided Loss (Section 3.2) to enable effective teacher guid- ance despite differing logit distributions.
Number of Chunks
Figure 3: Comparison of Sequence Overlap by Granularity. Sequence overlap between the corresponding chunks of stu- dent (TinyLlama) and teacher models differs significantly across varying levels of granularity (Number of Chunks). IoU (Intersection over Union) refers to the overlap ratio between the two sequences, while loS (Intersection over Student sequence) denotes the coverage of the student se- quence by the teacher sequence.
3.1. Token-level Lexical Alignment
Our primary objective is to enable the student model to receive fine-grained, token-level guidance from a teacher model during language modeling. The challenge arises when the student and teacher models have different vocabu- laries, leading to discrepancies in how the same text span is split into tokens. For example, single student token may span multiple teacher tokens or vice versa due to their unique vocabulary sets. To address this, we leverage character-level offsets [st, ed] representing the starting and end positions of a token, which can precisely map the tokens between two sequences according to the original position in the raw text.
We demonstrate that our method yields performance improvements proportional to the strength of the teacher model, regardless of vocabulary mismatch. This flexibility allows new, high-performing teacher models to be seamlessly integrated without the need for vocabulary compatibility, showcasing the practicality of our approach.
2. Preliminary Study
Our motivation begins with the challenge of aligning two token sequences that have been segmented differently by a teacher and student model. According to the show- case in Figure 2, various tokenizers generate quite different lenghh of token sequence. A straightforward approach to align them is to divide both sequences into an equal number of coarse- grainedd chunks (Xie et al., 2024; Xu et al., 2025). We first equally chunk each sequence into predefined num- ber of chunks and then evaluate the Intersection over Union (IoU) and Intersection over Student sequence (IoS) for each pair of mapped chunks to assess the quality of the mapping.
Let denote student tokens {-ī ............................ xx with char- acter offsets [st. edā] for each 2;S; and teacher tokens {aT .o7 ...,,,,]> with character offsets [st].e]"]] for each 2T.. By tracking these offsets, we can determine the exact text span covered by a student token a; and identify the corresponding teacher tokens { {x]"......]]) that fully cover that span. Specifically, for each student token æ; we define the mapping function (mapping(i) to be the index range of teacher tokens that covers x5 as follow as:
..q.........
-1,
if none teacher token covers x
As illustrated in Figure 3, we observe the degree of align- ment mismatch at various levels of granularity to understand how these two differently segmented sequences diverge. As we increase the number of chunks (i.c., move toward a finer-grained segmentation), the alignment between the two sequences deteriorates. On the one hand, small number of chunks shows higher overlap, however, this leads to a coarse- grained mapping. On the other hand, fine-grad finennrained mapping with large number of chunks exhibits lower overlap, due to the equally chunking strategy. As shown in Figure 3, this
(.),,,
if
25 C xT A x g [+1M
mapping|]]
A S E [1-11
Here, of [. denotes the concatenation of teacher tokens 2T I f++....... ................................ oT. The conditions 23 E xTf41. A o g 1.........
z [ ensure that (j,k) is the minimal index range that covers x.S. And, mapping[i] = 1 corresponds to an un-
3

---

Overcoming Vocabulary Mismatch: aggnotlary-agnostic Teacher Guided Language Modeling
Algorithm I Token-level Lexical Alignment
Here, & represents aggregation function (e.g. summation, maximum, or mean), applied to the teacher model's token loss over the range [j, As..
Input:
Student tokens (xx²) with offsets {ssts {d
Teacher tokens {x with offsets {[st] edj
We utilize the mapped teacher token losses to guide the reweighting of the student token's importance, as defined in 3.
Output: Mapping mapping[i
for each i-th student token x; do
(1) Get student character-level offsets:
((S) -Bi-p../ [M(=}) log P(x x ()
sti edf + =
(3)
(2) Find overlapping teacher range via binary searches:
lowldx lower bound of t edf > st
Specifically, W(x))) determines whether a given token is adopted based on the difference between the student and the corresponding teacher token losses, which reflects the importance of the token (Fan & Jaggi, 2023; Lin al., 2024). We apply top-k threshold to identify the most important tokens, ensuring that only tokens with significant contributions are retained, along with unmapped student tokens (further discussed in Section 6.2).
highldx + upper bound of { st < ed } -1
(i. k) + (lowldx, highldx)
(3) Map corresponding teacher tokens: IT [JA]]
mappingsi] + { 1++1, 1..., k}
end for return mapping
mapped student token, which is not lexically covered by any teacher tokens.
1, if Ls(x))) Lr(
or mapping/1] (.)) E Threshold
W(=a)
-1,
These mappings can be efficiently determined by perform- ing two binary searches for each student token: one to find the earliest teacher token index (lowldx) whose end offset exceeds st.5 and another to find the latest teacher token index (highldx) whose start offset remains below ed," The detailed procedure is outlined in Algorithm 1. Since each of these range searches can be completed in Olog M) time, where M is the number of teacher tokens, the overall com- plexity for all N student tokens is (NN log M).
0, otherwise
(4) Teacher token loss-based guidance enables the student model to receive guidance from the teacher model even when their vocabulary spaces differ, as each model com- putes its logit dimensions independently.
4. Experimental Setup
By defining a clear token aligmment procedure, we establish a one-to-many mapping (one student token x! potentially corresponding to multiple teacher tokens = =(j. This align- ment enables the student model to more precisely leverage the teacher model's outputs, facilitating a form of token- level supervision.
4.1. Dataset
(Paster et al., 2024), containing about 15 billion tokens sourced from math-related web pages in the Common Crawl.
Pretraining Corpus. We utilize
Evaluation Setup. To evaluate performance, we assess the model on 9 mathematical reasoning benchmarks cov- ering diverse domains, question formats (multiple-choice and open-ended), and difficulty levels (elementary to uni- versity): GSM8k (Cobbe et al., 2021), MATH (Hendrycks et al., 2021b), GSM-Hard (Gao et al., 2022), SVAMP (Patel et al., 2021), ASDiv (Miao al., 2020), MAWPS (Koncel- Kedziorski et al., 2016), TabMWP (TAB) (Lu et al., 2023), MathQA (MQA) (Amini al., 2019), MMLU- STEM (Hendrycks al., 2021a), and SAT (Azerbayev et al., 2024). We utilize few-shot cfain-of-thouuht (CoT) exam- ples (Wei et al., 2022) following the settings in Lin et al. (2024); Zhou et al. (2024).
3.2. Teacher Guided Language Modeling
Although Token-level Lexical Alignment aligns the two token sequences, the mismatch in logit distribution remains a challenge due to the differing vocabularies of the student and teacher models. To address this challenge, we leverage the loss values of the mapped teacher tokens to guide the importance of student tokens (Fan & Jaggi, 2023; Lin al., 2024).
Based on the Token-level Lexical Alignment described in Section 1, the causal language modeling for the i-th stu- dent token (x²) and the corresponding token losses for the
teacher tokens (xT).A)) spanning from j to le are defined as follows:
4.2. Baselines
Cs(xş)) = - log P(x.5 5.
KLD. When the teacher model shares the same vocab- ulary, one of the most representative teacher guided lan- guage modeling is knowledge distillation based on Kull- back-Leibler Divergence (KLD). Following Song et al.
(T((TT)))) = :DEELAN [ -log P( x

---

Overcoming Vocabulary Mismatch; agcooica Teacher Guided Language Modeling
(2020); Gu et al. (2024a), we incorporates the teacher- student logit KI. divergence term to student model's cross entropy as defined in Equation (I). Detailed equation is described in Appendix B.
Performance Comparison w/ various Teacher Models
30
KLD
ULD
Ours
25
Performance
20
ULD. Boizard et al. (2025) introduce the Universal Logit Distillation (ULD) loss, designed to align the probability distributions of a student model and teacher model with differing vocabularies. ULD loss minimizes the Wasserstein distance between the two distributions during finetuning on various downstream tasks, serving as an alternative to KL divergence. In this work, we compare the ULD loss- based logit distribution alignment with our teacher token loss-based guidance in the context of language modeling. Detailed equation is described in Appendix B.
15
10-
TinyLlama
Liemma
Mistral-ProXMain
DeepSeekMath
OwenZ Mathh
DeepSeelMMtt--R
Owen2 5 5.Math Inst
Figure 4: Performance Comparison Across Various Teacher Models. VocAgnoLMM consistently outperforms logit distribution- bannd baselines.
Rho-1. Lin et al. (2024) utilize guidance from a well- curated oracle reference model with a shared vocabulary (c.g., trained on GPT-generated datasets or targeted cor- pora) to enable efficient language modeling through vari- ous scoring methods. this study, we compare their ap- proach, which employs the lama architecture the reference model initially trained on OpenWebbMe (Paster et al., 2024), using multi critercc scoring based on token entropy and loss delta.
Performance improvement based on the Teacher
Teacher
ULD
ULD w/ Inst
Ours
Teacher Performance
80
Ours Inst
CPT
28
Porfommacce
Student Performanc
75
24
65
4.3. Implementation Details
22
60
We use LitGPT (Lightming-AI, 2023) to continually pretrain on 15B tokens from WebMath (Paster et al., 2024). Training is conducted on 32 H100 GPUs with a cosine learning rate scheduler (decaying from 8e-5 to 8e-6), a sequence length of 2048, and a global batch size of 2M tokens, following prior works (Zhang al., 2024a; Lin et al., 2024; Zhou et al., 2024). We apply a top-k threshold of 40%. Details are described in Appendix B.
20
55
Prstral-ProXMath (32K)
SeepSatt
Qwon2.5.Mat (150K)
(100K)
Teacher Model (# vocabs)
Figure 5: Comparison of Performance Improvements Across Different Teachers. VocAgnoLM effectively miti- gates vocabulary mismatch and leverages higher- geghhrrperformig teacher models to achieve significant performance gains, outperforming logit distribution based baselines.
Models. We conduct continual pretraining using TinyL- lama 1.1B (Zhang et al., 2024a), which has vocab- ulary size of 32,000 tokens. To provide teacher guid- ance, utilize 7B-scale, math-specialized teacher models: Llemma (Azerbayev et al., 2024), Mistral-ProXMMath (Zhou et al., 2024), DecpSeekMath (Shao et al., 2024), and Qwen2.5-Math (Yang et al., 2024). Llemma (Azer- bayev et al., 2024) shares the same vocabulary as TinyL- lama (Zhang et al., 2024a), and we use it as a teacher model evaluate the impact of using a same vocabulary. Details of teacher models are provided in Appendix A. Table 4 reports the performance of the teacher models.
the logit distribution, VocAgnoLM demonstrates superior performance. This difference becomes more pronounced when using a stronger teacher model, Llemma (Azer- bayev et al., 2024), compared to weaker teacher model Llama-CTTPP By focusing solely on teacher guidance under the same aligned token sequence, our teacher loss- based guidance effectively facilitates student model training.
Comparison with ULD (Different Vocabulary). The limitations of probabilistic distribution-basee distillation be- come more pronounced when using teacher models with dif- ferent vocabularies. As shown in Figure 4, VocAgnol sig- nificantly outperforms ULD (Boizard et al., 2025), with Ta- ble highlighting a substantial 33% performance gap when using Owen2.5-Math-Inee teacher model. In the pretraining stage, where a large vol-
5. Experiments
5.1. Main Results
Comparision with KLD (Same Vocabulary). As shown
(Yang et al., 2024) as the
Table 1, compared to the KLD approach that fully utilizes
5

---

Overcoming Vocabulary Mismatch: aocasilary-agnostcc Teacher Guided Language Modeling
Table I: Performance Comparison of Student Model (S) Guided by Various Teacher Models, Average scores for comparison with the Rho-1, I, following Lin et al. (2024) setup. Since SAT consists of only 32 multiple-choicn questions, we report AVG score without SAT to account for abnormal cases. The best results are in bold, while second-best ones are underlined.
MMLU STEM
AVG!
AVG
Model
Method GSM8K MATH SVAMP ASDiv MAWPS
TAB
MQA
SAT
AVG
(w/o ")
(AT)
TinyLlama (S)
2.7
10.9
17.9
20.5
12.5
13.9
16,4
21.9
13.3
11.5
12.2
TinyLlama-CPT
6.8
4.2
22
36.4
47.1
16.5
12.3
23.2
15.6
20.5
21.5
21.1
Teacher w/ Same Vocabulary
Rho-1
7.1
23.5
41.2
53.8
18
24.8
S TinyL Jama-CPT
KLD
6.8
5.6
22.7
37.1
49.7
17.9
12.1
23.5
15.6
21.2
22.3
21.9
S Sinyllama-CPP
Ours
7.4
4.6
21.7
37.7
48.0
16.7
13.0
22.5
25.0
21.8
22.1
21.5
S + Llemma
KLD
6.9
4.2
23.3
37.7
49.9
17.2
12.7
21.9
18.8
21.4
22.5
21.7
S Llemma
Ours
8.1
5.2
21.9
38.1
50.1
21.0
13.9
24.0
34.4
24.1
22.9
22.8
Teacher w/ Different Vocabulary
ULD
6.0
5.4
20.9
36.4
46.7
16.7
11.2
21.1
31.2
21.7
21.1
20.6
S Mistral-Pro PraMMahh
Ours
8.6
6.2
22.6
39.5
51.2
21.7
17.3
25.6
25.0
24.2
24.2
24.1
ULD
6.3
4.8
22.4
36.8
46.0
16.6
12.2
22.4
31,2
22.1
21.4
20.9
S MeepSeekMath
Ours
9.5
6.2
23.1
41.6
53.3
22.6
15.9
25.6
18.8
24.1
24.9
24.7
ULD
5.8
3.6
21.3
36.1
47.1
18.0
11.7
22,4
34.4
22.3
20.9
20.8
S Qwen2.5-Math
Ours
9.9
5.4
25.6
42.2
54.1
20.8
17.4
26.9
31.2
25.9
25.8
25.3
ULD
6.7
4.6
20.8
36.1
45.8
17.9
11.2
19.5
31.2
21.5
20.9
20.3
S DeepSeekKMth-RL
Ours
10.8
7.2
27.3
45.9
59.6
22.6
19.1
28.1
21.9
26.9
28.3
27.6
ULD
6.7
4.6
22.6
36.8
46.9
17.3
13.2
22.4
31.2
22.4
21.8
21.3
S + -wwen.55Math--mm
Ours
11.3
7.6
28.9
46.8
60.7
22.5
20.5
30.3
40.6
29.9
29.3
28.6
ume of tokens are processed, the impact of vocabulary mis- alignment becomes increasingly pronounced. This high- lights the inherent limitations of probabilistic distance-based logit alignment while emphasizing the necessity and effec- tiveness of fine-grained, token-level alignment approach.
6. Analysis
6.1. Importance of fine-grained sequence alignment
In this section, we extend the preliminary study presented in Section 2 to analyze the significance of Token-level Lexi- cal Alignment's 's' fine-grained, token-level mapping. Follow- ing the Section 2, we compare the effectiveness of teacher guidance under two alignment strategies: coarse-grained alignment using chunking and fine- aue-ggriiedd alignment by Token-level Lexical Alignment. For this analysis, we em- ploy the teacher and train on 5B tokens from OpenWebMath (Paster et al., 2024), applying the same teacher loss-based guidance detailed in Section 3.2.
5.2. Scalability with Different Teachers.
As illustrated in Figure 5, VocAgnoLM demonstrates consistent performance improvements when leveraging stronger teacher models, showing clear correlation with teacher quality. Compared to ULD (Boizard et al., 2025), VocAgnoLM achieves significantly greater performance gains and effectively follows the performance trends of the teacher model. Notably, even though the strongest teacher model, 10wen2.5-Math-IInniie
7B (Zhou et al., 2024) as the
(Yang et al., 2024), has the lowest vocabulary overlap ratio (6.32% Figure 1) with the student vocabulary, VocAgnoLM still effectively transfers more knowledge and achieves superior performance over logit Distribution-basee guidance. Additionally, when using DeepSeekMath (Shao et al., 2024) as the teacher model, VocAgnoLM demonstrates competitive performance against Rho-1 (Lin et al., 2024). Further mapping and guidance ex- amples across different teacher models provided Ap- pendix C.
In Table we observe performance changes by increasing the number of chunks from to 64. As shown in Fig- ure the performance improves the chunks become more fine- ainengranedd at first. However, when the number of chunks reaches 64, performance begins to degrade, and be- comes even worse than the results where chunks are selected randomly instead of using teacher guidance. According to the preliminary study, we measure the character-level IoU and loS between teacher and student chunks. We find that the number of chunks increases from 32 to 64, IoU de- creases sharply, and this drop corresponds to the significant
6

---

Overcoming Vocabulary Mismatch; agcnoiiary-agnostcc Teacher Guided Language Modeling
Table 2: Performance Comparison of Chunking and Token-level Lexical Alignment.
Num Chunk
MMLU STEM
AVG
AVG
Guidance
GSM8K MATH SVAMP ASDiv MAWPS TAB" MQA
SAT*
AVG
(W/o ")
(wie SAT)
Chunking Aligument
Random
8
3.6
3.2
19
30.3
39.5
15.7
10.6
20
25
18.5
17.7
17.7
Teacher
4.9
3.4
18.9
29.7
41.3
16.5
11.5
19.5
21.9
18.6
18.3
18.2
Random
16
3.6
2.4
18.3
29.7
39.0
16.4
11.0
19.3
25.0
18.3
17.3
17.5
Teacher
5.1
3.8
18.8
30.7
40.8
17.7
10.5
20.0
21.9
18.8
18.3
18.4
Random
32
4.2
2.6
19.5
30.2
39.6
16.6
11.5
19.1
25.0
18.7
17.9
17.9
Teacher
5.2
4.0
18.5
30.3
41.6
17.2
11.3
20.3
21.9
18.9
18.5
18.6
Random
64
3.9
3.0
19.0
29.7
38.5
16.0
11.5
19.1
25.0
18.4
17.6
17.6
Teacher
4.2
3.6
17.7
29.7
38.5
17.0
11.4
20.3
17.9
17.5
17.8
Token-level Lexical Alignment
Teacher
5.3
5.4
18.0
30.8
42.8
17.1
12.4
21.8
28.1
20.2
19.1
19.2
Chunking Aligning w/ Mistral-ProXMath
ity in how they should be handled, as they do not receive any guidance from the teacher model. To address this, we compare three strategies: Mean, Exclude, and Include.
20.5
1.00
20.0
0.95
0.90
The Mean strategy average all teacher token losses within batch maintain consistent guidance scale in Equation (4), The Exclude strategy discards all unmapped student tokens, assuming they lack semantic information. In contrast, the Include strategy trains all unmapped tokens, expecting the role of special tokens.
Performance
19.0
0.85
Over
0.80
18.5
0.75
As shown in Table 3, the Include strategy achieve the best performance. Notably, the Exclude strategy results in sig- nificant performance degradation, highlighting the critical role of special tokens in continual pretraining. This suggests that unmapped tokens, such as start and end tokens, have already been trained as critical elements during pretrain- ing, providing explicit signals to effectively interpret large amounts of text the corpus (Devlin et al., 2019; Newman et al., 2020; Yue et al., 2024).
17.5
32
64
Token-loval
Chunks
Random
Teacher
loU
lo5
Figure 6: Correlation Between Performance and Sequence Aligmment. Chunking alignment initially improves perfor- mance as the granularity increases, but performance sharply declines when the overlap (Char-level loU and IoS) de- creases significantly. Token-level Lexical Alignment main- tains high loS and achieves superior performance with fine- grained alignment.
6.3. How to aggregate multiple-mapped teacher tokens?
In Equation (2), various aggregation functions (.) can be considered handle the mapping of multiple teacher tokens to single student token. Intuitively, student tokens mapped to teacher tokens exhibiting high loss values are discarded. To effectively filter out abnormal teacher tokens, we evaluate two aggregation strategies: Max and Mean. As discussed in Section 6.2, the Include strategy is consistently applied for unmapped student tokens in these experiments.
performance degradation. In contrast, Token-level Lexical Alignment demonstrates superior performance with higher IoU. Due intrinsic token differences between the teacher and student at fine- fine-grained granularity, further compare character-level Intersection over Student (IoS). Token-level Lexical Alignment achieves 100% overlap. indicating that all student tokens are covered by teacher tokens, ensuring precise fineinnineed teacher guidance.
As shown in Table 3, when trained on dataset 2B tokens, the performance difference between the strategies was minimal. However, when trained for longer duration on 15B tokens. the Max strategy outperformed Mean, resulting in an approximately 1.3% improvement in AVG (wlo SAT),
6.2. How to deal with unmapped student tokens?
For student tokens that are not lexically mapped to any teacher tokens (e.g., special tokens), there exists ambigu-

---

Overcoming Vocabulary Mismatch; aocaoulary-agnostcc Teacher Guided Language Modeling
Table 3: Performance Comparison by Unmapped and Multi-Mapped Token Strategies.
Train Tokens
MMLU STEM
AVG AVG
Function
GSM8K MATH SVAMP ASDiv MAWPS TAB MQA
SAT
AVG
(w/o *)
(win
Unmapped Student Tokens Strategy
Mean
2B
3.7
4.6
16.2
24.5
32.2
14.6
10.1
15,8
25.0
16.3
15.2
15.2
Exclude
2B
1.4
3.8
2.8
7.6
9.3
4.1
4.0
15.6
12.5
6.8
4.8
6.1
Include
2B
3.6
3.6
18.2
25.7
35.9
14.1
12.7
16.5
18.8
16.6
16.6
16.3
Multi Multi-mapped Teacher Tokens Aggregation
Include+Mean
2B
3.6
3.6
18.2
25.7
35.9
14.1
12.7
16.5
18.8
16.6
16.6
16.3
Maxlude+Max
2B
3.6
4.0
17
26.2
35.9
14.9
12.5
16.0
18.8
16.5
16.5
16.3
Include+Mean
15B
8.3
4.6
24.0
40.2
53.9
20.8
13.0
23.5
28.1
24.3
24.0
23.8
Include+ Max
15B
8.6
6.2
22.6
39.5
51.2
21.7
17.3
25.6
25.0
24.2
24.2
24.1
7. Related Works
vocabulary distillation through an offline strategy based on difference sampling. In contrast, VocAgnoLM supports both offiin and online strategy, offering greater flexibility the choice of pretraining strategies. Moreover, while MiniPLM performs distillation the instance-level, VocAgnoLM fo- cuses token-level, allowing for more fine- fine-grained guid- ance.
7.1. Cross Vocabulary Alignment
Cross-vocabulary alignment has emerged as a critical re- search area due to tokenization discrepancies between dif- ferent LLMs. This alignment has been applied to various downstream tasks, such as knowledge distillation (Boizard et al., 2025; Zhang et al., 2024b; Cui et al., 2025), model ensemble (Xu al., 2024; Huang et al., 2024; Yu et al., 2024), cross-lingual transfer (Dobler & De Melo, 2023). Previous studies primarily rely matching-basee -baeed meth- ods (Fu et al., 2023; Wan et al., 2024), optimal transport (Boizard et al., 2025; Cui et al., 2025). vocabulary mapping matrices (Xu et al., 2024; Huang et al., 2024; Yu et al., 2024) or cross-model attention mechanism (Zhang et al., 2024b). While these approaches have shown effectiveness specific contexts, they often lack generality when applied to causal language modeling, In contrast, propose Token-level Lexical Alignment, simple yet effective method that lever- ages the guidance of teacher models to enhance general causal language modeling. Furthermore, inspired by Xie et al. (2024); Xu et al. (2025), we conduct comprehensive study on the effectiveness of alignment at various levels of granularity for language modeling.
Hard label Guidance. Another guidance training the student model using text generated by teacher model (a.k.a hard labels). (Kim & Rush, 2016; Hsieh et al., 2023; Peng et al., 2023; Zhou et al., 2024; Maini et al., 2024; Gunasekar et al., 2023). However, this method involves substantial overhead, as it requires constructing new training corpora for pretraining. An alternative strategy leveraging teacher models for data sample selection (Albalak et al., 2024), using heuristic filtering, classifier or perplexity- tering, domain specific filtering, deduplication. Irreducible Curriculum (Fan & Jaggi, 2023) extends this idea through batch selection methods for pretraining. However, these coarse -raiined approach lack the granularity required for more precise guidance. Rho-1 (Lin et al., 2024) addresses fine-grained token-level data selection. Nevertheless, the requirement of same teacher and student vocabulary limits the capability to transfer knowledge from various teacher models with different vocabulary. In this work, we propose novel method that enables token-level guidance without requiring the creation of new corpora or identical vocabu- laries between teacher and student models. By facilitating token-level alignment with any pre-trained or newly released model, our approach significantly enhances the flexibility and applicability of teacher-guide guided pretraining.
fil-
7.2. Teacher Guided Language Modeling
Soft label Guidance. Soft label guidance, a.k.a knowl- edge distillation, relies on large teacher model to transfer its logits distribution proach is use the logits distribution of teacher model to guide the student model through well-designed optimization objectives (Hinton et al., 2015; Gu et al., 2024a; Agarwal et al., 2024; Boizard et al., 2025), particularly for specific downstream tasks. However, these approaches typically require the the teacher and student models to share the same vocabulary, which limits their applicability when trans- ferring knowledge across models with different tokeniza- tion schemes. MiniPLM (Gu et al., 2024b) enables cross-
student model. A common ap-
8. Conclusion
In this work, we propose Vocabulary-agnostic Teacher Guided Language Modeling (VocAgnoL.M), method for training student models with strong teacher models re- gardless of vocabulary differences. We identify two key
8

---

Overcoming Vocabulary Mismatch: aocasulary-agnostic Teacher Guided Language Modeling
challenges: token sequence mismaich and logit distribu- tion divergence, and introduce Token-level Lexical Align- ment along with teacher loss-based guidance to address these issues. Our results demonstrate that student perfor- mance improves in proportion to the teacher model's ca- pabilities, effectively overcoming vocabulary mismatches. Furthermore, we highlight the significance of token align- ment by analyzing the impact of sequence misalignment caused by differences in granularity. Our findings suggest that precise token correspondence plays crucial role in teacher guidance, providing insights for future research on the effective utilization of teacher models in vocabulary- agnostic setting.
H., et al. A survey on data selection for language models. arXiv, 2024.
Amini, A., Gabriel, S., Lin, S., KodzissKa R., Choi, Y., and Hajishirzi, H. MathQA: Towards interpretable math word problem solving with operation based for- malisms. In NAACI, 2019.
Azerbayev, Z.,, Schoelkopf, H., Paster, K., Santos, M. D., McAleer, S. M., Jiang, A. Q., Deng, J., Biderman, S., and Welleck, S. Llemma: An open language model for mathematics. In ICIR, 2024.
Boizard, N., Haddad, K. E.,, HUDELOT, C., and Colombo, P. Towards cross-tokenizer distillation: the universal logit distillation loss for LLMs. TMLR, 2025.
Limitations
While we demonstrate VocAgnoLM TinyLlama 1.1B (Zhang et al., 2024a) using the 15B OpenWeb- Math (Paster et al., 2024), VocAgnoLM is designed to be broadly applicable across different models and datasets. Due to computational constraints, we present a case study demonstrating its effectiveness in continual pretraining on mathematical domain corpus, leaving further large-scale validation for future work.
Cobbe, K., Kosaraju, V., Bavarian, M., Hilton, J., Nakano, R., Hesse, C.,, and Schulman, J. Training verifiers to solve math word problems. arXiv, 2021.
Cui, X., Zhu, M., Qin, Y., Xie, L., Zhou, W., and Li, H. Multi-level optimal transport for universal cross-tokenizer knowledge distillation language models. In AAAI, 2025.
DeepSeek-AL. :, Bi, X., Chen, D., Chen, G., Chen, S., Dai, D., Deng, C., Ding, H., Dong, K., Du Q., Fu, Z., Gao, H., Gao, K., Gao, W., Ge, R., Guan, K., Guo, D., Guo, J., Hao, G., Hao, Z.., He, Y., Hu, W., Huang, P., Li, E., Li, G., Li, J., Li, Y., Li, Y. K., Liang, W., Lin, F., Liu, A. X., Liu, B., Liu, W., Liu, X., Liu, X., Liu, Y., Lu, H., Lu, S., Luo, F., Ma, S., Nic, X., Pei. T., Piao, Y., Qiu, J., Qu, H., Ren, T., Ren, Z., Ruan, C., Sha, Z., Shao, Z., Song, J., Su, X., Sun, J., Sun, Y., Tang, M., Wang, B., Wang. P., Wang, S., Wang, Y., Wang, Y., Wu, T., Wu. Y., Xie, X., Xie, Z., Xie, Z., Xiong, Y., Xu, H., Xu, R. X., Xu, Y., Yang, D.. You, Y., Yu, S., Yu, X., Zhang, B., Zhang, H., Zhang, L., Zhang. I.., 7hhnng M., Zhang, M., Zhang, W., Zhang, Y., Zhao, C., Zhao, Y., Zhou, S., Zhou, S., Zhu, Q. and Zou, Y. Deepseek Ilm: Scaling open-source language models with longtermism. arXiv, 2024.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning by improving pretraining for student teacher model. Our publicly available
models using a vocabulary-agnosee study is conducted on OpenWeeMatte dataset, and focuses solely on mathematical reasoning. We do not foresee significant ethical concerns but acknowledge that language models may still inherit biases from training data, which should be considered in broader applications.
Acknowledgements
This research was supported by the MSIT (Ministry of Sci- ence, ICT), Korea, under the Global Research Support Pro- gram in the Digital Field program (RS-2024-004366 supervised by the IITP (Institute for Information & Com- munications Technology Planming& & Evaluation). And, this project was also supported by Microsoft Research Asia.
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of deep bidirectional transformers for lan- guage understanding. In NAACL, 2019.
Dobler, K. and De Melo, G. Focus: Effective embedding ini- tialization for monolingual specialization of multilingual models. In EMNLP, 2023.
References
Agarwal, R., Vieillard, N., Zhou, Y., Stanczyk. P., Ramos, S., Geist, M., and Bachem, O. On-policy distillation of language models: Learning from self-generated mistakes. In ICLR, 2024.
Fan, S. and Jaggi, M. Irreducible curriculum for language model pretraining. arXiv, 2023.
Fu, Y., Peng, H., Ou, L., Sabharwal, A., and Khot, T. Spe- cializing smaller language models towards multi-step rea- soning. In ICMI., 2023.
Albalak, A., Elazar, Y., Xie, S. M., Longpre, S., Lambert, N., Wang, X., Muennighoff, N., Hou, B., Pan, L., Jeong,

---

Overcoming Vocabulary Mismatch; Yocabulary-agnoste Teacher Guided Language Modeling
Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang. Y., Callan, J., and Neubig. G. Pal: Program-aided language models. arXiv, 2022.
Gu, Y., Zhou, H., Meng, F,, Zhou, J., and Huang, M. Mini- plm: Knowledge distillation for pre-training language models. arXiv, 2024b.
Gunasekar, S., Zhang, Y., Aneja, J., Mendes, C. T., Giorno, A. D., Gopi, S., Javaheripi, M., Kauffmann, P., de Rosa, G., Saarikivi, O., Salim, A., Shah, S., Behl, H. S., Wang, X., Bubeck, S., Eldan, R., Kalai, A. T., Lee, Y. T., and Li, Y. Textbooks are all you need, arXiv, 2023.
Gemma-Tean Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahriari, B., Ramé, A., Ferret, J., Liu, P., Tafti, P. Friesen, A., Casbon, M., Ramos, S., Kumar, R., Lan, C. L., Jerome, S., Tsitsulin, A., Vieillard, N., Stanczyk, P., Girgin, S., Momchev, N., Hoffman, M., Thakoor, S., Grill, J.-B., Neyshabur, B., Bachem, O., Walton, A., Severyn, A., Par- rish, A., Ahmad, A., Hutchison, A., Abdagic, A., Carl, A., Shen, A., Brock, A., Coenen, A., Laforge. A., Pater- son, A., Bastian, B., Piot, B., Wu, B., Royal, B., Chen, C., Kumar, C., Perry, C., Welty, C., Proquette-Choo C. A., Sinopalnikov, D., Weinberger, D., Vijaykumar, D., Rogozińska, D., Herbison, D., Bandy, E., Wang. E., Noland, E., Moreira, E.., Senter, E., Eltyshev, E., Visin, F., Rasskin, G., Wei, G., Cameron, G., Martins, G., Hashemi, H., Klimezak-Plucinska,, H., Batra, H., Dhand, H., Nar- dini, I., Mein, J., Zhou, Svensson, J., Stanway, J., Chan, J.. Zhou, J. P., Carrasqueira, J., Iljazi, J., Becker, J., Fer- nande7, J., van Amersfoort, J., Gordon, J., Lipschultz, J., Newlan, J., yeong Ji, J., Mohamed, K., Badola, K., Black, K. Millican, K., McDonell, K., Nguyen, K., Sodhia, K., Greene, K., Sjoesund. L. L., Usui, L., Sifree L., Heuer- mann, L., Lago, L., McNealus, L.., Soares, L. B., Kil- patrick, L., Dixon, L., Martins, L., Reid, M., Singh, M., Iverson, M., Görner, M.. Velloso, M., Wirth, M., Davi- dow, M., Miller, M., Rahtz, M. Watson, M., Risdal, M., Kazemi, M., Moynihan, M., Zhang, M., Kahng, M., Park, M., Rahman, M., Khatwani, M., Dao, N., Bardoliwalla, N., Devanathan, N., Dumai, N., Chauhan, N., Wahltinez, O., Botarda, P., Barnes, P., Barham, P., Michel, P., Jin, P. Georgiev, P., Culliton, P., Kuppala, P., Comanescu, R., Merhej, R., Jana, R., Rokni, R. A., Agarwal, R., Mullins, R., Saadat, S., Carthy, S. M., Cogan, S., Perrin, S., Amold, S. M. R., Krause, S., Dai, S., Garg, S., Sheth, S., Ronstrom, S., Chan, S., Jordan, T.. Yu. T., Eccles, T., Hennigan, T., Kocisky, T., Doshi, T., Jain, V., Yadav, V., Meshram, V., Dharmadhikari, V., Barkley, W., Wei, W., Ye. W. Han, W., Kwon, W., Xu, X., Shen, Z., Gong, Z., Wei, Z., Cotruta, V., Kirk, P., Rao, A., Giang, M., Peran, L., Warkentin, T., Collins, E., Barral, J., Ghahramani, Z. Hadsell, R., Sculley, D., Banks, J., Dragan, A., Petrov, S., Vinyals, O., Dean, J., Hassabis, D., Kavukcuoglu, K., Farabet, C., Buchatskaya, E., Borgeaud, S., Fiedel, N., Joulin, A., Kenealy, K., Dadashi, R., and Andreev,
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song. D., and Steinhardt, J. Measuring massive multitask language understanding. In ICLR, 2021a.
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E.,, Song, D., and Steinhardt, J. Measuring mathematical problem solving with the math dataset. In NeurIPS, 2021b.
Hinton, G. E., Vinyals, O., and Dean, J. Distilling the knowledge in a neural network. arXiv, 2015.
Hsich, C.-Y., Li, C.-L., Ych, C.-k., Nakhost, H., Fujii, Y., Ratner, A., Krishna, R., Lee, C.-Y., and Pfister, T. Distill- ing step-by-step! outperforming larger language models with less training data and smaller model sizes. In Find- ings of ACL. July 2023.
Huang, Y., Feng, X., Li, B., Xiang, Y., Wang. H., Qin, B., and Liu, T. Enabling ensemble learning for heterogeneous large language models with deep parallel collaboration. arXiv, 2024.
Jiang, A. Q.. Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M.- A., Stock, P., Scao, T. L., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mistral 7b. arXiv, 2023.
Kim, Y. and Rush, A. M. Sequence-level knowledge distil- lation. In EMNLP, 2016.
Konzel-Kedziorsii R., Roy, S., Amini, A., Kushman, N., and Hajishirzi, H. MAWPS: A math word problem repos- itory. In NAACL, 2016.
Lightning-Al. Lightning Baaiing-AAIIIS1 AI/litgpt, 2023.
Litgpt. 2023.
Lin, Z., Gou, Z., Gong, Y., Liu, X., Shen, Y., Xu, R., Lin, C., Yang, Y., Jiao, J., Duan, N., and Chen, W. Rho-1: Not all tokens are what you need. In NeurIPS, 2024.
A. Gemma 2: Improving open language models at a practical size. arXiv, 2024.
Lu, P., Qiu, L., Chang, K.-W., WU, Y. N., Zhu, S.-C., Ra- jpurohit, T., Clark, P., and Kalyan, A. Dynamic prompt learning via policy gradient for cemi-structured mathe- matical reasoning. In ICLR, 2023.
Gu, Y., Dong, L., Wei, F., and Huang, M. Minillm: Knowl- edge distillation of large language models. In ICLR, 2024a.
10

---

Overcoming Vocabulary Mismatch: agenoic Teacher Guided Language Modeling
Maini, P., Seto, S., Bai, H., Grangier, D., Zhang, Y., and Jaitly, N. Rephrasing the weh: A recipe for compute and data-efficient language modeling. In ACL, 2024.
Lachaux, M.-A., Lavril, T., Lee, J., Liskovich, D., Lu, Y.,
Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subramanian, R., Tan, X. E.,, Tang, B., Taylor, R., Williams, A., Kuan, X., Xu, P., Yan, Z., Zarov, L., Zhang, Y., Fan, A., Kambadur, M.. Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T. Llama 2: Open foundation and fine-tuned chat models. arXiv, 2023.
Meta. Llama 3.2 model card, 2024.
URL
https: /ttpp:://gttubb com/meta-1lame 11aam-mmdes /bl ob/mat/./..ooo...oo n n/mode s 1/l1ammaaa ama3 SROODE__CARE md.
Miao, S.-Y., Liang, C.-C., and Su, K.-Y. A diverse corpus for evaluating and developing english math word problem solvers. In ACL, 2020.
Wan, F., Huang, X., Cai, D., Quan, X., Bi, W., and Shi, S. Knowledge fusion of large language models. In ICLR, 2024.
Muralidharan, S., Sreenivas, S. T., Joshi, R., Chochowski, M., Patwary, M., Shoeybi, M., Catanzaro, B., Kautz, J., and Molchanov, P. Compact language models via pruning and knowledge distillation. arXiv, 2024.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. and Zhou, D. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS, 2022.
Newman, B., Hewitt, J., Liang. P., and Manning, C. D. The EOS decision and length extrapolation. In BlackboxNLP Workshop, 2020.
Xie, J., Cheng, P., Liang, X., Dai, Y., and Du, N. Chunk, align, select: A simple seggsenuence processing method for transformers. In ACI. 2024.
Paster, K., Santos, M. D., Azerbayev, Z., and Ba, J. Open- webmath: An open dataset of high-quality mathematical web text. In ICLR, 2024.
Xu, Y., Lu, J., and Zhang, J. Bridging the gap between different vocabularies for I1m ensemble. In NAACL, 2024.
Xu, Y., Chen, J., Wu, J., and Zhang, J. Hit the sweet spot! span-level ensemble for large language models. COL- ING, 2025.
Patel, A., Bhattamishra, S., and Goyal, N. Are nlp models really able to solve simple math word problems? In NAACL, 2021.
Yang. A., Zhang, B., Hui, B., Gao, B., Yu, B., Li, C., Liu, D., Tu, J., Zhou, J., Lin, J., Lu, K., Xue, M., Lin, R., Liu, T., Ren, X., and Zhang, Z. Qwen2.5-matt technical report: Toward mathematical expert model via self self-improvement. arXiv, 2024.
Peng, B., Li, C., He, P., Galley, M., and Gao, J. Instruction tuning with gpt-4. arxiv, 2023.
Qwen, :, Yang. A., Yang, B., Zhang. B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., Lin, H., Yang, J. Tu, J., Zhang, J., Yang, J., Yang. J., Zhou, J., Lin, J., Dang, K., Lu, K., Bao, K., Yang, K., Yu, L., Li, M., Xue, M., Zhang, P. Zhu, Q. Men, R., Lin, R., Li, T., Tang, T., Xia, T., Ren, X., Ren, X., Fan, Y., Su, Y., Zhang, Y., Wan, Y., Liu, Y., Cui, Z., Zhang, Z., and Qiu, Z. Qwen2.5 technical report. arXiv, 2024.
Yu, L., Jiang, W., Shi, H., Yu, J., Liu, Z., Zhang, Y., Kwok, J. T., Li, Z.,, Weller, A., and Liu, W. Metamath: Boot- strap your own mathematical questions for large language models. arXiv, 2023.
Yu, Y.-C., Kuo, C.-C., Ye, Z., Chang, Y.-C., and Li, Y.-S. Breaking the ceiling of the Ilm community by treating token generation as classification for ensembling. In Findings of EMNLP, 2024.
Shao, Z., Wang, P. Zhu, Q., Xu, R., Song. J., Bi, X., Zhang, H., Zhang, M., Li, Y. K., Wu, Y., and Guo, D. Deepseek- math: Pushing the limits of mathematical reasoning in open language models. arXiv, 2024.
Yue, Z., Zhang, L., and Jin, Q. Less is more: Mitigating mul- timodal hallucination from an EOS decision perspective. In ACL, 2024.
Song, K., Sun, H., Tan, X., Qin, T., Lu, J., Liu, H., and Liu, T.-Y. Lightpaff: A two-stage distillation framework for pre-training and fine-tuning, arXiv, 2020.
Zhang, P., Zeng, G., Wang, T., and Lu, W. Tinyllama: An open-source small language model. arXiv, 2024a.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, I.., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V. Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S.,
Zhang, S., Zhang, X., Sun, Z., Chen, Y., and Xu, J. Dual- space knowledge distillation for large language models. In EMNLP, 2024b.
Zhou, F,, Wang, Z., Liu, Q., Li, J., and Liu, P. Program- ming every example: Lifting pre-training data quality like experts at scale, arXiv, 2024.

---

Overcoming Vocabulary Mismatch; aocasilary-agnostcc Teacher Guided Language Modeling
Table 4: Performance of Teachhr Models on Math Evaluation Suite.
Num. Vocabs
MMLU STEM
Teacher Model
GSMSK MATH SVAMP
ASDiv MAWPS
TAB MQA
SAT
AVG
Llemma
32K
38.8
17.2
56.1
69.1
82.4
48.7
41.0
45.4
59,4
50.9
Protral
32K
51.0
22.4
64.9
72.9
89.2
49.8
53.0
54.2
75.0
59.2
DeepSeekMath
100K
64.1
34.2
74.0
83.9
92.4
63.4
62.4
56.4
84.4
68.4
DeepSeekMath--R
100K
86.2
50.2
87.7
91.1
96.6
64.9
56.9
24.5
15.6
63.7
Qwen2.5-Me M-tth
150K
85.8
57.4
88.2
91.7
96.3
67.3
75,9
69.4
93.8
80.6
Qwen2.55-Math Intttrruut
150K
88.3
74.0
90.3
90.8
93.8
81.6
81.0
65.2
90.6
84.0
A. Teacher Model Details
Vocabulary Details. The tokenization schemes of the teacher models vary significantly. Mistral-ProXMatth (Zhou et al.,
2024) adopts Byte Pair Encoding (BPE) with vocabulary size 32,000 tokens. DeepSeekMath (Shao al., 2024) employs Byte-level Byte Pair Encoding (BBPE) with larger vocabulary size of 100,000 tokens, while Qwen2.5-Math (Yang et al., 2024) utilizes BPE with the largest vocabulary size of 150,000 tokens among the teacher models. These variations highlight the diversity of vocabularies for each teacher models.
Teacher Performances. Table 4, we report the performance of teacher models compare the impact of teacher guidance based their capabilities.
B. Further Implementation Details
KLD. Following the setup outlined by Song et al. (2020) and Gu et al. (2024a), we combine the KI--Divergence loss calculated between the teacher and student output distribution (pf,p?) p!') with the cross-entropy loss the student model, using the same weighting ratio.
1 Σ
( (x))
log P(x: x2
KL (p.
(5)
ULD. Boizard et al. (2025) proposes Universal Logit Distance (ULD) loss, shown in Equation (6), measure the logit distribution distance between teacher and student with different vocabularies. ULD utilizes Wasserstein distance (WD) to calculate the distance of teacher and student output distribution (pf, pf') While Boizard et al. (2025) primarily focus downstream tasks, we explore the hyperparameter values [0.3, 0.5, 1.0, 1.5] to determine those most suitable for continual pretraining on WebMath (Paster et al., 2024). Based on the results Figure 7b, we adopt = 0.5 in this study. For the further details, please refer to Boizard et al. (2025).
S
CULD (x),,)²)))))
log P(x x + A
wD(p: 1Pp))
(6)
1=
Llemma. Llemma (Azerbayev et al., 2024) adopts the Llama-2 (Touvron et al., 2023) vocabulary, similar to TinyL- lama (Zhang et al., 2024a), with the addition of 16 noise tokens. However, we observe that the cumulative probability of these tokens is overally negligible, below x 10-7, Consequently, this study disregards the probabilities noise token and considers Llemma (Azerbayev al., 2024) as teacher model that shares the same vocabulary as TinyLlama (Zhang et al., 2024a).
Threshold. We explore the impact of the top-k threshold on a 2B-token subset of the corpus, evaluating thresholds ranging from 40% to 80%, following the settings in Lin et al. (2024). Based on the results in Figure 7a, we select 40% threshold for our experiments.
12

---

Overcoming Vocabulary Mismatch; aocaoulary-agnostcc Teacher Guided Language Modeling
20
Downz S-tth
DrepSeekMarh
22.0
Qwen2.5 M2th
19
Proxtttth
21.3
Protral
Performance
28
Performanc
21.0
20.5
40%
30%
60%
70%
80%
0.3
0.5
15
Threshold
ULD lambda
(a) Performance of student model guided by various teacher models based on different Top-K K thresholds
(b) Performance analysis of X on ULD (Boizard et al., 2025).
Figure 7: Performance Comparison for Hyperparameter Searching.
Mistral-rooMMtth
DeepSeekMatth
Owen2255
The
and
uperator
Fun be
ited
Approxt
imate
Inarginal
iration
unknown
scattering
eigen
values
und
cig
actors
It
decom
boses
The
and
operater
can
priiiied
Appro
mnal
zatioon
unknowel
seallic
ing
eigenvalues
eigen
docom
boses
matris
TinLLLaMA
LLAMA
TlayLLaMA
Figure 8: Examples of teacher guidance demonstrating how teacher models influence the training of Liny (Zhang et al., 2024a). Orange solid arrows represent the guidance provided by the teacher model, while gray dashed arrows indicate tokens that were not included the top-k threshold by teacher guidance.
Table Performance Comparison Using Teacher Models with Comparable Performance: Same vs. Different Vocabularies. tl Average scores for abnormal cases, following the setup in Table
Vocab w/ Student
AVG (Wio
AVG
Model
Method
AVG
Teacher Performance
MetaMath- Llemm-
Same
52.2
62.1
57.2
Different
59.2
58.9
57.2
Student (S) Distilled Performance
+ MetaMath-LLemma
Same
KLD
14.2
14.3
14.4
S + Mistral-ProXMMr ProXMath
Different
Ours
16.6
16.6
16.3
C. Teacher Guidance Examples
Figure 8 illustrates how the teacher model effectively guides important tokens. Even when multiple teacher tokens mapped to single student token (1:N mapping), the teacher provides meaningful alignment. Furthermore, less important tokens (such as and "IIt" in Figure 8), do not pass the top-k threshold, highlighting the effectiveness of the guidance process.
D. Additional Comparison with Other Baselines
Comparable Teacher Models with Same Different Vocabularies. To assess the effectiveness of our approach, conduct an additional experiment on 2B tokens by comparing it against standard distillation setup using two teacher models with comparable performance: KotaMaah-Lleema-- (Yu et al., 2023), which shares the same vocabulary as the student model, and Mistral-Pro Proh-hhh (Zhou et al., 2024), which uses a different vocabulary. As shown in Table 5, VocAgnoLM performs better even when using a teacher model with a different vocabulary, despite both teacher models achieving the same performance on the AVG metric.
13

---

Overcoming Vocabulary Mismatch: aocasilary-agnostic Teacher Guided Language Modeling
Table 6: Comparison of Scaled Continual Pretraining Baseline and VocAgnoLM.
AVG
Model
CPT
13.6
CPT (Ir /= 0.4)
15.3
S Mistral-ProXMar
16.3
S MeepSeekMath
17.3
S Qwen2.5-Math 18.8
Scaled Continual Pretraining Baseline. To evaluate the utility of teacher loss based guidance, we compare our method against continual pretraining baseline (CPT) on 2B tokens, where the training signal is uniformly amplified. Specifically, as we apply top-40% threshold based on teacher guidance (see Appendix B), we rescale the learning rate by factor 0.1 2.5x to match the overall signal strength delivered to the important tokens. As shown in Table 6, our method, using various teacher models, outperforms CPT (Ir/ = 0.4). Although CPT delivers a similar amount of signal strength by increasing the learning rate, it also amplifies the effect of unimportant tokens. These results highlight the effectiveness of our teacher guidance, and provide a way to estimate the impact of various teacher models terms of CPT-equivalent training scale.
of Ir
14


Reviews:
[{"review_id": "5mhKvKRWeu", "reviewer": "Reviewer_HDRn", "summary": "This paper proposes VocAgnoLM, a method to overcome vocabulary mismatch in knowledge distillation for language models. It introduces Token-level Lexical Alignment for precise token mapping and Teacher Guided Loss to adjust training signals. Experiments show up to 46% improvement over baseline methods, enabling effective distillation from stronger teacher models despite vocabulary differences.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's thoughtful feedback and constructive suggestion. We address the points raised below.\n\n### **Impact of top-k threshold (Selected Ratio) on scaling trends.** (Claims and Evidence #1, #2)\n- As discussed in Appendix B and Figure 7a, we explore the effect of the top-k threshold using a 2B-token subset of the corpus, following the experimental setup of Lin et al [2]. Based on this analysis, we adopt a top-k threshold of 40% ( 60% Masking Ratio ) in our study (L253).\n- Consistent with prior work [1,2], we observe that selecting more tokens from a fixed corpus does not necessarily lead to better performance. Importantly, as shown in Figure 7a, the optimal threshold may vary depending on the choice of teacher model. Taking into account both our empirical results in Appendix B and findings from previous study [1,2], we chose the 40% threshold.\n\n### **Insights from established findings, and highlight our representative tasks.** (Claims and Evidence #3)\n- As mentioned in our limitations section, we agree that VocAgnoLM is designed to be broadly applicable across different models and datasets.. Due to our limited pretraining resources, we focused our efforts on the moderately scaled 15B OpenWebMath corpus. **As reviewer bj8e acknowledged, although our work focuses on math/reasoning tasks, we made every effort to evaluate the model through a diverse set of representative tasks to ensure broad applicability within the domain.**\n- As a complementary perspective, we would like to share an intuition that motivated our design choice. **Several prior works [1,2,3] have provided promising evidence that selective token/dataset training may generalize beyond narrow domains, not only on math domain corpus, but also on general-domain corpora such as ThePile, SlimPajama, StarCoderData.** Motivated by these findings, our study aims to validate whether the proposed method can effectively follow the teacher model’s expertise within a domain, using a moderately scaled math-domain web corpus.\n- We hope that these prior findings, along with our own results (e.g., Figure 5, which shows performance improvement using an instruction-tuned teacher model) can help estimate VocagnoLM’s potential for general-domain performance. We consider extending our experiments to broader general-domain corpora a valuable direction for future work.\n\n[1] *Mindermann et al., Prioritized Training on Points that are learnable, Worth Learning, and Not Yet Learnt, In ICML 2022.*\n\n[2] *Lin et al., RHO-1: Not All Tokens Are What You Need, In NeurIPS 2024.*\n\n[3] *Xie et al., Data Selection for Language Models via Importance Resampling, In NeurIPS 2023.*\n\n\n### **Comparison with scaled CPT baseline.** (Experimental Designs and Analyses)\nThank you for the insightful suggestion. Following your suggestion, we conducted an additional experiment on 2B tokens.\n\nIn VocAgnoLM, we apply a top 40% threshold based on teacher guidance. To match the signal strength of important tokens, we rescaled learning rate by a factor of lr / 0.4 = 2.5x.\n\nAs shown in the table below, our method, using various teacher models, outperforms CPT (lr /= 0.4). Although CPT delivers a similar amount of signal strength by increasing the learning rate, it also amplifies the effect of unimportant tokens. These results support the effectiveness of our teacher guidance, and allow us to estimate the impact of various teacher models in terms of CPT-equivalent training scale.\n\n| Setting                 |  AVG (w/o SAT) |\n|------------------------|----------------|\n| CPT                    | 13.6       |\n| CPT  (lr /= 0.4)    |  15.3           |\n| S + Mistral-ProXMath   | 16.3           |\n| S + DeepSeekMath       | 17.3           |\n| S + Qwen2.5-Math       | 18.8           |\n\n\n### **Difference with MiniPLM.** (Questions for Authors)\nThank you for pointing this out. We noticed that the order of the last two sentences in Section 7.2 may have been mistakenly switched. MiniPLM enables cross-vocabulary distillation through an offline KD strategy. However, our method supports **both online and offline KD**, offering more flexibility in the choice of pretraining strategies. Additionally, while MiniPLM performs distillation at the instance-level, our approach focuses on **token-level**, which marks a key difference. We will revise the sentence accordingly and describe this difference in the final version.", "claims_and_evidence": "In Equation (4), certain tokens are masked during pretraining with the proposed VocAgnoLM method, potentially reducing the effective token count. Given that current scaling laws [1,2] primarily examine the relationship between performance and training computation, this masking mechanism may inherently introduce a scaling disadvantage. To better understand this limitation, more empirical analysis is encouraged, including:  \n\n1. Masking Ratios: What are the token masking ratios across different experimental settings in the paper?  \n2. Scaling Trends: How does the scaling behavior of VocAgnoLM compare to baseline methods in terms of training computation?  \n3. Generalization to Broader Domains: How does the method perform on general-domain tasks, where more critical tokens may exist beyond the math-focused setting?\n\n[1] Scaling Laws for Neural Language Models.\n[2] Training Compute-Optimal Large Language Models.", "methods_and_evaluation_criteria": "N/A", "theoretical_claims": "There is no theoretical claim proposed in the paper.", "experimental_designs_or_analyses": "From Equation (4), the proposed method improves upon the baselines by masking out \"unimportant tokens,\" thereby amplifying the supervision signal from the \"important tokens.\" However, a much simpler way to achieve a similar effect is to increase the learning rate uniformly across all tokens, without masking. For instance, if the masking ratio is 20%, raising the learning rate by a factor of 1/(1-0.2) = 1.25x in the standard CPT baseline would result in a comparable signal strength for important tokens as in the knowledge distillation setting. To further validate the effectiveness of Teacher Guided Language Modeling, it would be beneficial to compare VocAgnoLM against this learning rate adjustment baseline.", "relation_to_broader_scientific_literature": "This paper proposes a method to tackle the tokenization mis-matching problem during knowledge distillation.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "In lines 435-439, the authors claim that MiniPLM[1] requires the teacher and student models to share the same vocabulary. However, [1] shows that MiniPLM enables cross-family KD where the vocabulary of the student and teacher models can be different. Is this a mistake in the literature review?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zQXaSXyWGX", "reviewer": "Reviewer_bj8e", "summary": "The paper addresses the challenge of vocabulary mismatches between teacher and student language models during knowledge distillation. I believe this is a well-motivated and important topic since it is difficult to do the logits-level distillation between student and teacher models with different tokenizers. To overcome this, the authors propose the method that consists of two main components:\n• Token-level Lexical Alignment: A procedure to map student tokens to the corresponding teacher tokens using character-level offsets, thereby aligning token sequences despite different vocabularies.\n• Teacher Guided Loss: A reweighting scheme that leverages the teacher’s token-level loss as guidance for training the student, overcoming divergences in logit distributions.\nThe approach is validated on a math-focused pretraining corpus (OpenWebMath) and evaluated on a suite of mathematical reasoning benchmarks, showing significant performance improvements—especially when using teacher models with very low vocabulary overlap.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer’s insightful comments and valuable suggestions. We are glad that the reviewer acknowledged the importance of the problem and appreciated our comprehensive analysis. Below, we provide our detailed responses to the points raised.\n\n### **Distinction between \"Teacher-generated Knowledge\" and \"Teacher-Guided Language Modeling\"** (Exp Designs or Analyses #1)\nWe would first like to clarify that the scope of this work specifically targets the pretraining stage. Unlike (sequence-level) knowledge distillation [1] approaches trained on generated tokens in the fine-tuning stage, it is difficult to define the “generated tokens” of the teacher model during the pretraining stage. In the extreme case, a teacher model could reproduce the same input corpus or generate an entirely new 15B corpus using the teacher model’s internal knowledge.\n\nBesides, training the student model on new knowledge generated by the teacher model differs from our objective of teacher-guided language modeling within a given corpus. However, while such an approach goes beyond the scope of this work, we appreciate the suggestion and believe it as a valuable direction worth exploring further.\n\n[1] *Kim et al., Sequence-Level Knowledge Distillation, In EMNLP 2016.*\n\n### **Comparison with the standard KD using same tokenizer teacher** (Exp Designs or Analyses #2, Weakness)\n\nWe appreciate the reviewer’s suggestion for further experiments. We present an additional experimental result on 2B tokens that offers relevant insight. Specifically, we compare our method against the standard KD approach using two teacher models of comparable performance: MetaMath-Llemma-7B (which shares the same tokenizer as the student model) and Mistral-ProXMath-7B (which uses a different tokenizer).\n\nAs shown in the table below, our method performs better even when using a teacher model with a different vocabulary, despite both two teacher models achieving the same performance on the AVG(w/o SAT) metric.\n\n| Model                      | Tokenizer w/ Student | Method | AVG  | AVG (w/o *) | AVG (w/o SAT) |\n|---------------------------|----------------------|--------|------|-------------|----------------|\n| ***Teacher Model Performance*** |||||  \n| MetaMath-Llemma-7B        | Same                 | -      | 52.2 | 62.1        | 57.2           |\n| Mistral-ProXMath-7B       | Different            | -      | 59.2 | 58.9        | 57.2           |\n| ***Student (S) Model Performance*** |||||  \n| S + MetaMath-Llemma-7B    | Same                 | KLD    | 14.2 | 14.3        | 14.4           |\n| S + Mistral-ProXMath-7B   | Different            | Ours   | 16.6 | 16.6        | 16.3           |\n\nWe also clarify that our Token-level Lexical Alignment is a deterministic and straightforward solution based on character offset. While simple, it ensures 100% overlap (Figure 6), and plays a critical role in enabling reliable teacher guidance across different tokenizers. We believe that our alignment mechanism contributes to the effectiveness of our method even when compared to standard KD with the same tokenizer, as demonstrated by the results in the table.\n\n### **Clarification on Computational Overhead.** (Exp Designs or Analyses #3)\nTo report the additional computational overhead, we decompose into two components compared to standard KD, Token-level Lexical Alignment and Loss-based Guidance.\n\n- First, the loss-based guidance shares most of its computational operations with standard KLD. When measuring 1M tokens, KLD and loss-based guidance require approximately 15.36 TFLOPs, **showing no significant difference in computation cost.**\n- On the other hand, Token-level Lexical Alignment is a CPU-bound operation, so we report latency instead of FLOPs.. When measuring on a 2048 token sequence using the Mistral-ProXMath teacher model, the alignment step takes approximately 0.047 seconds, averaged on 1000 repeated runs. While this mapping process introduces a small amount of overhead, it can be amortized during preprocessing time. **Especially in pretraining stage, corpus is typically packed to the maximum sequence length, so the mapping process can be efficiently performed during preprocessing time.**\n\nWe hope this addresses the reviewer’s concern regarding computational overhead. We’ll include this point in the final version.\n\n### **Response for \"Essential References Not Discussed\"**\nThank you for the new suggestion. Minitron (Muralidharan et al., 2024) has already been cited in the introduction (Line 48-49). We also agree that Pretraining Exploration is a valuable reference that highlights the importance of knowledge distillation during pretraining. In the final version, we will revise the first paragraph of the introduction to emphasize on the importance of KD-based pretraining.", "claims_and_evidence": "I believe the claims (the motivation and effectiveness of the design) are generally well-supported by the experiments. The potential questions are listed in the other section. The experimental results indicate that VocAgnoLM improves performance by up to 46% compared to naive continual pretraining and outperforms logit distribution alignment baselines (e.g., KLD and ULD). Detailed ablation studies are provided to show how choices in token alignment granularity, handling of unmapped tokens, and aggregation strategies affect performance.", "methods_and_evaluation_criteria": "The proposed methods make sense to me while they are somehow heuristic and lack theoretical support for their effectiveness.\n\na) Token-level Lexical Alignment: 1) Utilizes character-level offsets from both teacher and student tokenizers. 2) Employs binary search techniques to establish a one-to-many mapping for each student token.\nb) Teacher Guided Loss: 1) Computes the loss for a student token and aggregates the losses of its corresponding teacher tokens. 2) Applies a top-k threshold strategy to reweight the importance of each student token based on the discrepancy between the student and teacher losses.\n\nAlthough the evaluation is mainly concentrated on the math/reasoning tasks, I believe they are representative. They compare performance on multiple math reasoning benchmarks (e.g., GSM8K, MATH, SVAMP, ASDiv, MAWPS, and others). Comparisons against standard distillation approaches (KLD and ULD) are also provided. Ablation studies that assess the impact of different alignment granularities and token handling strategies, which demonstrate the effectiveness of the design. The weakness of the evaluation is mentioned in the previous section.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "It is appreciated that the author compares the performance of VocAgnoLM to KLD-based and ULD-based distillation, and they also provide ablation study to demonstrate the effectiveness of the design. \n\nHowever, the evaluation is not comprehensive enough to demonstrate the bounds of the effectiveness of the proposed methods. We does want to understand the performance gain/loss of the proposed methods but the comparison seems not to be very fair. It would be great if the following two comparisons are provided: 1) learn from the logits of advanced models (e.g. qwen-math) vs. learn from their generated tokens. 2) Two similar student models (with different tokenizers) learn from the same teacher (or teachers with similar performance) - one with identical tokenizer vs one with different tokenizer. By this two experiments, we can understand the bound/limit of the proposed methods, which can be very helpful.\n\nAlso, the computational complexity can be high, it would be great if more comprehensive and explicit data points can be provided.", "relation_to_broader_scientific_literature": "I believe the proposed methods are important and relevant to general LLM pretraining, especially for knowledge distillation domain. Previous works mainly consider the distillation between models of the same tokenizer. Recent works 'Towards cross-tokenizer distillation: the universal logit distillation loss for LLMs' proposes methods for cross-tokenizer distillation. In this paper, the proposed methods are shown to outperform the existing cross-token KD methods and improve the performance based on KD process.", "essential_references_not_discussed": "There are more existing literatures considering the knowledge distillation (not necessarily the cross-tokenization KD), such as the KD for pretraining exploration (https://arxiv.org/pdf/2410.16215) and the Nvidia minitron (https://arxiv.org/pdf/2407.14679). It is good to include more previous papers on KD to further emphasize the importance of the KD-based pretraining.", "other_strengths_and_weaknesses": "* Strength:\nComprehensive Analysis: Extensive ablation studies and comparisons with existing methods (KLD, ULD) strengthen the evidence for the proposed approach.\n\nPractical Impact: By enabling the use of high-performing teacher models regardless of vocabulary, the method broadens the applicability of teacher-guided pretraining, particularly in domain-specific settings like mathematics.\n\n* Weakness:\n\nThe methods are heuristic, and we are not sure how much the proposed methods will compromise/influence the KD performance, compared to the standard KD with the same tokenizer.\n\nThe topic is of great importance and I will raise my score if concerns can be properly addressed.", "other_comments_or_suggestions": "Please kindly refer to the weakness discussed above.", "questions_for_authors": "Please kindly refer to the question in the previous sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pB0tn6kSim", "reviewer": "Reviewer_ZCju", "summary": "This paper proposes a Vocabulary-agnostic Teacher Guided Language Modeling for guiding the training of smaller student models by large teacher models. This method tries to bridge the gap caused by vocabulary mismatch in different models. The proposed approach comprises two key components: Token-level Lexical Alignment and Teacher Guided Loss, both of which contribute to the enhancement of performance.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the reviewer’s helpful feedback. We are pleased that the reviewer recognized the motivation and analysis of our work. We provide our response as below.\n\n### **Clarification of the “Vocabulary Mismatch” Notion.**   (Claims And Evidence)\nWe appreciate the chance to clarify our terminology and the motivation behind it.\n\n- **Tokenizer Differences Originate from \"Vocabulary\" Choices:**\nThe issue of \"tokens extracted by different models\" fundamentally stems from the fact that these models use different tokenizers. When building a tokenizer, the primary goal is to construct a vocabulary. Since different model series typically start with tokenizers based on distinct vocabularies, we believe that the fundamental reason for the observed mismatch lies in the vocabulary itself. Thus, we refer to this problem as a *vocabulary mismatch*.\n\n- **\"Vocabulary\" Mismatch Leads to \"Token\" Sequence and Logit Mismatches:**\nThe consequences of vocabulary mismatch manifest as differences in token sequences and divergences in logit distributions. Since the token sequence is determined by the tokenizer’s vocabulary, we treat “token sequence mismatch” as a downstream result of vocabulary mismatch. Moreover, in transformer-based models, the vocabulary size directly determines the dimensionality of the final logit layer. Therefore, “logit distribution divergence” is more precisely attributed to differences in vocabulary, rather than just token-level differences.\n\n- **Terminological Consistency with Prior Work:**\nA large body of prior work [1,2,3] has used terms in a way that implicitly refers to mismatches in vocabulary. In this context, using the term *vocabulary mismatch* aligns with established usage in the field. The suggestion that it is misleading may in fact run counter to prevailing terminology practices in related literature.\n\n[1] *Cui et al., Multi-level optimal transport for universal cross-tokenizer knowledge distillation on language models. In AAAI 2025.*\n\n[2] *Boizard et al., Towards cross-tokenizer distillation: the universal logit distillation loss for LLMs. TMLR, 2025.*\n\n[3] *Xu et al., Bridging the gap between different vocabularies for llm ensemble. In NAACL 2024.*\n\n\n### **Highlighting Our Contribution on Vocabulary Mismatch in Pretraining.**   (Weaknesses)\nWe would like to highlight that **addressing *vocabulary mismatch* in the pretraining stage** is one of the main contributions of our work.\nWhile knowledge distillation during pretraining is a well-established technique, in practice, vocabulary mismatch often limits the use of diverse teacher models.\nTo our best knowledge, this aspect remains underexplored in the pretraining literature, and we believe **this is the first work** to directly address vocabulary mismatch in this context.\nVocagnoLM offers **a simple yet effective** solution to this problem, with practical benefits as follows:\n- **Practicality of Token-Level Lexical Alignment**\n  - While our token-level lexical alignment may not constitute a novel theoretical contribution, we argue that this is precisely what makes it a **practically effective** solution to the vocabulary mismatch problem. Our proposed method performs lexical alignment based on character offsets, which theoretically ensures that every student token must be contained within one or more teacher tokens. In practice, as shown in Figure 6, we observe 100% Intersection-of-String (IoS), demonstrating that two token sequences can be efficiently and accurately aligned using only a simple approach.\n  - Furthermore, this practical nature allows for both *online mapping* during training and *offline mapping* during preprocessing. This flexibility is particularly advantageous in large-scale pretraining scenarios, where optimization at the training stage is crucial.\n  - We highlight the simplicity and efficiency of our mapping function, which supports various weighting options in the aggregation step. While we only explore a simple aggregation function in this paper, we leave the exploration of more advanced variants as future work.\n\n- **Advantages of Loss-Based Teacher Guidance for Logit Distribution Divergence**\n  - Even after resolving the token sequence mismatch, the issue of logit distribution divergence remains. Traditional KL divergence cannot adequately address this, and existing distance-based methods such as ULD, can incur additional information loss during alignment.\n  - In contrast, our approach is based on the teacher’s loss, which allows us to circumvent the problem arising from the dimensional mismatch between vocabularies.\n\nIn summary, we **decompose the issues caused by vocabulary mismatch into two sub-problems (token sequence mismatch / logit distribution divergence)** and design separate solutions for each. We demonstrate that applying these techniques in the pretraining stage leads to effective improvement.", "claims_and_evidence": "The authors argue that the perceived vocabulary mismatch is, in fact, a mismatch of tokens extracted by different models. Presenting this as a vocabulary mismatch appears to introduce a misleading notion.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are reasonable.", "theoretical_claims": "The authors lack corresponding theoretical analysis, and the proposed solution tends to be more engineering-oriented.", "experimental_designs_or_analyses": "Yes, I have checked the validity of experimental designs, and they are  acceptable.", "relation_to_broader_scientific_literature": "There's nothing to add", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "**Strengths**\n\nThe presentation of the paper is good, with each section being clear. The focused problem is well-articulated, with an in-depth analysis and preliminary tiny experiments to illustrate it.\n\nThe suggested method achieves effective performance improvements. \n\nThe scalability with different teachers has also been sufficiently validated.\n\n**Weaknesses**\n\nThe contributions of this paper are insufficient. Firstly, the Token-level Lexical Alignment appears to be an engineering operation, and the theoretical analysis of the effectiveness of the proposed mapping is lacking. Secondly, the technical contribution of weighting different tokens is not substantial enough.", "other_comments_or_suggestions": "No", "questions_for_authors": "Please refer to the Weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper tackles a pervasive obstacle in teacher‑student distillation for LLMs: when teacher and student use different tokenizers, their token sequences and logits don’t align. This work offers two pragmatic fixes during pretraining: (1) Token‑level lexical alignment, which uses simple character‑offset matching (binary search over tokenizer spans) to deterministically map each student token to one or more teacher tokens. (2) Teacher‑Guided Loss, which aggregates the teacher’s per‑token losses (over the aligned spans), then applies a top‑k reweighting so the strongest signals drive student updates—avoiding spurious supervision on “unmatched” or low‑importance tokens. Overall, given the practical significance, strong empirical evidence, and the fact that vocabulary mismatch is a known but under‑addressed barrier in pretraining‑stage knowledge distillation, I recommend acceptance of the paper. The authors should carefully address the reviewers' remaining suggestions though, for example, those related to adding a broader set of evaluations.
Decision: Accept (poster)