Overcoming Vocabulary Mismatch: Vocabulary-agnostic Teacher Guided
  Language Modeling
Authors: name: Haebin Shin, name: Lei Ji, name: Xiao Liu, name: Yeyun Gong

Using large teacher models to guide the training of smaller student models
has become the prevailing paradigm for efficient and effective learning.
However, vocabulary mismatches between teacher and student language models pose
significant challenges in language modeling, resulting in divergent token
sequences and output distributions. To overcome these limitations, we propose
Vocabulary-agnostic Teacher Guided Language Modeling (VocAgnoLM), a novel
approach that bridges the gap caused by vocabulary mismatch through two key
methods: (1) Token-level Lexical Alignment, which aligns token sequences across
mismatched vocabularies, and (2) Teacher Guided Loss, which leverages the loss
of teacher model to guide effective student training. We demonstrate its
effectiveness in language modeling with 1B student model using various 7B
teacher models with different vocabularies. Notably, with
Qwen2.5-Math-Instruct, a teacher model sharing only about 6% of its vocabulary
with TinyLlama, VocAgnoLM achieves a 46% performance improvement compared to
naive continual pretraining. Furthermore, we demonstrate that VocAgnoLM
consistently benefits from stronger teacher models, providing a robust solution
to vocabulary mismatches in language modeling.