DDK: Distilling Domain Knowledge for Efficient Large Language Models
Authors: name: Jiaheng Liu, name: Chenchen Zhang, name: Jinyang Guo, name: Yuanxing Zhang, name: Haoran Que, name: Ken Deng, name: Zhiqi Bai, name: Jie Liu, name: Ge Zhang, name: Jiakai Wang, name: Yanan Wu, name: Congnan Liu, name: Wenbo Su, name: Jiamang Wang, name: Lin Qu, name: Bo Zheng

Despite the advanced intelligence abilities of large language models (LLMs)
in various applications, they still face significant computational and storage
demands. Knowledge Distillation (KD) has emerged as an effective strategy to
improve the performance of a smaller LLM (i.e., the student model) by
transferring knowledge from a high-performing LLM (i.e., the teacher model).
Prevailing techniques in LLM distillation typically use a black-box model API
to generate high-quality pretrained and aligned datasets, or utilize white-box
distillation by altering the loss function to better transfer knowledge from
the teacher LLM. However, these methods ignore the knowledge differences
between the student and teacher LLMs across domains. This results in excessive
focus on domains with minimal performance gaps and insufficient attention to
domains with large gaps, reducing overall performance. In this paper, we
introduce a new LLM distillation framework called DDK, which dynamically
adjusts the composition of the distillation dataset in a smooth manner
according to the domain performance differences between the teacher and student
models, making the distillation process more stable and effective. Extensive
evaluations show that DDK significantly improves the performance of student
models, outperforming both continuously pretrained baselines and existing
knowledge distillation methods by a large margin.