BitNet Distillation
Authors: name: Xun Wu, name: Shaohan Huang, name: Wenhui Wang, name: Ting Song, name: Li Dong, name: Yan Xia, name: Furu Wei
Organization: Microsoft Research

In this paper, we present BitNet Distillation (BitDistill), a lightweight
pipeline that fine-tunes off-the-shelf full-precision LLMs (e.g., Qwen) into
1.58-bit precision (i.e., ternary weights {-1, 0, 1}) for specific downstream
tasks, achieving strong task-specific performance with minimal computational
cost. Specifically, BitDistill incorporates three key techniques: the SubLN
module, as introduced in BitNet; multi-head attention distillation, based on
MiniLM; and continual pre-training, which serves as a crucial warm-up step to
mitigate the scalability issue of the performance gap between finetuned
full-precision and 1.58-bit LLMs on specific tasks. Experimental results show
that BitDistill achieves performance comparable to the full-precision
counterpart models across model size, while enabling up to 10x memory savings
and 2.65x faster inference on CPUs. Code is available at
https://github.com/microsoft/BitNet.