BitNet Distillation
Authors: name: Xun Wu; name: Shaohan Huang; name: Wenhui Wang; name: Ting Song; name: Li Dong; name: Yan Xia; name: Furu Wei
Published: 2025-10-15 18:28:12

Abstract: In this paper, we present BitNet Distillation (BitDistill), a lightweight
pipeline that fine-tunes off-the-shelf full-precision LLMs (e.g., Qwen) into
1.58-bit precision (i.e., ternary weights {-1, 0, 1}) for specific downstream
tasks, achieving strong task-specific performance with minimal computational
cost. Specifically, BitDistill incorporates three key techniques: the SubLN
module, as introduced in BitNet; multi-head attention distillation, based on
MiniLM; and continual pre-training, which serves as a crucial warm-up step to
mitigate the scalability issue of the performance gap between finetuned
full-precision and 1.58-bit LLMs on specific tasks. Experimental results show
that BitDistill achieves performance comparable to the full-precision
counterpart models across model size, while enabling up to 10x memory savings
and 2.65x faster inference on CPUs. Code is available at
https://github.com/microsoft/BitNet.