BitNet: Scaling 1-bit Transformers for Large Language Models
Authors: name: Hongyu Wang; name: Shuming Ma; name: Li Dong; name: Shaohan Huang; name: Huaijie Wang; name: Lingxiao Ma; name: Fan Yang; name: Ruiping Wang; name: Yi Wu; name: Furu Wei
Published: 2023-10-17 17:59:15

Abstract: The increasing size of large language models has posed challenges for
deployment and raised concerns about environmental impact due to high energy
consumption. In this work, we introduce BitNet, a scalable and stable 1-bit
Transformer architecture designed for large language models. Specifically, we
introduce BitLinear as a drop-in replacement of the nn.Linear layer in order to
train 1-bit weights from scratch. Experimental results on language modeling
show that BitNet achieves competitive performance while substantially reducing
memory footprint and energy consumption, compared to state-of-the-art 8-bit
quantization methods and FP16 Transformer baselines. Furthermore, BitNet
exhibits a scaling law akin to full-precision Transformers, suggesting its
potential for effective scaling to even larger language models while
maintaining efficiency and performance benefits.