DB-LLM: Accurate Dual-Binarization for Efficient LLMs (2024)

Abstract: Large language models (LLMs) have significantly advanced the field of natural language processing, while the expensive memory and computation consumption impede their practical deployment. Quantization emerges as one of the most effective methods for improving the computational efficiency of LLMs. However, existing ultra-low-bit quantization always causes severe accuracy drops. In this paper, we empirically investigate the micro and macro characteristics of ultra-low bit quantization and present a novel Dual-Binarization method for LLMs, namely DB-LLM. For the micro-level, we take both the accuracy advantage of 2-bitwidth and the efficiency advantage of binarization into account, introducing Flexible Dual Binarization (FDB). By splitting 2-bit quantized weights into two independent sets of binaries, FDB ensures the accuracy of representations and introduces flexibility, utilizing the efficient bitwise operations of binarization while retaining the inherent high sparsity of ultra-low bit quantization. For the macro-level, we find the distortion that exists in the prediction of LLM after quantization, which is specified as the deviations related to the ambiguity of samples. We propose the Deviation-Aware Distillation (DAD) method, enabling the model to focus differently on various samples. Comprehensive experiments show that our DB-LLM not only significantly surpasses the current Stateof-The-Art (SoTA) in ultra-low bit quantization (e.g., perplexity decreased from 9.64 to 7.23), but also achieves an additional 20% reduction in computational consumption compared to the SOTA method under the same bit-width. Our code is available at https://github.com/HonChen/DB-LLM.

heading: 1 Introduction
text: Recently, Large Language Models (LLMs), such as ChatGPT and LLaMA (Touvron et al., 2023a)
*Equal contribution. †Corresponding author.
have catalyzed a paradigm shift in various natural language processing tasks (Zhong et al., 2023; Peng et al., 2023; Lu et al., 2023). Their unprecedented capabilities evolved from a massive memory footprint (e.g., billion-scale parameters), which constrains the widespread application of LLMs on resource-limited devices. Several compression schemes are thus proposed to reduce the memory demands of LLMs, which can be roughly categorized into weight quantization (Frantar et al., 2022; Lin et al., 2023), network pruning (Sun et al., 2023; Ma et al., 2023; He et al., 2022), knowledge distillation (Gu et al., 2023; Zhong et al., 2024) and low-rank factorization (Xu et al., 2023; Yuan et al., 2023). Among these methods, weight quantization is highly effective and practical since it achieves the best trade-off between the performance and the cost of the compression process. Nevertheless, although many works (Shao et al., 2023; Shang et al., 2023) attempt to quantize LLMs to ultra-low-bit (e.g., 2- bit), their performance is unsatisfactory and falls far short of industrial application requirements.
Ultra-low-bit quantization (≤ 4 bits), as an extremely efficient form of quantization, enjoys over 8× memory compression ratio. Despite these specialized weight-only quantization schemes achiev-
8719
ing savings in storage consumption, they still cannot avoid costly floating-point arithmetic. Moreover, we notice that they will cause catastrophic degradation in accuracy.
For instance, despite the application of advanced 2-bit quantization techniques, a 65B model still falls marginally short of the performance level attained by a 7B model (Shao et al., 2023). And fully binarized Large Language Models are almost impracticable (Shang et al., 2023). The rationale lies in two important aspects: From the microlevel perspective: We empirically observe that the symmetric Gaussian distribution of pre-trained weights poses great challenges when quantizing to extremely low-bit (1-bit and 2-bit). Binarization suffers from poor representation capability, leading to a collapse in performance. While 2-bit quantization alleviates this issue to some extent, it still exhibits limited efficiency and presents optimization obstacles. Thus, directly applying the aforementioned strategies to LLMs is suboptimal which necessitates a novel specialized operator. From the macro-level perspective: We make in-depth investigations of the prediction preferences and discover the low-bit LLMs exhibit a form of distortion, far from the original long-tail distribution of the fullprecision models. Especially, the extremely low-bit LLMs tend to potentially predict head classes when encountering ambiguous samples. This tendency highlights a potential bias in their performance.
To address these issues, we propose a novel Dual Binarization method to achieve accurate 2-bit LLMs in a data-free manner, dubbed as DB-LLM. Specifically, we (1) introduce a Flexible Dual Binarization (FDB) to enhance the representation capability by flexible dual-binarizer, while fully leveraging the efficiency benefits of the binarized parameter. Explicitly, we initialize an INT2 counterpart as the intermediary, splitting its weights into dual-binarized representations deftly in our DBLLM. Then, in a data-free manner, we fine-tune the scales to further enhance the representation capability. Second, we (2) propose a Deviation-Aware Distillation (DAD) to mitigate the distorted preferences. DAD jointly leverages the student-teacher entropy as an ambiguous indicator and further amplifies the sample-wise ambiguity by re-weighting the distillation loss. This method enables the lowbit LLMs to perceive the uncertainty of each sample, which fulfills the balanced knowledge transfer.
Extensive experiments on several benchmark datasets and model families show that DB-LLM
outperforms the existing state-of-the-art (SOTA) quantization methods by a convincing margin (see Figure 1). For example, our DB-LLM achieves perplexities of 5.52 and 4.84 under 2-bit weight on LLaMA-1-30B and LLaMA-1-65B respectively, comparable to full-precision LLaMA-1-7B (perplexity of 5.68) and even surpassing the 3-bit AWQ (Lin et al., 2023), which highlights its superiority and versatility. To summarize, our main contributions are:
• We present Flexible Dual Binarization, which transcends data format constraints, maximizing representation capability while maintaining the efficiency of binary operations.
• We analyze the distortion related to prediction preference in the ultra-low bit LLMs and introduce a Deviation-aware Distillation to emphasize the ambiguous samples.
• Extensive experiments on Llama1&2 families spanning 7∼70B show that our DB-LLM significantly and consistently outperforms prior quantization strategies on various tasks.

heading: 2 Related Work

heading: 2.1 LLM Quantization
text: The quantization schemes of LLM can be briefly classified into two fields: weight-only quantization (Frantar et al., 2022; Lin et al., 2023; Chee et al., 2023) and weight-activation quantization (Wei et al., 2023; Xiao et al., 2023; Shao et al., 2023; Zhu et al., 2023). The first approach concentrates on reducing the model storage while the second one simultaneously accelerates the inference speed. For the weight-only quantization, GPTQ (Frantar et al., 2022) proposes a layer-wise quantization that compensates the rounding errors with second-order information. AWQ (Lin et al., 2023) prioritizes preserving the salient weights by the activation magnitude. QuIP (Chee et al., 2023) introduces quantization with incoherence processing, optimizing quantization in large language models but introducing additional overhead during inference. For the weight-activation quantization, several efforts (Xiao et al., 2023; Wei et al., 2023; Liu et al., 2023a; Shao et al., 2023) shift the challenge of outliers from activations to weights with per-channel scaling transformation, including optimization-free methods (Wei et al., 2023; Xiao et al., 2023) and optimization-based methods (Shao
et al., 2023; Liu et al., 2023a). However, these works undergo non-trivial performance degradation in ultra-low-bit (e.g., 2-bit). In contrast, our method achieves satisfactory accuracy.

heading: 2.2 Network Binarization
text: BNN (Hubara et al., 2016) is a radical quantization form to compress weights and activations into only 1 bit. Following the success of binarization in computer vision (Rastegari et al., 2016; Liu et al., 2018; Qin et al., 2020; Liu et al., 2020), its exploration in natural language processing also attracts wide research interest. BinaryBERT (Bai et al., 2021) equivalently splits the weights of welltrained TernaryBERT (Zhang et al., 2020) and further fine-tune it to enhance the performance. Subsequent works aim to binarize both weight and activations, which is more challenging. BiBERT (Qin et al., 2021) revisits the performance bottleneck (i.e., softmax function) and proposes Bi-Attention to tackle information degradation. BIT (Liu et al., 2022) introduces a two-set binarization scheme, applying different mapping levels for non-negative and positive-negative activation layers. Most recently, PB-LLM (Shang et al., 2023) first attempts to bianrize the un-salient weights for LLM. Yet, such a mixed-precision manner limits its hardware deployment and extreme storage savings.

heading: 3 Methodologies

heading: 3.1 Preliminaries
text: In this section, we briefly review the necessary backgrounds. We consider the quantization and binarization as follows:
Uniform quantization is the most widely used method. For the k-bit setting, the quantization and de-quantization procedures can be written as:
wq = clamp(⌊w s ⌉,−2k−1, 2k−1 − 1), (1)
ŵ = s · wq ≈ w, (2)
where Wq is the quantized integer and s is the scaling factor determined by max(|W|)
2k−1 . To overcome the non-differentiable issue in the backward propagation, the Straight-Through-Estimator (STE) (Courbariaux et al., 2015) is introduced to compute the approximate gradient.
The traditional BNNs binarize the network parameters (weights and activations) into 1-bit. The binarization on weights can be achieved by apply-
ing the sign function for the forward propagation:
wb = sign(w) = { 1 if w ≥ 0 −1 otherwise , (3)
where w and wb represent the 32-bit floating-point weight and 1-bit binarized weight.

heading: 3.2 Flexible Dual Binarization
text: These days, researchers discover the weights of LLMs exhibit symmetric Gaussian distribution and a small fraction of salient weights is critical to the quantization performance (Lin et al., 2023; Shao et al., 2023). We make in-depth investigations about the optimization from multi-low-bit perspectives (see Figure 4). The binarization suffers from poor representation capabilities. The remaining two levels converge towards 0 (shown in blue in Figure 3), which neglects the salient weights and is attributed to the highest loss values. Alternatively, 2-bit quantization naturally overcomes the representation bottleneck (expression span exceeds twice that of binarization in Figure 3). The minimum loss point is significantly reduced while the loss surface is still steep which brings the optimization difficulty.
To combine the notable efficiency inherent in binarization and the flexible representation capabilities of 2-bit quantization, we propose the Flexible Dual Binarization (FDB) whose loss landscape is flat and enjoys the lowest loss. Our FDB primarily consists of the initialization phase and the fine-tuning phase. It first inherits the considerably high-performing initialization from relatively highbit LLMs and then fine-tunes the scales to further enhance the representation capability. In particular, we consider a 2-bit LLM as a proxy to be sufficient to tackle this obstacle (in Figure 3) thus we split its quantized weights into two separate 1-bit. We formulate such a splitting process as follows:
ŵ = s ·wq = α1 ·wb1 + α2 ·wb2, (4) where wb1,w b 2 represent two 1-bit weights and α1, α2 are their corresponding scaling factors. To achieve the isometric step s between quantization levels in Equation 4 and maintain higher sparsity, we revisit the binarization levels and adjust it to {0, 1}. To illustrate, suppose that α1 is positive and α2 is negative in Figure 5. Thus the initial value of α1, α2 can be expressed as:
α1 := 2s, α2 := −s. (5)
The quantization parameters, α1 and α2 will be optimized during the fine-tuning stage, which leads to the non-isometric quantization levels (in Figure 3). Therefore, our goal is to compare the magnitude between values and level center in Figure 5:
wb1 = H(w − α1 + α2
2 ), (6)
wb2 = H(−(w − α1 ·wb1 − α2 2 )), (7)
where H(·) is the unit step function, defined as 0 for negative values and 1 for positive values. Therefore, the whole forward process of FDB is expressed as:
ŷ = α1 · (wb1 ⊗ x) + α2 · (wb2 ⊗ x), (8)
Where x and ŷ denote inputs and outputs of the current layer respectively, and ⊗ denotes the inner product with bitwise operation.
It is noteworthy that our elaborate Flexible Dual Binarization (FDB) enjoys multiple advantages: 1) it inherits and enhances the superior representation capacities of ultra-low bit quantization, 2) it capitalizes on the considerable efficiency derived from bitwise operation, 3) it maintains the notable high sparsity characteristic of ultra-low bit quantization.

heading: Discussion on compression and acceleration.
text: We have innovated the sparsity of neural network weights by decomposing traditional 2-bit weights into dual 1-bit representations. This method, applied in the LLaMA-1-7B model, significantly increases the average weight sparsity, exceeding 60%. Notably, there is a distinct variation in the degree of sparsity between wb1 and w b 2, with the sparsity of wb2 consistently surpassing 70%. This enhanced sparsity level is not only instrumental in drastically reducing the computational power requirements, potentially leading to significant acceleration in processing speed, but also facilitates more compression of wb2 using various encoding methods (Van Leeuwen, 1976; Han et al., 2016). Theoretically, this approach could reduce the average bit size of the overall weights to approximately 1.88 bits (Shannon, 1948). These reductions, as previously mentioned, are significant when compared to traditional quantization methods, highlighting the superior efficiency of our approach.
Discussion on flexibility. As shown in Figure 4, we compare the layer-wise loss landscapes in bi-
L os
s
L os
s
narization, 2-bit quantization, and Flexible Dual Binarization (FDB). Our FDB achieves a minimum loss comparable to that of 2-bit quantization but significantly differs from binary quantization. FDB features a flatter optimization surface, which allows it to maintain a lower loss over a considerable range, reflecting its flexibility in net-wise optimization. Given that our FDB can be initialized through 2-bit quantization, the closeness of their lowest loss points also indicates that further optimization of FDB is likely to be easier.
Inspired by LLM-QAT (Liu et al., 2023b), we can further utilize distillation techniques to efficiently fine-tune the quantization parameters using the original full-precision model, without the need for introducing additional data. This data-free approach helps avoid the risk of overfitting.

heading: 3.3 Deviation-aware Distillation
text: The tokenizer construction of current mainstream Large Language Models is based on Byte Pair Encoding (BPE) (Gage, 1994; Sennrich et al., 2016), which leverages the long-tail corpus. Similarly, we observe that the prediction preference of full-
precision LLMs obeys the long-tail distribution in Figure 6(a). However, the predictions of extremely low-bit models deviate from such longtail distribution and exhibit increased distortion (in Figure 6(b)), which manifests as a bias towards high-frequency words. In particular, the quantized models are more inclined to predict head classes (i.e., the higher frequency region of the vocabulary). Statistically, we count the prediction deviations given the same inputs, and the low-bit model is about 1.6 times more likely to predict commonly occurring head classes than the less frequent tail classes, indicating a bias towards more prevalent categories.
To delve deeper into the reason for distortion, we explore the failure predictions and utilize the information entropy (Shannon, 1948) to measure their corresponding uncertainty, defined by:
H(P ) = − C∑
i=1
pi log(pi), (9)
where C is the class number and pi is the probabil-
ity of i-th class. As shown in Figure 7, surprisingly, the entropy of the teacher/student model is consistent with the task loss (i.e., cross-entropy). It means that the quantized model struggles with making predictions for ambiguous samples. Considering previous observations, it is reasonable to assume that the effectiveness of the quantized model decreases when dealing with ambiguous samples, leading to a preference for more conservative predictions.
Inspired by these findings, we propose the Deviation-Aware Distillation (DAD) which prioritizes uncertain samples by utilizing a pair of entropy (i.e., teacher-student entropy) as a difficulty indicator. Specifically, the twin entropy is multiplied into the original loss function as two terms, which is defined as:
ℓDAD = H(P t)γ ·H(P s)1−γ ·ℓCE(P t, P s), (10)
where superscript t and s denote teacher and student models respectively. ℓCE(P t, P s) is the crossentropy loss between quantized student logits P s and teacher logits P t. The overall distill loss is:
ℓtotal = λ · ℓDAD + ℓCE , (11)
where λ is the trade-off parameter. Eventually, we analyze the issue of head class convergence in ultra-low-bit student models and propose the DAD loss to address it. DAD utilizes the teacher-student entropy as a challenge indicator and pays sufficient attention to the ambiguous samples by reweighting the distillation loss, which promotes the more balanced transfer of knowledge from full-precision teacher models.

heading: 4 Experiments
text: Models and datasets We conduct extensive experiments on LLaMA-1 (Touvron et al., 2023a) and
LLaMA-2 (Touvron et al., 2023b) families. To evaluate the effectiveness of our DB-LLM, we measure the perplexity for the language generation tasks (i.e., WikiText2 (Merity et al., 2016) and C4 (Raffel et al., 2020), and accuracy for the zero-shot tasks (i.e., PIQA (Bisk et al., 2020), ARC (Clark et al., 2018), HellaSwag (Zellers et al., 2019) and WinoGrande (Sakaguchi et al., 2021).
Baselines We mainly compare DB-LLM with the state-of-the-art weight-only quantization methods, including RTN (round-to-nearest quantization), GPTQ (Frantar et al., 2022), AWQ (Lin et al., 2023), QuIP (Chee et al., 2023), OmniQuant (Shao et al., 2023) and the partially binarized strategy PB-LLM (Shang et al., 2023). To unify the model weights to a 2-bit representation, we set the ratio of salient weights (8-bit representation) in the PBLLM to 17 ( 1 7 × 8 + 67 × 1 = 2bits).
Implementations Following LLM-QAT (Liu et al., 2023b), we construct the data-free calibration set which comprises 20k samples. Note that the quantization parameters are optimized for only 1 epoch with a batch size of 2. The γ and λ in Deviation-Aware Distillation is set to 0.1 equally. We adopt the AdamW (Loshchilov and Hutter, 2018) as an optimizer and the learning rate is set to 1e−5.

heading: 4.1 Main results
text: We conduct extensive experiments on LLaMA families across different model sizes (7B∼70B) and evaluation tasks (the detailed results of LLaMA-2 can be found in Appendix A.2). Note that we focus on the performance of extremely low-bit settings (i.e., W2A16).
For the language generation tasks, as seen in Table 1 and Table 2, some previous methods, such as AWQ, suffer from non-trivial performance degradation (Perplexity at the level of e5). Fortunately, our DB-LLM consistently achieves lower perplexity for all the datasets. For instance, DB-LLM averages a 1.68 improvement in perplexity over OmniQuant on LLaMA-1-7B. When the model becomes larger, DB-LLM still obtains approximately 0.80 reduction in perplexity, which showcases the effectiveness and versatility. Notably, we found that our scheme even surpasses the RTN, and AWQ under W3A16, which further indicates the strong performance of DB-LLM. To the best of our knowledge, our 2-bit LLaMA-1-30B outperforms the fullprecision LLaMA-1-7B with 3.7× storage savings.
Moreover, our method is also demonstrated advantages in zero-shot tasks in Table 5. DB-LLM still outperforms other state-of-the-art strategies by a large margin. For instance, our approach improves the accuracy of LLaMA-1-7B by 6.39% and 5.45% on HellaSwag and Winogrande, respectively. Meanwhile, for LLaMA-1-65B, DB-LLM is close to FP results (less than 4% accuracy degradation).

heading: 4.2 Ablation Studies
text: To better understand the effectiveness of our method, we provide detailed ablation studies to show the effect of each component and the proposed deviation-aware loss.
Ablation for each component: Table 3 shows the effect of each component. When removing the DAD component, the perplexity slightly increases by about 0.1%-0.2% since the quantized model struggles to predict ambiguous samples. Furthermore, the FDB component is critical as the performance decreases significantly (7.77 to 18.32) without a fine-tuning procedure. Our well-designed FDB flexibly enhances the representation capability and promotes efficient computation.
Ablation for Deviation-Aware Loss: To investigate the impact of key hyper-parameter γ, we conduct the ablation experiments in Table 4. We find that simply introducing the student entropy (γ = 0) or teacher entropy (γ = 1) adversely affects the performance, and the teacher model is more convincing. Hence, by validation, we set γ to 0.1 in all our experiments, which is a sweet spot that unites the teacher-student entropy to guide the quantized model.
Ablation for dataset sizes: We conduct experiments on different dataset sizes and compare the training overhead with OmniQuant in Table 7. For a 7B model, OmniQuant (Shao et al., 2023) trained for 20 epochs using 128 samples, which, according to the paper, takes about 1.1 hours on a single A100 GPU. Our method only requires training for a single epoch. By reducing the amount of data

heading: Method Model Size Sparsity FLOPs
text: used by our DB-LLM, such as adopting a dataset of only 2.5K in size, it only takes approximately 1.1 GPU hours as well. However, our approach still outperforms OmniQuant, with a perplexity reduc-

heading: Method Dataset Size Ppl Avg. Acc Avg. GPU hours
text: tion of about 0.4 and 1.6% accuracy improvement on downstream tasks. Furthermore, our DB-LLM’s accuracy continues to improve as the size of the dataset increases.
Ablation for group sizes: We add experimental results for our DB-LLM under different group sizes (64, 128) in Table 8, which are widely used. As
the group size increases, we observe a decline in model performance. Notably, our DB-LLM at a group size of 128 surpasses OmniQuant at a group size of 64 (8.63 vs 8.90 on WikiText2 and 10.81 vs 11.79 on C4).

heading: 4.3 Storage Saving and Speedup
text: As shown in Table 6, we specifically calculate the model size, sparsity, and computational complexity of several compression methods of the LLaMA1-7B model. In terms of model size, we have introduced a negligible amount of quantization parameters. However, as analyzed in the previous Section 3.2, higher sparsity can lead to a lower average number of bits per weight. This suggests that there is further potential for model size reduction. Despite this, the overall model size is almost identical to that of 2-bit quantization. More importantly, our model exhibits significantly higher sparsity, which substantially reduces the computational complexity during model inference. We measure computational complexity by the number of floating-point operations (FLOPs) required for a single inference (Sun et al., 2023; Ma et al., 2023; Liu et al., 2018). The FLOPs decreases from 423.4 billion to 29.8 billion, indicating a reduction of approximately 14.2 times.
1-bit matrix operations lack some underlying support, but our method is also a weight-only quantization method, which can benefit from deploying packed weights without introducing additional operations, compared to QuIP (Chee et al., 2023).

heading: 5 Conclusion
text: In this paper, we present DB-LLM, an accurate Dual-Binarization approach for efficient Large Language Models (LLMs). Through a detailed analysis of extremely low-bit quantization and binarization, we’ve outlined the advantages and disadvantages of each method. Capitalizing on these insights, we meticulously develop the Flexible Dual Binarization to represent weights efficiently. This method transcends the constraints imposed by data formats. Additionally, we examine the macro-level prediction deviations in low-bit quantization and introduce Deviation-Aware Distillation, which directs
the model to focus more on ambiguous samples. Our experiments show that our method surpasses the current state-of-the-art (SOTA) in 2-bit quantization accuracy and also greatly reduces computational demands compared to traditional techniques.

heading: Limitations
text: While our DB-LLM demonstrates considerable advancements in ultra-low bit quantization, there are still avenues for further exploration and improvement. Firstly, the potential of full binarization for even more extreme bit-width compression presents an area that warrants additional investigation. This approach could further reduce computational demands but needs careful consideration to maintain model accuracy. Secondly, our current methodology primarily focuses on weight quantization, leaving the quantization of activation and scale values as a promising area for future research. Delving into these aspects could yield additional gains in efficiency and model performance, making LLMs even more accessible for practical applications.

heading: Ethics Statements
text: We take ethical considerations seriously and strictly adhere to the ACL Ethics Policy. This paper proposes a flexible dual binarization algorithm and a deviation-aware distillation method to improve the computational efficiency of LLMs. All employed models and datasets in this paper are publicly available and have been widely adopted by researchers. All experimental results upon these open models and datasets are reported accurately and objectively. Thus, we believe that this research will not pose any ethical issues.

heading: Acknowledgments
text: We are grateful to the anonymous reviewers and the area chair for their insightful comments and suggestions. This work was supported by the National Science and Technology Major Project (2021ZD0110503), the National Natural Science Foundation of China (No. 62306025, No. 92367204), and Beijing Municipal Science and Technology Project (Nos. Z231100010323002). Xuebo Liu was sponsored by CCF-Tencent RhinoBird Open Research Fund.

heading: A Example Appendix

heading: A.1 Details about FDB
text: In the initialization phase, we use the GPTQ (Frantar et al., 2022) to quickly obtain the 2-bit quantized weights and integrate the zero points of asymmetric quantization into the scales. The positive and negative values of the scales correspond to two different zero points.

heading: A.2 More experimental results
text: See Table 9 for more results. The Results show a similar trend to that of LLaMA-1 in Table 1, demonstrating the effectiveness and universality of our proposed method.

heading: Future Work
text: Secondly, our current methodology primarily focuses on weight quantization, leaving the quantization of activation and scale values as a promising area for future research. Delving into these aspects could yield additional gains in efficiency and model performance, making LLMs even more accessible for practical applications