ARB-LLM: Alternating Refined Binarizations for Large Language Models

Authors: Zhiteng Li, Xianglong Yan, Tianao Zhang, Haotong Qin, Dong Xie, Jiang Tian, zhongchao shi, Linghe Kong, Yulun Zhang, Xiaokang Yang
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Binarization, LLM

Abstract:
Large Language Models (LLMs) have greatly pushed forward advancements in natural language processing, yet their high memory and computational demands hinder practical deployment. Binarization, as an effective compression technique, can shrink model weights to just 1 bit, significantly reducing the high demands on computation and memory. However, current binarization methods struggle to narrow the distribution gap between binarized and full-precision weights, while also overlooking the column deviation in LLM weight distribution. To tackle these issues, we propose ARB-LLM, a novel 1-bit post-training quantization (PTQ) technique tailored for LLMs. To narrow the distribution shift between binarized and full-precision weights, we first design an alternating refined binarization (ARB) algorithm to progressively update the binarization parameters, which significantly reduces the quantization error. Moreover, considering the pivot role of calibration data and the column deviation in LLM weights, we further extend ARB to ARB-X and ARB-RC. In addition, we refine the weight partition strategy with column-group bitmap (CGB), which further enhance performance. Equipping ARB-X and ARB-RC with CGB, we obtain ARB-LLM$_{\text{X}}$ and ARB-LLM$ _{\text{RC}} $ respectively, which significantly outperform state-of-the-art (SOTA) binarization methods for LLMs.
As a binary PTQ method, our ARB-LLM$ _{\text{RC}} $ is the first to surpass FP16 models of the same size. Code: https://github.com/ZHITENGLI/ARB-LLM.

Decision: accept (poster)
Reviewer scores: 8, 5, 8

Reviews and discussion:
<thread>
creation date:
1730449160847

thread_type:
official_review

<thread_object>
creator: reviewer_mucn

<content>
summary:
Authors propose to ALTERNATING REFINED BINARIZATION (ARB) which is based on BiLLM paper. They introduce correction term δµ to the original mean µ, effectively mitigating the distribution shift. As a result α, B and µ have to be refined.

They further extend ARB to ARB-X and ARB-RC, where 
ARB-X is extension of ARB with calibration data and loss function which is based on papers Optimal Brain Compression and GPTQ.

ARB-RC introduces a column-wise scaling factor αc to better handle parameter variations across columns, while eliminating the redistribution parameter µ to enhance compression in LLM.

In addition they combine ARB-X and ARB-RC with weight partition strategy based on columngroup bitmap (CGB).
And call final versions as ARB-LLMX and ARB-LLMRC respectively, which outperform state-of-the-art (SOTA) binarization methods for LLMs.

ARB-LLMRC outperforms FP16 models of the same size (in GB).

strengths:
Authors proposed several improvements of BiLLM method and show that ARB-LLMRC outperforms FP16 models of the same size (in GB) and outperforms BiLLM (it is SOTA of LLM post training binarization)

Extensive ablation study of the proposed methods showed effectiveness of Effectiveness of ARB, ARB-LLMX, ARB-LLMRC its combination with CGB and calibration data size, iteration number and group number.

Conducted extensive experiments on the LLaMA, LLaMA-2, and LLaMA3 families. 

They did a thorough analysis of time and memory consumption for proposed methods.

They will release all the code and models.

weaknesses:
Q: On Figure 1, authors show Pareto curve and demonstrate that  ARB-LLM-RC outperforms the same-size FP16 models and previously published binarized models. So authors show an improvements over previous models binarization, but it does not show the whole picture in terms of pareto curve: they only compare binarized models vs fp16 (on Figure 1). But it will be informative to show also 4bits and 8bits models. For example paper "The case for 4-bit precision: k-bit Inference Scaling Laws" shows that in terms of pareto curve, for the same model size, 4bits has the best accuracy. In addition to 4bits and 8bits it would be great to plot 3bits too on pareto curve (given that 3bits is already presented in multiple Tables in this paper).

questions:
Q1: Is it ARB second-order binarization is called in the paper " higher-bit representation, i.e., second-order binarizatio".  Is effectively 2bits quantization? 

Q2: Please explain why BiLLM accuracy in Table 2 does not match the results in the original BiLLM paper (shown in Table 3 LLaMA and LLaMA2)
</content>
</thread_object>

</thread>
<thread>
creation date:
1730692446981

thread_type:
official_review

<thread_object>
creator: reviewer_h9m3

<content>
summary:
The paper describes an Alternating Refined Binarization (ARB) framework for performing post-training quantization (PTQ) of 1-bit for large language models (LLM). The framework is based on the observation that there is a distribution gap between the binarized weights and the floating-point weights. Through the iterative refinement process of ARB, the quantization error can be reduced. The authors also provide two variants that use calibration data and address column deviation. They conduct experiments on OPT and LLAMA models, achieving better results compared to other binary PTQ methods such as BiLLM.

strengths:
The paper identifies and addresses the distribution shift between floating-point weights and binary weights through an iterative refinement method. The study shows that through the ARB process, the quantization error decreases.

The paper further extends the method by adding a calibration dataset and column-wise scales, resulting in better performance compared to BiLLM models.

The paper offers a comprehensive ablation study and time/memory analysis, showing the trade-offs of each component.

weaknesses:
The authors claim that the weight bit is 1.11 bits, but based on the memory comparison, the model takes approximately 3GB of memory on the Llama-7B model, which is similar in size to GPTQ-3bit of the same model, with inferior performance. Can the authors clarify this?

It seems the approach works better on the older OPT model than the Llama family, with Llama3's performance, in particular, being much worse than floating-point. Can the authors showcase the performance on more recent models like Phi/Llama3.2 to address this concern?

questions:
1. Can the authors provide any runtime inference metrics to showcase the practical feasibility of the proposed implementation?

2. Could the authors introduce the meaning of each term in equation (6) so that the reader can understand without referring to prior papers?

3. Can the authors clarify the actual memory consumption and compare it to PTQ methods under the same memory budget?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730703850655

thread_type:
official_review

<thread_object>
creator: reviewer_mmbq

<content>
summary:
This paper proposes a novel post-training quantization method alternating Refined Binarization for quantizing LLM using binary weight. The existing methods suffering the mis-alignment from the distribution of the binarized weight with the full precision counterparts. The existing method also didn't address the column-wise deviation. The proposed framework progressively refines binarization parameters to minimize the quantization error. The proposed framework has 4 components. 1. alternating refined binarization which iteractively update the binarized weight. 2. The proposed method is also improved by calibration data. 3. A more accurate binarization can be achieved by column-wise scaling factor. 4. A column-group bitmap strategy was applied in the paper to improve quantization by maximizing bitmap utilization through salient and non-salient division. The experiment results show impressive results.

strengths:
1. The proposed ARB-LLM method is innovative, which also addresses a critical problem in binary quantization.
2. The experiments on various LLM families demonstrate the superiority of the proposed methods.
3. The paper also provides thorough ablation studies the effectiveness of the individual components of ARB-LLM.

weaknesses:
1. The main weakness of the paper is the evaluation seems limited to perplexity and average accuracy. The paper would be more interesting if there were trade-offs on more tasks.

questions:
Does the proposed method perform consistently on 7 zero-shot QA datasets?

Is the binarized model able to get reasonable performance on long context, math and reasoning dataset?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732202373354

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response to all reviewers and area chairs for a brief summary

comment:
Dear reviewers and area chairs,

We sincerely thank all reviewers and area chairs for their valuable time and insightful comments.

We are pleased to note that:
1. Reviewer mmbQ acknowledges the innovation of our method, and both Reviewer mmbQ and H9m3 recognize that it effectively addresses critical challenges in binary quantization.
2. Reviewers H9m3 and muCn appreciate the comprehensive time and memory analysis provided in our work.
3. All reviewers commend the extensive nature of our experiments, particularly the ablation study, and recognize the superiority of our proposed method, which outperforms the previous SOTA method, BiLLM.

We have responded to each reviewer individually and would like to summarize our responses here:
1. We conduct experiments on **additional metrics, including F1 score and chrF**， further demonstrating the effectiveness of our proposed method.
2. We perform experiments on **various types of datasets (long context, math, and reasoning)** to showcase the versatility and effectiveness of our method.
3. We clarify the **memory usage of our method**, highlighting that it achieves **better performance than GPTQ-2bit while using less memory**.
4. We evaluate our method on newly introduced LLMs, **Phi-3-mini and Phi-3-medium**, demonstrating that it also **performs well on recent models**.
5. We provide a **runtime inference comparison**, showing that our method can **accelerate inference compared to FP16 and PB-LLM models**.
6. We extend the **Pareto curve analysis**, showing that **our method delivers the best performance under the same memory constraints**.
7. We clarify details in the paper, including the **accuracy of QA datasets**, an explanation of **second-order binarization**, and the **results of BiLLM**.

We extend our gratitude again to all reviewers and area chairs!

Best regards,

Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a new Alternating Refined Binarization (ARB) framework for performing post-training quantization (PTQ) of 1-bit for LLM. After rebuttal, it received scores of 588. On one hand, reviewers acknowledge that the proposed method is innovative, which effectively addresses critical challenges in binary quantization. The experiments are comprehensive, with thorough ablation studies. On the other hand, the reviewer who gave a score of 5 still shows concerns regarding runtime and performance. The authors have tried to address the reviewer's concerns during the rebuttal. Overall, the AC thinks that the merits of the paper outweigh the flaws, therefore, decided to recommend acceptance by the end.