ARB-LLM: Alternating Refined Binarizations for Large Language Models

## Abstract

Abstract Large Language Models (LLMs) have greatly pushed forward advancements in natural language processing, yet their high memory and computational demands hinder practical deployment. Binarization, as an effective compression technique, can shrink model weights to just 1 bit, significantly reducing the high demands on computation and memory. However, current binarization methods struggle to narrow the distribution gap between binarized and full-precision weights, while also overlooking the column deviation in LLM weight distribution. To tackle these issues, we propose ARB-LLM, a novel 1-bit post-training quantization (PTQ) technique tailored for LLMs. To narrow the distribution shift between binarized and full-precision weights, we first design an alternating refined binarization (ARB) algorithm to progressively update the binarization parameters, which significantly reduces the quantization error. Moreover, considering the pivot role of calibration data and the column deviation in LLM weights, we further extend ARB to ARB-X and ARB-RC. In addition, we refine the weight partition strategy with column-group bitmap (CGB), which further enhance performance. Equipping ARB-X and ARB-RC with CGB, we obtain ARB-LLM X X {}_{\text{X}} start_FLOATSUBSCRIPT X end_FLOATSUBSCRIPT and ARB-LLM RC RC {}_{\text{RC}} start_FLOATSUBSCRIPT RC end_FLOATSUBSCRIPT respectively, which significantly outperform state-of-the-art (SOTA) binarization methods for LLMs.
As a binary PTQ method, our ARB-LLM RC RC {}_{\text{RC}} start_FLOATSUBSCRIPT RC end_FLOATSUBSCRIPT is the first to surpass FP16 models of the same size. The code and models will be available at https://github.com/ZHITENGLI/ARB-LLM .

## 1 Introduction

Figure: Figure 1: OPT performance on 7 zero-shot Question Answering (QA) datasets. Our ARB-LLM${}_{\text{RC}}$ outperforms the same-size FP16 models.
Refer to caption: x1.png

Recently, Transformer-based (Vaswani, ) large language models have shown impressive performance across various natural language processing tasks. However, this unprecedented capability is largely attributed to the sheer scale of these models, which often encompass billions of parameters. For instance, open pre-trained Transformer (OPT) series (Zhang et al., ) includes various models, with the largest boasting 66B parameters. Similarly, the LLaMA family (Touvron et al., ) features variants such as LLaMA3-70B, showcasing even larger architectures. The substantial memory requirements for inference in such large models (e.g., 150 GB memory for a 70B model) pose significant challenges for their deployment on mobile devices.

The study of compressing LLMs can be categorized into weight quantization (Lin et al., ; Frantar et al., ), low-rank factorization (Zhang et al., ; Yuan et al., ), network pruning (Sun et al., ; Frantar & Alistarh, ), and knowledge distillation (Zhong et al., ; Gu et al., ). Among these, binarization, a specific technique within the realm of quantization, is particularly distinguished for its ability to achieve extreme memory compression, reducing storage requirements to as low as 1 bit. Given the substantial size of LLMs, some binarization methods adopt the post-training quantization (PTQ) framework to enable a rapid transition from full-precision models to compact binarized versions, requiring minimal resources (e.g., binarizing a 70B model in one 80 GB GPU).

Recent binary PTQ methods, such as PB-LLM (Shang et al., ) and BiLLM (Huang et al., ), emphasize the identification of salient weights, which are crucial for model performance (Lin et al., ). The higher-bit representation and refined searching strategy for salient weights help to achieve a better trade-off between performance and storage. Despite their success, the refinement of the binarization process itself remains largely unaddressed, resulting in a significant difference between the binarized weights and their full-precision counterparts. This gap presents a considerable obstacle to further enhance the performance of binary LLMs.

Figure: Figure 2: Distribution shift between the mean of binarized and full-precision weights. Top: distribution shift of one row. Bottom: distribution shifts of multiple rows. Each row represents a top view of the corresponding upper image.
Refer to caption: x2.png

To minimize quantization error during the binarization process, we revisit the solutions for the binarization objective. Our analyses reveal that:
(i) The current approach is suboptimal due to the distribution shift between binarized and full-precision weights after binarization. As shown in Figure 2, the mean of the binarized weights is not aligned with the full-precision mean. Consequently, refining the binarization parameters based on the initial distribution of the binarized weights can yield a more accurate estimation of the original weights. Furthermore, this refinement can be alternately applied to different binarization parameters, ultimately leading to a significantly improved estimation.
(ii) While the calibration set is small, it plays a crucial role in the quantization of LLMs. However, the integration of calibration data for updating binarization parameters, which reflects a more realistic scenario, remains underexplored.
(iii) The weight distribution in LLMs exhibits noticeable column-wise deviations (see Figure 3), suggesting that the standard row-wise binarization method is inflexible and potentially unsuitable. Thus, incorporating both row and column scaling factors can produce more representative binarization results.

Figure: Figure 3: Left: Full-precision weights exhibit column-wise deviations. Right: BiLLM (Huang et al., ) with row-wise binarization smooths the deviations. Our ARB-RC with row-column-wise binarization effectively preserves them.
Refer to caption: x3.png

With the above observations and analyses, we first propose Alternating Refined Binarization (ARB) to align the distribution between binarized and full-precision weights in standard binarization process. Then, we extend this approach by incorporating the calibration data and row-column-wise scaling factors, leading to two advanced extensions: ARB-X and ARB-RC. Additionally, based on previous methods, which divide salient and non-salient weights and group weights by magnitude, we refine the integration of these two divisions by a column-group bitmap (CGB).

Our key contributions can be summarized as follows:

- •
We propose a novel binarization framework ARB, designed to progressively align the distribution between binarized and full-precision weights. In addition, we provide rigorous theoretical analyses of the quantization error throughout the progressive updates.
- •
Building on the basic ARB framework, we develop two advanced extensions: ARB with calibration data (ARB-X), and ARB along row-column axes (ARB-RC). They are tailored to address specific challenges in binarized large language models.
- •
We propose a refined strategy to combine the salient column bitmap and group bitmap (CGB), which improves the bitmap utilization and further enhances the performance.
- •
Extensive experiments demonstrate that our ARB-LLM${}_{\text{RC}}$ (ARB-RC + CGB) significantly outperforms SOTA binary PTQ methods while requiring less memory. Furthermore, ARB-LLM${}_{\text{RC}}$, for the first time, surpasses same-size FP16 models on zero-shot QA datasets.

## 2 Related Works

### 2.1 Network Binarization

Network binarization compresses the parameters to only 1 bit ($\pm 1$) by using the sign function.
Then, the straight through estimator (STE) (Bengio et al., ) is used to tackle the gradient vanishing during back-propagation if training a binary network.
Binary weight network (BWN) (Rastegari et al., ) implemented binarization on weights while maintaining full-precision activations. XNOR-Net (Rastegari et al., ) extended this by binarizing weights and activations.
They both focus on standard first-order binarization and employ a scaling factor $\alpha$ to reduce quantization error. Network sketching (Guo et al., ) extended the first-order binarization by proposing a binary coding quantization (BCQ) to approximate the full-precision weights with multiple binary matrices. Xu et al. () improved BCQ by using a binary search tree to determine the optimal code. However, both methods are tailored for scenarios involving multiple binarizations and are not applicable to standard first-order binarization processes.
In another direction, OneBit (Xu et al., ) extended the scaling factor to both weights and activations. BinaryMoS (Jo et al., ) introduced several scaling experts to improve the performance. Nevertheless, training these models requires substantial resources. For example, training OneBit on LLaMA-7B takes 7 days using 8 A100-80GB GPUs.

### 2.2 Large Language Model Quantization

Current quantization techniques for large language models mainly fall into quantization-aware training (QAT) and post-training quantization (PTQ) frameworks.

Quantization-Aware Training (QAT).  QAT integrates quantization into the training process, enabling the model to adapt to low-bit representations. Recent works have successfully applied QAT to LLMs. LLM-QAT (Liu et al., ) addressed data barrier issues in QAT training by adding data-free distillation. EfficientQAT (Chen et al., ) proposed an optimized QAT framework with two stages (i.e., Block-AP and E2E-QP) to reduce QAT’s memory and computational overhead for LLMs. However, QAT still requires considerable computational resources, including significant GPU memory and more training time. Therefore, LLM quantization techniques such as QLoRA (Dettmers et al., ) focused on parameter-efficient fine-tuning methods, which enhanced the efficiency of QAT. Nevertheless, the efficiency of the LLM quantization methods remained unsatisfactory.

Post-Training Quantization (PTQ).  PTQ applied quantization directly to the existing model weights instead of retraining it. Therefore, it is significantly faster and more resource-efficient than QAT. Recent studies have effectively deployed PTQ in LLMs. RTN rounds weights to the nearest quantization level in order to ensure efficient runtimes when quantizing LLMs.
Works like ZerqQuant (Yao et al., ) and BRECQ (Li et al., ) enhanced quantization accuracy by adding additional grouping labels for custom quantization blocks. GPTQ (Frantar et al., ) utilized layer-wise quantization and reduced the quantization error by second-order error compensation.
Moreover, PB-LLM (Shang et al., ), SpQR (Dettmers et al., ), and BiLLM (Huang et al., ) implemented a hybrid approach by selectively quantizing salient weights with low bits while binarizing non-salient weights.
In addition, Smoothquant (Xiao et al., ) proposed a strategy of scaling weight and activation outliers, which simplified quantization. Thereafter, AWQ (Lin et al., ) and OWQ (Lee et al., ) also proposed scale transformations of salient weights for activation features to preserve their model capacity.
Our work belongs to the category of binary PTQ, achieving a significant improvement over the SOTA method BiLLM.

Figure: Figure 4: Overview of our ARB series. ARB: alternating refine mean, row scaling factor, and binarized matrix. ARB-X: introducing calibration data into the update of binarization parameters. ARB-RC: alternating refine row and column scaling factors.
Refer to caption: x4.png

## 3 Method

Overview.  As shown in Figure 4, to progressively align the distribution between binarized and full-precision weights in LLMs, we first propose a framework Alternating Refined Binarization (ARB) in Section 3.1. Based on ARB framework, we propose the Alternating Refined Binarization with calibration data (ARB-X) to enhance the usage of the calibration set, which is crucial for binary LLMs. Additionally, we introduce the Alternating Refined Binarization along row-column axes (ARB-RC) to address the column deviation challenge in LLM weights. These methods are detailed in Sections 3.2 and 3.3, respectively. Finally, we discuss our refined strategy to combine salient column bitmap and group bitmap (CGB) in Section 3.4. Our final models, ARB-LLM${}_{\text{X}}$ and ARB-LLM${}_{\text{RC}}$, are obtained by equipping ARB-X and ARB-RC with CGB respectively.

### 3.1 Alternating Refined Binarization (ARB)

We begin by discussing standard weight binarization in LLMs. For a full-precision weight $\mathbf{W}\in\mathbb{R}^{n\times m}$, we define the objective of binarization as (with dimension broadcasting omitted for simplicity)

$$ $\mathop{\arg\min}\limits_{\alpha,\mathbf{B}}||\widetilde{\mathbf{W}}-\alpha \mathbf{B}||^{2}_{F},\quad\text{where}\,\widetilde{\mathbf{W}}=\mathbf{W}-\mu, \,\mu=\frac{1}{m}\sum_{j=1}^{m}\mathbf{W}_{.j}\,,$ (1) $$

where $\alpha\in\mathbb{R}^{n}$ denotes the row-wise scaling factor, and $\mathbf{B}\in\{+1,-1\}^{n\times m}$ is a binary matrix.

Since the mean of $\mathbf{W}$ is not necessarily zero, a common practice is to apply a row-wise redistribution before binarization. After redistribution, the weights achieve a row-wise zero-mean distribution, which facilitates the binarization process. Under the objective of binarization (Equation 1), the optimal solutions for $\alpha$ and $\mathbf{B}$ can be solved with $\alpha=\frac{1}{m}\sum_{j=1}^{m}|\widetilde{\mathbf{W}}_{.j}|$ and $\mathbf{B}=\operatorname{sign}(\widetilde{\mathbf{W}})$ respectively.
Then we can define the quantization error $\mathcal{L}_{1}$ after binarization as

$$ $\mathcal{L}_{1}=||\mathbf{W}-\widehat{\mathbf{W}}||_{F}^{2},\quad\text{where} \,\widehat{\mathbf{W}}=\alpha\mathbf{B}+\mu\,.$ (2) $$

Moving forward, we aim to investigate how to reduce the quantization error $\mathcal{L}_{1}$. We first define the residual matrix as $\mathbf{R}=\mathbf{W}-\widehat{\mathbf{W}}$. In analyzing the residual matrix $\mathbf{R}$, we observe a distribution shift in $\mathbf{R}$, where the mean of $\mathbf{R}$ is not always zero due to inevitable errors during the binarization process (see Figure 2). To address this, we introduce a correction term $\delta_{\mu}$ to the original mean $\mu$, effectively mitigating the distribution shift. The refined mean is defined as follows:

$$ $\mu_{\text{refine}}=\mu+\delta_{\mu},\quad\text{where}\,\delta_{\mu}=\frac{1}{ m}\sum_{j=1}^{m}\mathbf{R}_{.j}.$ (3) $$

This is equivalent to taking the partial derivative of $\mathcal{L}_{1}$ with respect to $\mu$ and setting it to 0 0, as shown in Figure 4. Since $\mu$ has been updated to $\mu_{\text{refine}}$, the original
$\alpha$ and $\mathbf{B}$ are no longer optimal solutions for quantization error $\mathcal{L}_{1}$. To further minimize the quantization error, the optimal solutions for $\alpha_{\text{refine}}$ and $\mathbf{B}_{\text{refine}}$ can be obtained by setting $\partial\mathcal{L}_{1}/\partial\alpha=0$, leading to the following expressions:

$$ $\alpha_{\text{refine}}=\frac{1}{m}\operatorname{diag}(\mathbf{B}^{\top}( \mathbf{W}-\mu_{\text{refine}})),\quad\mathbf{B}_{\text{refine}}=\operatorname {sign}(\mathbf{W}-\mu_{\text{refine}}).$ (4) $$

After refining $\mu$, $\alpha$, and $\mathbf{B}$, we can obtain the $\widehat{\mathbf{W}}_{\text{refine}}$ as $\widehat{\mathbf{W}}_{\text{refine}}=\alpha_{\text{refine}}\cdot\mathbf{B}_{
\text{refine}}+\mu_{\text{refine}}$.
We find that this parameter update strategy can be extended to an iterative algorithm.

In each iteration, we sequentially update $\mu$, $\alpha$, and $\mathbf{B}$ to ensure they are the optimal solutions under the current quantization error $\mathcal{L}_{1}$. The pseudocode is shown in Algorithm 1, which extends ARB with group mask (a bitmap detailed in Section 3.4). Moreover, we theoretically analyze the quantization error during the ARB process and derive a specific value for the reduced quantization error after $\tau$ iterations, as stated in Theorem 1. The proof is provided in supplementary file.

###### Theorem 1 .

To achieve better quantization precision, we extend ARB to second-order binarization and apply it to salient weights, following BiLLM (Huang et al., ). The second-order binarized matrix $\widehat{\mathbf{W}}$ is

$$ $\widehat{\mathbf{W}}=\alpha_{1}\mathbf{B}_{1}+\alpha_{2}\mathbf{B}_{2}+\mu,$ (6) $$

where $\mu_{1}$ and $\mu_{2}$ can be combined together as $\mu=\mu_{1}+\mu_{2}$.
Based on the first-order ARB, we use Equation 3 to update $\mu$, then sequentially update $\alpha_{1}$ and $\alpha_{2}$ by setting $\partial\mathcal{L}_{1}/\partial\alpha_{1}=0$ and $\partial\mathcal{L}_{1}/\partial\alpha_{2}=0$ respectively, leading to the following formulas:

$$ $\tilde{\alpha}_{1}=\frac{1}{m}\operatorname{diag}(\mathbf{B}_{1}^{\top}( \mathbf{W}-\mu_{\text{refine}}-\alpha_{2}\mathbf{B}_{2})),\quad\tilde{\alpha}_ {2}=\frac{1}{m}\operatorname{diag}(\mathbf{B}_{2}^{\top}(\mathbf{W}-\mu_{\text {refine}}-\tilde{\alpha}_{1}\mathbf{B}_{1})).$ (7) $$

Figure: Algorithm 1 First-Order Alternating Refined Binarization

The final step is to update the binary matrices $\mathbf{B}_{1}$ and $\mathbf{B}_{2}$. The objective of refining $\mathbf{B}_{1}$ and $\mathbf{B}_{2}$ is:

$$ $\widetilde{\mathbf{B}}_{1},\widetilde{\mathbf{B}}_{2}=\mathop{\arg\min}\limits _{\mathbf{B}_{1},\mathbf{B}_{2}}||\mathbf{W}-\mu_{\text{refine}}-\tilde{\alpha }_{1}\mathbf{B}_{1}-\tilde{\alpha}_{2}\mathbf{B}_{2}||_{\ell 1}.$ (8) $$

Since $\mathbf{B}_{1},\mathbf{B}_{2}\in\{+1,-1\}^{n\times m}$, there are only four possible combinations for ($\tilde{\alpha}_{1}\mathbf{B}_{1}+\tilde{\alpha}_{2}\mathbf{B}_{2}$). Thus, we construct a candidate vector $\mathbf{V}=\{-\tilde{\alpha}_{1}-\tilde{\alpha}_{2},-\tilde{\alpha}_{1}+\tilde
{\alpha}_{2},+\tilde{\alpha}_{1}-\tilde{\alpha}_{2},+\tilde{\alpha}_{1}+\tilde
{\alpha}_{2}\}\in\mathbb{R}^{4}$, then use binary search to find the combination that is closest to ($\mathbf{W}-\mu_{\text{refine}}$). The corresponding elements of $\widetilde{\mathbf{B}}_{1}$ and $\widetilde{\mathbf{B}}_{2}$ are then determined accordingly. Detailed pseudocode is provided in supplementary file.

###### Theorem 1 .

### 3.2 ARB with Calibration Data (ARB-X)

Although the ARB algorithm can effectively reduce the quantization error $\mathcal{L}_{1}$, we observe that the weight matrix $\mathbf{W}$ operates in conjunction with the input data to produce the output. It means that $\mathcal{L}_{1}$ alone does not fully capture the true impact of quantization. To address this issue, we introduce calibration data $\mathbf{X}$ and define a new quantization error $\mathcal{L}_{2}$ as $\mathcal{L}_{2}=||\mathbf{W}\mathbf{X}-\widehat{\mathbf{W}}\mathbf{X}||_{F}^{2}$. Based on $\mathcal{L}_{2}$ and the ARB algorithm, we propose an extended algorithm, naming ARB-X.

Reformulation. However, incorporating calibration data necessitates a large number of matrix multiplications when computing $\mathcal{L}_{2}$, substantially increasing computational overhead, and often making the combination of calibration data impractical. To address this issue, we reformulate the error computation by decoupling the calibration data and weight matrix as:

$$ $\mathcal{L}_{2}=\langle\mathbf{S},\mathbf{R^{\top}R}\rangle_{F}=\text{Tr}( \mathbf{R}\mathbf{S}\mathbf{R}^{\top}),\quad\text{where}\,\,\mathbf{S}=\sum \nolimits_{b}\mathbf{X}_{b}^{T}\mathbf{X}_{b}\,,\mathbf{R}=\mathbf{W}-\mu- \alpha\mathbf{B}.$ (9) $$

$\mathbf{X}\in\mathbb{R}^{B\times L\times m}$ denotes the calibration data with batch size $B$, sequence length $L$, and embedding dimension $m$. By compressing the high-dimensional tensor $\mathbf{X}$ into a 2D matrix $\mathbf{S}\in\mathbb{R}^{m\times m}$ and precomputing it, we can significantly reduce the computational overhead.
To quantify the efficiency improvement of our reformulation, we define the speedup ratio $\eta$, which denotes the ratio between the time complexity of the original error computation and that of the revised method. We present the theoretical result in Theorem 2, with the proof provided in supplementary file.

###### Theorem 2 .

Typically, we set $n$ to 4,096, $B$ to 128, $L$ to 2,048, $T$ to 15, and $k$ to 128. Under these circumstances, $\eta$ is proportional to 389, meaning that the reformulated method is approximately 389$\times$ faster than the original one. Further details are provided in supplementary file.

Parameter Update. By combining the parameter updating strategy with the reformulated $\mathcal{L}_{2}$, we can derive the parameter update formulas for ARB-X by setting $\partial\mathcal{L}_{2}/\partial\mu=0$ and $\partial\mathcal{L}_{2}/\partial\alpha=0$,
which results in the sequential updates of $\mu$ and $\alpha$ respectively:

$$ $\displaystyle\mu=\frac{\mathbf{1}^{\top}\mathbf{S}(\mathbf{W}-\alpha\mathbf{B} )^{\top}}{\mathbf{1}^{\top}\mathbf{S}\mathbf{1}},\quad\alpha=\frac{ \operatorname{diag}(\mathbf{B}\mathbf{S}(\mathbf{W}-\mu)^{\top})}{ \operatorname{diag}(\mathbf{B}\mathbf{S}\mathbf{B}^{\top})}.$ (11) $$

More details are provided in supplementary file. It is worth noting that, during this process, the matrix $\mathbf{B}$ is not updated. Since $\mathbf{B}$ consists of discrete values (i.e., +1 and -1), it is not possible to update $\mathbf{B}$ directly by setting the partial derivative of $\mathcal{L}_{2}$ with respect to $\mathbf{B}$ to zero. The pseudocodes for the first-order and second-order ARB-X are provided in supplementary file.

###### Theorem 2 .

### 3.3 ARB along Row-Column Axes (ARB-RC)

Previous binarization methods use a row-wise scaling factor $\alpha^{r}$ for weight binarization. However, our analyses of the numerical distribution of the weight matrix $\mathbf{W}$ in LLMs reveal significant deviations across columns, with some columns exhibiting notably larger values (Figure 3). As a result, using a single row-wise scaling factor may not effectively capture the distribution characteristics of LLM parameters. Additionally, the weight distribution shows a mean close to zero, making the redistribution to zero-mean less effective in LLM binarization.

To address this, we propose the ARB-RC algorithm, which introduces a column-wise scaling factor $\alpha^{c}$ to better handle parameter variations across columns, while eliminating the redistribution parameter $\mu$ to enhance compression in LLMs. The row-column binarization process is performed as follows:

$$ $\alpha^{r}=\frac{1}{m}\sum_{j=1}^{m}|\mathbf{W}_{.j}|,\quad\alpha^{c}=\frac{1} {n}\sum_{j=1}^{n}|\frac{\mathbf{W}_{j.}}{\alpha_{j}^{r}}|,\quad\mathbf{B}= \operatorname{sign}(\mathbf{W}).$ (12) $$

Then, we can obtain the binarized matrix as $\widehat{\mathbf{W}}=\alpha^{r}\alpha^{c}\mathbf{B}$, where removing $\mu$ while introducing $\alpha^{c}$ reduces parameters but improves model performance. However, introducing $\alpha^{c}$ without adopting an alternating parameter update strategy fails to improve performance and can even increase quantization error. Thus, it is necessary to combine $\alpha^{c}$ with the discussed ARB algorithm. In this approach, we optimize the parameters using the quantization error $\mathcal{L}_{1}$. Although the quantization error $\mathcal{L}_{2}$ is more aligned with real-world conditions, our analysis shows that incorporating $\mathbf{X}$ in the ARB-RC method results in parameter coupling, making optimization difficult (detailed in supplementary file). Thus, based on $\mathcal{L}_{1}$, we can update $\alpha^{r}$ and $\alpha^{c}$ by setting $\partial\mathcal{L}_{1}/\partial\alpha^{r}=0$ and $\partial\mathcal{L}_{1}/\partial\alpha^{c}=0$ respectively:

$$ $\displaystyle\alpha^{r}=\frac{\operatorname{diag}(\mathbf{W}(\alpha^{c}\mathbf {B})^{\top})}{\operatorname{diag}((\alpha^{c}\mathbf{B})(\alpha^{c}\mathbf{B}) ^{\top})},\quad\alpha^{c}=\frac{\operatorname{diag}(\mathbf{W}^{\top}(\alpha^{ r}\mathbf{B}))}{\operatorname{diag}((\alpha^{r}\mathbf{B})^{\top}(\alpha^{r} \mathbf{B}))}.$ (13) $$

The first-order and second-order pseudocodes of the ARB-RC are provided in supplementary file.

**Table 1: Perplexity of RTN, GPTQ, PB-LLM, BiLLM, and our methods on OPT family. The columns represent the perplexity results on WikiText2 datasets with different model sizes.**
| Method | Block<br>Size | Weight<br>Bits | 1.3B | 2.7B | 6.7B | 13B | 30B | 66B |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Full Precision | - | 16.00 | 14.62 | 12.47 | 10.86 | 10.13 | 09.56 | 09.34 |
| RTN | - | 3.00 | 13,337.38 | 15,594.72 | 5,797.32 | 3,357.01 | 1,566.00 | 6,126.09 |
| GPTQ | 128 | 3.00 | 16.45 | 13.61 | 11.31 | 10.47 | 09.71 | 10.55 |
| RTN | - | 2.00 | 11,272.65 | 9,505.76 | 28,363.14 | 194,086.78 | 169,616.47 | 1,165,864.25 |
| GPTQ | 128 | 2.00 | 121.64 | 59.53 | 20.81 | 20.05 | 13.04 | 46.38 |
| \cdashline1-9<br>RTN | - | 1.00 | 17,165.72 | 36,516.69 | 11,550.91 | 69,863,488.00 | 6,485.99 | 184,796.30 |
| GPTQ | 128 | 1.00 | 8,719.58 | 11,700.13 | 6,633.13 | 1,743,929.88 | 14,083.15 | 11,045.36 |
| PB-LLM | 128 | 1.70 | 239.81 | 278.27 | 144.25 | 74.59 | 28.30 | 27.66 |
| BiLLM | 128 | 1.11 | 69.05 | 48.61 | 47.65 | 18.75 | 13.86 | 12.05 |
| \cdashline1-9<br>ARB-LLM${}_{\text{X}}$ | 128 | 1.11 | 45.40 | 34.37 | 20.07 | 15.47 | 12.36 | 11.23 |
| ARB-LLM${}_{\text{RC}}$ | 128 | 1.11 | 26.63 | 19.84 | 14.92 | 12.92 | 11.12 | 10.30 |

### 3.4 Column-Group Bitmap (CGB)

Inspired by BiLLM (Huang et al., ), we partition the entire set of weights into salient and non-salient columns, and apply higher-bit representation, i.e., second-order binarization, to the salient weights.
However, different from BiLLM, we not only divide the non-salient weights into sparse and concentrated groups but also divide salient weights in a similar manner. This approach allows for more efficient use of both column bitmap and group bitmap, as shown in Figure 5.

To identify the sensitivity of weights, i.e., salient weights, we follow well-established PTQ methods by utilizing the Hessian matrix as a standard criterion. The sensitivity is computed as $s_{i}=w_{i}^{2}/[\mathbf{H}^{-1}]_{ii}^{2},$ where $\mathbf{H}$ represents the Hessian matrix for each layer, and $w_{i}$ represents the original value of each weight element. Weight columns with higher $s_{i}$ are selected as salient columns, which are then marked using the salient column bitmap. For more details, please refer to Huang et al. ().

Figure: Figure 5: Comparison between BiLLM and our combination of column and group bitmaps.
Refer to caption: x5.png

For non-salient columns, BiLLM further divides them into sparse and concentrated groups based on their magnitude, marking them using a group bitmap. Although this grouping strategy significantly reduces the quantization error, it can be further refined since some regions of the group bitmap are underutilized. As shown on the left side of Figure 5, the salient columns of the group bitmap remain unused. Thus, to better utilize the space of the group bitmap, we optimize the combination of the column bitmap and group bitmap. Specifically, we further categorize the salient weights into sparse and concentrated groups, which improve the quantization accuracy of salient weights without increasing bitmap storage. Our combination format is defined as follows:

$$ $\mathbf{G_{s}}=\mathbf{1}_{n}\mathbf{C_{s}}^{\top}\odot\mathbf{G},\quad\mathbf {G_{ns}}=\mathbf{1}_{n}\mathbf{C_{ns}}^{\top}\odot\mathbf{G},$ (14) $$

where $\mathbf{G_{s}}$ and $\mathbf{G_{ns}}$ represent group bitmaps for salient and non-salient weights, respectively. $\mathbf{C}_{s}$ indicates the salient columns, while $\mathbf{C_{ns}}=\neg\mathbf{C_{s}}$ indicates the non-salient columns. We extend the column bitmap along the row axis and then perform element-wise multiplication with the group bitmap to obtain the final partitions. Experiments demonstrate that our Column-Group Bitmap (CGB) further enhances the quantization performance when applied to ARB algorithms. Additionally, following BiLLM, we adopt the block-wise compensation (Frantar et al., ; Frantar & Alistarh, ) to mitigate quantization errors. For further details, please refer to their papers.

**Table 2: Perplexity of RTN, GPTQ, PB-LLM, BiLLM, and our methods on LLaMA family. The columns represent the perplexity results on WikiText2 dataset with different model sizes. N/A: LLaMA-2 lacks a 30B version, and LLaMA-3 lacks both 13B and 30B versions. *: LLaMA has a 65B version, while both LLaMA-2 and LLaMA-3 have 70B versions.**
| Model | Method | Block<br>Size | Weight<br>Bits | 7B/8B* | 13B | 30B | 65B/70B* |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  | Full Precision | - | 16.00 | 05.68 | 05.09 | 04.10 | 03.53 |
|  | RTN | - | 3.00 | 25.54 | 11.40 | 14.89 | 10.59 |
|  | GPTQ | 128 | 3.00 | 08.63 | 5.67 | 04.87 | 04.17 |
|  | RTN | - | 2.00 | 106,767.34 | 57,409.93 | 26,704.36 | 19,832.87 |
|  | GPTQ | 128 | 2.00 | 129.19 | 20.46 | 15.29 | 08.66 |
| \cdashline2-8<br>LLaMA | RTN | - | 1.00 | 168,388.00 | 1,412,020.25 | 14,681.76 | 65,253.24 |
|  | GPTQ | 128 | 1.00 | 164,471.78 | 131,505.41 | 10,339.15 | 20,986.16 |
|  | PB-LLM | 128 | 1.70 | 82.76 | 44.93 | 23.72 | 12.81 |
|  | BiLLM | 128 | 1.09 | 49.79 | 14.58 | 09.90 | 08.37 |
| \cdashline2-8 | ARB-LLM${}_{\text{X}}$ | 128 | 1.09 | 21.81 | 11.20 | 08.66 | 07.27 |
|  | ARB-LLM${}_{\text{RC}}$ | 128 | 1.09 | 14.03 | 10.18 | 07.75 | 06.56 |
|  | Full Precision | - | 16.00 | 05.47 | 04.88 | N/A | 03.32 |
|  | RTN | - | 3.00 | 542.80 | 10.68 | N/A | 07.53 |
|  | GPTQ | 128 | 3.00 | 6.44 | 05.46 | N/A | 03.88 |
|  | RTN | - | 2.00 | 17,788.94 | 51,145.61 | N/A | 26,066.13 |
|  | GPTQ | 128 | 2.00 | 52.22 | 23.63 | N/A | 08.18 |
| \cdashline2-8<br>LLaMA-2 | RTN | - | 1.00 | 157,058.34 | 47,902.32 | N/A | 160,389.91 |
|  | GPTQ | 128 | 1.00 | 59,758.69 | 22,926.54 | N/A | 14,219.35 |
|  | PB-LLM | 128 | 1.70 | 66.41 | 236.40 | N/A | 28.37 |
|  | BiLLM | 128 | 1.08 | 32.31 | 21.35 | N/A | 13.32 |
| \cdashline2-8 | ARB-LLM${}_{\text{X}}$ | 128 | 1.08 | 21.61 | 14.86 | N/A | 07.88 |
|  | ARB-LLM${}_{\text{RC}}$ | 128 | 1.08 | 16.44 | 11.85 | N/A | 06.16 |
|  | Full Precision | - | 16.00 | 06.14 | N/A | N/A | 02.86 |
|  | RTN | - | 3.00 | 2,194.98 | N/A | N/A | 13,592.69 |
|  | GPTQ | 128 | 3.00 | 18.68 | N/A | N/A | 06.65 |
|  | RTN | - | 2.00 | 1,335,816.13 | N/A | N/A | 481,927.66 |
|  | GPTQ | 128 | 2.00 | 1,480.43 | N/A | N/A | 82.23 |
| \cdashline2-8<br>LLaMA-3 | RTN | - | 1.00 | 1,353,698.38 | N/A | N/A | 375,658.34 |
|  | GPTQ | 128 | 1.00 | 1,121,260.50 | N/A | N/A | 130,516.50 |
|  | PB-LLM | 128 | 1.70 | 73.08 | N/A | N/A | 22.96 |
|  | BiLLM | 128 | 1.06 | 55.80 | N/A | N/A | 66.30 |
| \cdashline2-8 | ARB-LLM${}_{\text{X}}$ | 128 | 1.06 | 31.98 | N/A | N/A | 14.15 |
|  | ARB-LLM${}_{\text{RC}}$ | 128 | 1.06 | 27.42 | N/A | N/A | 11.10 |

## 4 Experiments

### 4.1 Setup

All the experiments are conducted with PyTorch (Paszke et al., ) and Huggingface (Paszke et al., ) on a single NVIDIA A800-80GB GPU. We implement 15 iterations for ARB-LLM${}_{\text{X}}$ and ARB-LLM${}_{\text{RC}}$ to ensure the convergence of binarization parameters. Following Frantar et al. () and Huang et al. (), we use 128 samples from C4 (Raffel et al., ) dataset as calibration data.

Models and Datasets.    We conduct extensive experiments on the LLaMA, LLaMA-2, and LLaMA-3 families (Touvron et al., ), the OPT family (Zhang et al., ), and instruction-tuned LLMs Vicuna (Chiang et al., ). To evaluate the effectiveness of our proposed ARB-LLM${}_{\text{X}}$ (ARB-X + CGB) and ARB-LLM${}_{\text{RC}}$ (ARB-RC + CGB), we measure the perplexity of LLM’s outputs on WikiText2 (Merity et al., ), PTB (Marcus et al., ), as well as a part of the C4 (Raffel et al., ) data. Moreover, we also evaluate the accuracy for 7 zero-shot QA datasets: ARC-c (Clark et al., ), ARC-e (Clark et al., ), BoolQ (Clark et al., ), Hellaswag (Zellers et al., ), OBQA (Mihaylov et al., ), PIQA (Bisk et al., ), and Winogrande (Sakaguchi et al., ).

**Table 3: Perplexity of GPTQ, PB-LLM, BiLLM, and our methods on Vicuna family. The columns represent the perplexity results on WikiText2 datasets with different model sizes.**
| Method | Block<br>Size | Weight<br>Bits | 7B | 13B |
| --- | --- | --- | --- | --- |
| Full Precision | - | 16.00 | 06.34 | 05.57 |
| GPTQ | 128 | 2.00 | 688.08 | 37.97 |
| PB-LLM | 128 | 1.70 | 58.68 | 2,506.44 |
| BiLLM | 128 | 1.08 | 39.36 | 43.39 |
| \cdashline1-5<br>ARB-LLM${}_{\text{X}}$ | 128 | 1.08 | 22.79 | 13.76 |
| ARB-LLM${}_{\text{RC}}$ | 128 | 1.08 | 17.60 | 13.38 |

Baselines.    We mainly compare our ARB series with BiLLM (Huang et al., ), the SOTA PTQ approach on binary LLMs. Other recent PTQ algorithms, such as RTN (round-to-nearest), GPTQ (Frantar et al., ), and PB-LLM (Shang et al., ) are also selected.

### 4.2 Main Results

We follow BiLLM to report the average bit-width of all methods, where our methods have the same bit-width as BiLLM. Table 1 presents the perplexity comparison of the OPT family across different model sizes. It can be observed that both ARB-LLM${}_{\text{X}}$ and ARB-LLM${}_{\text{RC}}$ significantly outperform SOTA BiLLM, and reduce the perplexity by up to 68.7% without increasing weight bit-width.
Table 2 presents the perplexity comparison on LLaMA1&2&3 families, which also suggests the superior performance of our ARB-LLM. It is noteworthy that ARB-LLM${}_{\text{RC}}$ outperforms RTN with 3-bit quantization on some models, such as the LLaMA1&3 families, LlaMA2-70B model, as well as OPT family. Similarly, ARB-LLM${}_{\text{RC}}$ also surpasses GPTQ with 3-bit quantization on OPT-66B model. For instruction-tuned Vicuna comparison shown in Table 3, ARB-LLM${}_{\text{X}}$ and ARB-LLM${}_{\text{RC}}$ also show superior performance, surpassing SOTA binary PTQ method BiLLM for a large margin. Regarding average accuracy on QA datasets, ARB-LLM${}_{\text{X}}$ and ARB-LLM${}_{\text{RC}}$ both significantly outperform previous methods, as shown in Figure 6. More results are provided in the supplementary file.

**Table 4: Ablation study on LLaMA-7B, where all ARB methods are equipped with CGB except for ablation (b). Results are measured by perplexity, with final results highlighted in bold.**
| Method | Calibration<br>update | Row-column<br>update | WikiText2 $\downarrow$ | C4 $\downarrow$ |
| --- | --- | --- | --- | --- |
| BiLLM | - | - | 49.79 | 46.96 |
| \cdashline1-5<br>ARB | ✗ | ✗ | 22.67 | 26.44 |
| ARB-LLM${}_{\text{X}}$ | ✓ | ✗ | 21.81 | 22.73 |
| ARB-LLM${}_{\text{RC}}$ | ✗ | ✓ | 14.03 | 17.92 |

Figure: Figure 6: Average accuracy of 7 zero-shot QA datasets on LLaMA1&2&3 families.
Refer to caption: x6.png

### 4.3 Ablation Study

Effectiveness of Advanced Variants.     To validate the effectiveness of our advanced variants ARB-LLM${}_{\text{X}}$ and ARB-LLM${}_{\text{RC}}$, we compare them with the vanilla ARB algorithm in Table 4a. First, we observe that the vanilla ARB already significantly outperforms BiLLM. Furthermore, by introducing either the calibration update or the row-column update to the binarization process, performance is further improved. This demonstrates that our advanced variants, ARB-LLM${}_{\text{X}}$ and ARB-LLM${}_{\text{RC}}$, can further enhance the performance of binary LLMs based on ARB.

Effectiveness of CGB.     To demonstrate the effectiveness of our column-group bitmap (CGB), we conduct an ablation study in Table 4b. In this study, the absence of CGB does not imply the exclusion of partitioning but rather the use of the partitioning strategy used by BiLLM. The results show that CGB further enhances the performance of both ARB-LLM${}_{\text{X}}$ and ARB-LLM${}_{\text{RC}}$. Notably, even when using BiLLM’s partitioning strategy, our methods significantly outperform BiLLM.

Column Bitmap and Group Bitmap.     We use a column bitmap to differentiate between salient and non-salient weights, and a group bitmap to separate weights based on their magnitude. The combination of column and group bitmaps creates four distinct zones. As shown in Table 4c, we explore the effect of decoupling this combination by using either the column bitmap or the group bitmap individually. It is evident that using the column bitmap or group bitmap only will result in a significant performance drop. Omitting both column bitmap and group bitmap entirely (i.e., #group=1), which reduces the method to naive binarization, leads to complete failure.

Calibration Set Size.     Similar to other PTQ methods, our ARB-LLM requires a small calibration set of just 128 samples. We further incorporate the calibration data into the update of binarization parameters in ARB-LLM${}_{\text{X}}$. To explore the effect of calibration set size on performance, we compare results using different set sizes, as shown in Table 4d. It can be observed that using fewer calibration samples (e.g., 64) results in a performance drop, while increasing the calibration set size from 128 to 256 yields similar results. This indicates that our ARB-LLM${}_{\text{X}}$ requires only a small calibration set. Even with just 64 samples, ARB-LLM${}_{\text{X}}$ significantly outperforms the baseline BiLLM.

ARB Iteration Number.     We use 15 iterations for the main results (Table 1, Table 2, Table 3, and Figure 6), as all parameters have fully converged. To explore the impact of different iteration numbers, we compare results using 1, 3, and 15 iterations in Table 4e. As can be seen, regardless of the iteration number, the perplexity of ARB-LLM${}_{\text{X}}$ and ARB-LLM${}_{\text{RC}}$ significantly outperforms the baseline BiLLM. Increasing the iteration number further reduces perplexity, yet they can achieve superior results even with just one iteration.
Additionally, we visualize the changes in the scaling factor $\alpha$ throughout the alternating iterations to provide further insights in supplementary file.

Group Number.     Following BiLLM (Huang et al., ), we introduce an additional bitmap for grouping weights, which has been demonstrated to enhance performance. To explore the impact of group size, we expand the group bitmap from a 1-bit to a 2-bit system, increasing the number of groups from 2 to 4. As shown in Table 4f, increasing the number of groups leads to better performance, especially for ARB-LLM${}_{\text{X}}$, which outperforms ARB-LLM${}_{\text{RC}}$ with the same number of groups. Yet, this also results in extra storage (about 0.8 GB for LLaMA-7B). In contrast, using only one group (i.e., the first row of Table 4c) results in total failure. Given the additional storage overhead, the 2-group configuration strikes a good balance between performance and memory efficiency.

### 4.4 Time and Memory Analyses

**Table 5: Time comparison between BiLLM and our ARB-LLM methods on LLaMA-7B.**
| Method | CGB | #Iter=1 | #Iter=3 | #Iter=15 |
| --- | --- | --- | --- | --- |
| BiLLM | - | 45 min (#Iter=0) |  |  |
| \cdashline1-5<br>ARB-LLM${}_{\text{X}}$ | ✗ | 52 min | 59 min | 70 min |
| ARB-LLM${}_{\text{X}}$ | ✓ | 72 min | 78 min | 88 min |
| ARB-LLM${}_{\text{RC}}$ | ✗ | 48 min | 49 min | 53 min |
| ARB-LLM${}_{\text{RC}}$ | ✓ | 67 min | 68 min | 76 min |

Time Comparison.     As a binary PTQ framework, ARB-LLM eliminates the need for fine-tuning. The alternating algorithm requires more computation to align the distribution progressively, yet this overhead is acceptable. In Table 5, ARB-LLM${}_{\textbf{RC}}$ with 15 iterations requires only 21 more minutes than BiLLM, while ARB-LLM${}_{\textbf{RC}}$ (without CGB) requires only 3 more minutes than BiLLM using just 1 iteration.
The combination of CGB results in an increase of time overhead, due to the percentile search for optimal splitting.

**Table 6: Memory comparison between FP16, PB-LLM, BiLLM, and our ARB-LLM methods.**
| Method | CGB | LLaMA-7B | LLaMA-13B |
| --- | --- | --- | --- |
| FP16 | - | 13.48 GB | 26.03 GB |
| \cdashline1-4<br>PB-LLM | - | 2.91 GB | 5.33 GB |
| BiLLM | - | 2.93 GB | 5.36 GB |
| \cdashline1-4<br>ARB-LLM${}_{\text{X}}$ | ✗ | 2.93 GB | 5.36 GB |
| ARB-LLM${}_{\text{X}}$ | ✓ | 3.23 GB | 5.95 GB |
| ARB-LLM${}_{\text{RC}}$ | ✗ | 2.63 GB | 4.77 GB |
| ARB-LLM${}_{\text{RC}}$ | ✓ | 2.83 GB | 5.17 GB |

Memory Comparison.     In Table 6, ARB-LLM${}_{\text{X}}$ requires the same amount of memory as BiLLM, yet delivers significantly improved performance. ARB-LLM${}_{\text{RC}}$, which replaces the row-wise mean with a column-wise scaling factor, achieves a higher compression ratio along with better performance. Although the refined column-group bitmap (CGB) strategy requires more memory due to more scaling factors, the combination of ARB-RC and CGB still results in lower storage requirements than BiLLM, while delivering outstanding performance. As shown in Table 6, ARB-LLM${}_{\text{RC}}$ with or without CGB both require less storage than previous methods. The computation formulas can be found in supplementary file.

## 5 Conclusion

In this work, we propose ARB-LLM, a series of alternating refined binarization (ARB) methods for LLMs. Through the analyses of the distribution shift between binarized and full-precision weights, we propose an alternating refinement of binarization parameters to progressively align the weight distribution. Moreover, we extend the basic ARB by equipping the calibration data and scaling along row-column axes, resulting in ARB-X and ARB-RC respectively. Additionally, we propose a refined strategy to better combine the salient column bitmap and group bitmap. Our experiments on multiple open-source LLM families show that the final models ARB-LLM${}_{\text{X}}$ and ARB-LLM${}_{\text{RC}}$ can further push the performance boundary from the SOTA binary PTQ methods.

## Peer Reviews

confidence: 2
contribution: 4
initial_rating: 8
presentation: 4
questions: Does the proposed method perform consistently on 7 zero-shot QA datasets?

Is the binarized model able to get reasonable performance on long context, math and reasoning dataset?
soundness: 4
strengths: 1. The proposed ARB-LLM method is innovative, which also addresses a critical problem in binary quantization.
2. The experiments on various LLM families demonstrate the superiority of the proposed methods.
3. The paper also provides thorough ablation studies the effectiveness of the individual components of ARB-LLM.
summary: This paper proposes a novel post-training quantization method alternating Refined Binarization for quantizing LLM using binary weight. The existing methods suffering the mis-alignment from the distribution of the binarized weight with the full precision counterparts. The existing method also didn't address the column-wise deviation. The proposed framework progressively refines binarization parameters to minimize the quantization error. The proposed framework has 4 components. 1. alternating refined binarization which iteractively update the binarized weight. 2. The proposed method is also improved by calibration data. 3. A more accurate binarization can be achieved by column-wise scaling factor. 4. A column-group bitmap strategy was applied in the paper to improve quantization by maximizing bitmap utilization through salient and non-salient division. The experiment results show impressive results.
weaknesses: 1. The main weakness of the paper is the evaluation seems limited to perplexity and average accuracy. The paper would be more interesting if there were trade-offs on more tasks.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. Can the authors provide any runtime inference metrics to showcase the practical feasibility of the proposed implementation?

2. Could the authors introduce the meaning of each term in equation (6) so that the reader can understand without referring to prior papers?

3. Can the authors clarify the actual memory consumption and compare it to PTQ methods under the same memory budget?
soundness: 3
strengths: The paper identifies and addresses the distribution shift between floating-point weights and binary weights through an iterative refinement method. The study shows that through the ARB process, the quantization error decreases.

The paper further extends the method by adding a calibration dataset and column-wise scales, resulting in better performance compared to BiLLM models.

The paper offers a comprehensive ablation study and time/memory analysis, showing the trade-offs of each component.
summary: The paper describes an Alternating Refined Binarization (ARB) framework for performing post-training quantization (PTQ) of 1-bit for large language models (LLM). The framework is based on the observation that there is a distribution gap between the binarized weights and the floating-point weights. Through the iterative refinement process of ARB, the quantization error can be reduced. The authors also provide two variants that use calibration data and address column deviation. They conduct experiments on OPT and LLAMA models, achieving better results compared to other binary PTQ methods such as BiLLM.
weaknesses: The authors claim that the weight bit is 1.11 bits, but based on the memory comparison, the model takes approximately 3GB of memory on the Llama-7B model, which is similar in size to GPTQ-3bit of the same model, with inferior performance. Can the authors clarify this?

It seems the approach works better on the older OPT model than the Llama family, with Llama3's performance, in particular, being much worse than floating-point. Can the authors showcase the performance on more recent models like Phi/Llama3.2 to address this concern?

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: Q1: Is it ARB second-order binarization is called in the paper " higher-bit representation, i.e., second-order binarizatio".  Is effectively 2bits quantization? 

Q2: Please explain why BiLLM accuracy in Table 2 does not match the results in the original BiLLM paper (shown in Table 3 LLaMA and LLaMA2)
soundness: 3
strengths: Authors proposed several improvements of BiLLM method and show that ARB-LLMRC outperforms FP16 models of the same size (in GB) and outperforms BiLLM (it is SOTA of LLM post training binarization)

Extensive ablation study of the proposed methods showed effectiveness of Effectiveness of ARB, ARB-LLMX, ARB-LLMRC its combination with CGB and calibration data size, iteration number and group number.

Conducted extensive experiments on the LLaMA, LLaMA-2, and LLaMA3 families. 

They did a thorough analysis of time and memory consumption for proposed methods.

They will release all the code and models.
summary: Authors propose to ALTERNATING REFINED BINARIZATION (ARB) which is based on BiLLM paper. They introduce correction term δµ to the original mean µ, effectively mitigating the distribution shift. As a result α, B and µ have to be refined.

They further extend ARB to ARB-X and ARB-RC, where 
ARB-X is extension of ARB with calibration data and loss function which is based on papers Optimal Brain Compression and GPTQ.

ARB-RC introduces a column-wise scaling factor αc to better handle parameter variations across columns, while eliminating the redistribution parameter µ to enhance compression in LLM.

In addition they combine ARB-X and ARB-RC with weight partition strategy based on columngroup bitmap (CGB).
And call final versions as ARB-LLMX and ARB-LLMRC respectively, which outperform state-of-the-art (SOTA) binarization methods for LLMs.

ARB-LLMRC outperforms FP16 models of the same size (in GB).
weaknesses: Q: On Figure 1, authors show Pareto curve and demonstrate that  ARB-LLM-RC outperforms the same-size FP16 models and previously published binarized models. So authors show an improvements over previous models binarization, but it does not show the whole picture in terms of pareto curve: they only compare binarized models vs fp16 (on Figure 1). But it will be informative to show also 4bits and 8bits models. For example paper "The case for 4-bit precision: k-bit Inference Scaling Laws" shows that in terms of pareto curve, for the same model size, 4bits has the best accuracy. In addition to 4bits and 8bits it would be great to plot 3bits too on pareto curve (given that 3bits is already presented in multiple Tables in this paper).

Meta-review: This paper proposes a new Alternating Refined Binarization (ARB) framework for performing post-training quantization (PTQ) of 1-bit for LLM. After rebuttal, it received scores of 588. On one hand, reviewers acknowledge that the proposed method is innovative, which effectively addresses critical challenges in binary quantization. The experiments are comprehensive, with thorough ablation studies. On the other hand, the reviewer who gave a score of 5 still shows concerns regarding runtime and performance. The authors have tried to address the reviewer's concerns during the rebuttal. Overall, the AC thinks that the merits of the paper outweigh the flaws, therefore, decided to recommend acceptance by the end.

Decision: Accept (Poster)