ARB-LLM: Alternating Refined Binarizations for Large Language Models
Authors: Zhiteng Li
Xianglong Yan
Tianao Zhang
Haotong Qin
Dong Xie
Jiang Tian
zhongchao shi
Linghe Kong
Yulun Zhang
Xiaokang Yang
Venue: ICLR 2025 Poster (2025)
Keywords: Binarization
LLM
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We propose ARB-LLM, which alternately updates binarization parameters to reduce quantization error during the binarization of large language models (LLMs).

Abstract: Large Language Models (LLMs) have greatly pushed forward advancements in natural language processing, yet their high memory and computational demands hinder practical deployment. Binarization, as an effective compression technique, can shrink model weights to just 1 bit, significantly reducing the high demands on computation and memory. However, current binarization methods struggle to narrow the distribution gap between binarized and full-precision weights, while also overlooking the column deviation in LLM weight distribution. To tackle these issues, we propose ARB-LLM, a novel 1-bit post-training quantization (PTQ) technique tailored for LLMs. To narrow the distribution shift between binarized and full-precision weights, we first design an alternating refined binarization (ARB) algorithm to progressively update the binarization parameters, which significantly reduces the quantization error. Moreover, considering the pivot role of calibration data and the column deviation in LLM weights, we further extend ARB to ARB-X and ARB-RC. In addition, we refine the weight partition strategy with column-group bitmap (CGB), which further enhance performance. Equipping ARB-X and ARB-RC with CGB, we obtain ARB-LLM$_{\text{X}}$ and ARB-LLM$ _{\text{RC}} $ respectively, which significantly outperform state-of-the-art (SOTA) binarization methods for LLMs.
As a binary PTQ method, our ARB-LLM$ _{\text{RC}} $ is the first to surpass FP16 models of the same size. Code: https://github.com/ZHITENGLI/ARB-LLM.

Published as a conference paper at ICLR 2025
ARB-LLM: ALTERNATING REFINED BINARIZATIONS FOR LARGE LANGUAGE MODELS
Zhiteng : Xianglong Yan' Tianao Zhang', Haotong Qin",
Dong Xie Yulun Zhang't Xiaokang Yang!
Jiang Tian Zhongchao Shi Linghe Kong Shanghai Jiao Tong University, 2ETH Zürich, LLenvvo Research
ABSTRACT
Large Language Models (LLMs) have greatly pushed forward advancements in natural language processing, yet their high memory and computational demands hinder practical deployment. Binarization, as an effective compression technique, can shrink model weights to just 1 bit, significantly reducing the high demands on computation and memory. However, current binarization methods struggle narrow the distribution gap between binarized and full-precision weights, while also overlooking the column deviation LLM weight distribution. To tackle these issues, we propose ARB-LLM, novel 1-bit post-training quantization (PTQ) technique tailored for LLMs. To narrow the distribution shift between binarized and full-precision weights, we first design alternating refined binarization (ARB) algorithm to progressively update the binarization parameters, which significantly reduces the quantization error. Moreover, considering the pivot role of calibration data and the column deviation in LLM weights, we further extend ARB to ARB-X and ARB-RC. In addition, we refine the weight partition strategy with column- group bitmap (CGB), which further enhance performance. Equipping ARB-X and ARB-RC with CGB, we obtain ARB-LLMy and ARB-LLMRC respectively. which significantly outperform state-of-the-arr (SOTA) binarization methods for LLMs. As a binary PTQ method, our ARB-LLMRR the first to surpass FP16 models of the same size. Code: https: //g/ggtuuul com/ ZHITENGLI/
INTRODUCTION
Recently, Transformer- (Vaswani, 2017) large language models have shown impressive performance across various natural language pro- cessing tasks. However, this unprecedented ca- pability is largely attributed to the sheer scale of these models, which often encompass billions of parameters. For instance, open pre-trained Trans- former (OPT) series (Zhang et al., 2022) includes various models, with the largest boasting 66B pa- rameters. Similarly, the I.LMMA family (Touvron et al., 2023) features variants such as I.LaMA3- 70B, showcasing even larger architectures. The substantial memory requirements for inference such large models (e.g., 150 GB memory for 70B model) pose significant challenges for their deployment on mobile devices.
FR16
PS-LLM
0 668
BILLM
668
Avg. Accuracy
ARB-LLMMc (Ours)
6.78
868
8.78
2.71
01.38
A/
2.78
Ar1338
6.78
1.38
2.78
10"
10
10a
Model Size (GB, log scale)
Figure 1: OPT performance 7 zero-shot Question Answering (QA) datasets. Our ARB- LLMgc outperforms the same-size FP16 models.
The study of compressing LLMs can be categorized into weight quantization (Lin et al., 2024; Frantar al., 2023), low-rank factorization (Zhang al., 2024; Yuan al., 2023), network pruning (Sun et al., 2024; Frantar & Alistarh, 2023), and knowledge distillation (Zhong et al., 2024; Gu et al., 2024). Among these, binarization, a specific technique within the realm of quantization, is particularly distinguished for its ability to achieve extreme memory compression, reducing storage requirements to low bit. Given the substantial size of LLMs, some binarization methods adopt the post-training quantization (PTQ) framework to enable rapid transition from full- preciseciiionn models to compact binarized versions, requiring minimal resources (e.g., binarizing 70B model in one 80 GB GPU).
"qqual contribution Corresponning authors: Linghe Kong, kong@ Yulun Zhang, yulun 100@ @m0.l.ml.com

---

Published as a conference paper at ICLR 2025
Recent binary PTQ methods, such as PB-LLM (Shang et al., 2024) and BiLLM (Huang et al., 2024), emphasize the identification of salient weights, which are crucial for model performance (Lin et al., 2024). The higher-bit representation and refined searching strategy for salient weights help to achieve a better trade-off between performance and storage. Despite their success, the refinement of the binarization process itself remains largely unaddressed, resulting in significant difference between the binarized weights and their full-precision counterparts. This gap presents considerable obstacle to further enhance the performance of binary LLMs.
To minimize quantization error during the binarization process, we revisit the solutions for the binarization ob- jective. Our analyses reveal that: (i) The current approach suboptimal due to the distribution shift between bina- rized and full-precision weights after binarization. As shown in Figure 2, the mean of the binarized weights is not aligned with the full-precision mean. Consequently, refining the binarization parameters based on the initial distribution of the binarized weights can yield more
Distribution
Shift 6,
oiferrsciionn Mesta
Full-grccision
Binarized
Binsriized
Row
curate estimation of the original weights. Furthermore, this refinement can be alternately applied to different binarization parameters, ultimately leading to a signifi- cantly improved estimation. (ii) While the calibration set is small. it plays a crucial role in the quantization of LLMs. However, the integration of calibration data for updating binarization parameters, which reflects a more realistic scenario, remains underexplored. (iii) The weight distribution in LLMs exhibits noticeable column- wise deviations (see Figure 3), suggesting that the stan- dard row-wise binarization method is inflexible and potentially unsuitable. Thus, incorporat- ing both row and column scaling factors can produce more representative binarization results.
-0.010
0.000
0.005
0.010
Value
Distribution shift between the mean of binarized and full-precision weights. Top: distribution shift of one row. Bottom:
Figure 2:
distribution shifts of mul- tiple rows. Each row represents a top view of the corresponding upper image.
With the above observations and anal- yses, we first propose Alternating Re- fined Binarization (ARB) to align the distribution between binarized and full-precision weights in standard bi- narization process. Then, we ex- tend this approach by incorporating the calibration data and row-column- wise scaling factors, leading to two advanced extensions: ARB-X and ARB-RC. Additionally, based on pre- vious methods, which divide salient and non-salient weights and group weights by magnitude, we refine the integration of these two divisions by a column-group bitmap (CGB).
Full-precision Weights
BiLLM
2200
Row
Column
41
ARB-RC
07
4000
Value
Row
Column
Row
Column
Figure 3: Left: Full-precision weights exhibit column-wise deviations. Right: BiL.LM (Huang et al., 2024) with row- wise binarization smooths the deviations. Our ARB-RC with row-column-wise binarization effectively preserves them.
Our key contributions can be summarized as follows:
We propose a novel binarization framework ARB, designed to progressively align the distribution between binarized and full-precision weights. In addition, provide rigorous theoretical analyses the quantization error throughout the progressive updates.
Building on the basic ARB framework, we develop two advanced extensions: ARB with calibration data (ARB-X), and ARB along row-column axes (ARB-RC). They are tailored to address specific challenges in binarized large language models.
We propose refined strategy to combine the salient column bitmap and group bitmap (CGB), which improves the bitmap utilization and further enhances the performance.
Extensive experiments demonstrate that our ARB-LLMRC (ARB-RC CGB) significantly outperforms SOTA binary PTQ methods while requiring less memory. Furthermore, ARB- L.MRRc. for the first time, surpasses same-size FP16 models on zero-shot QA datasets.
2

---

Published as a conference paper at ICLR 2025
2
RELATED WORKS
2.1 NETWORK BINARIZATION
Network binarization compresses the parameters to only 1 bit (+11) by using the sign function. Then, the straight through estimator (STE) (Bengio et al., 2013) is used to tackle the gradient vanishing during back-propagation if training a binary network. Binary weight network (BWN) (Rastegari et al., 2016) implemented binarization on weights while maintaining full-precision activations. XNOR- Net (Rastegari et al., 2016) extended this by binarizing weights and activations. They both focus on standard first-order binarization and employ a scaling factor a to reduce quantization error. Network sketching (Guo et al., 2017) extended the first-order binarization by proposing a binary coding quantization (BCQ) to approximate the full-precision weights with multiple binary matrices. Xu et al. (2018) improved BCQ by using binary search tree to determine the optimal code. However, both methods are tailored for scenarios involving multiple binarizations and are not applicable to standard first-order binarization processes. In another direction, OneBit (Xu et al., 2024) extended the scaling factor to both weights and activations. BinaryMoS (Jo et al., 2024) introduced several scaling experts to improve the performance. Nevertheless, training these models requires substantial resources. For example, training OneBit on LLaMA-7B takes 7 days using 8 A A100-80GB GPUs.
2.2 LARGE LANGUAGE MODEL QUANTIZATION
Current quantization techniques for large language models mainly fall into quantization- aware training (QAT) and post-training quantization (PTQ) frameworks.
Quantization-Aware Training (QAT). QAT integrates quantization into the training process, enabling the model to adapt to low-bit representations. Recent works have successfully applied QAT to LLMs. LLM- LLM-QAT (Liu et al., 2024) addressed data barrier issues in QAT training by adding data-free distillation. EfficientQAT (Chen et al., 2024) proposed an optimized QAT framework with two stages (i.e. Block-AP and E2E-QP) to reduce QAT's memory and computational overhead for LLMs.. However, QAT still requires considerable computational resources, including significant GPU memory and more training time. Therefore, LLM quantization techniques such as QLoRA (Dettmers al., 2024a) focused on parameter-efficient fine-tuning methods, which enhanced the efficiency of QAT. Nevertheless, the efficiency of the LLM quantization methods remained unsatisfactory.
post-Training Quantization (PTQ). PTQ applied quantization directly to the existing model weights instead of retraining it. Therefore, it is significantly faster and more resource-efficient than QAT. Recent studies have effectively deployed PTQ in LLMs. RTN rounds weights to the nearest quantization level in order to ensure efficient runtimes when quantizing L.Mss. Works like ZeroQuant (Yao et al., 2022) and BRECQ (Li et al., 2021) enhanced quantization accuracy by adding additional grouping labels for custom quantization blocks. GPTQ (Frantar al., 2023) utilized layer-wise quantization and reduced the quantization error by second-order error compensation. Moreover, PB-LLM (Shang al., 2024), SpQR (Dettmers et al., 2024b), and BiLLM (Huang et al., 2024) implemented a hybrid approach by selectively quantizing salient weights with low bits while binarizing non-salient weights. In addition, Smoothquant (Xiao et al., 2023) proposed strategy of scaling weight and activation outliers, which simplified quantization. Thereafter, AWQ (Lin al.. 2024) and OWQ (Lee al.. 2024) also proposed scale transformations of salient weights for activation features to preserve their model capacity. Our work belongs to the category of binary PTQ. achieving significant improvement over the SOTA method BiLLM.
METHOD
Overview. As shown in Figure 4, to progressively align the distribution between binarized and full-precision weights in LLMs, we first propose a framework Alternating Refined Binarization (ARB) in Section 3.1. Based on ARB framework, we propose the Alternating Refined Binarization with calibration data (ARB-X) to enhance the usage of the calibration set, which is crucial for binary LLMs. Additionally, we introduce the Alternating Refined Binarization along row-column axes (ARB-RC) to address the column deviation challenge in LLM weights. These methods are detailed in Sections 3.2 and 3.3, respectively. Finally, we discuss
refined strategy to combine salient column bitmap and group bitmap (CGB) in Section 3.4. Our final models, ARB-LLMy and ARB-LLMRC. obtained by equipping A ARB-X and ARB-RC with CGB respectively.
3

---

Published as a conference paper at ICLR 2025
Input
Initialization
Alternating Refinement
Output
ARB
Ar
F. [w- #1
ARB-X
8.4 -
ICg -0
84g
02
-0
ARB-RC
dG
26
@ of
6- WT
42
02
5hhr
Figure 4: Overview of our ARB series. ARB: alternating refine mean, row scaling factor, and binarized matrix. ARB-X: introducing calibration data into the update of binarization parameters. ARB-RC: alternating refine row and column scaling factors.
3.1
ALTERNATING REFINED BINARIZATION (ARB)
We begin by discussing standard weight binarization in LLMs. For a full-precision weight W E Rnxm, we define the objective of binarization as (with dimension broadcasting omitted for simplicity)
1
m
arg min o,B
| where W W M4 P
Wi
(1)
m
where a E I2' denotes the row-wise scaling factor, and B E [+1,-1) is a binary matrix.
Since the mean of W is not necessarily zero, common practice is to apply row-wise redistribution before binarization. After redistribution, the weights achieve a row-wise zero-mean distribution, which facilitates the binarization process. Under the objective of binarization (Equation (1)), the optimal solutions for a and B can be solved with a 1 Then we can define the quantization error Li after binarization
m Σラ [W of and B sign( () respectively,
L I W where W =B cB + n..
Moving forward, we aim to investigate how to reduce the quantization error L1. We first define the residual matrix as R = W- W. In analyzing the residual matrix R, we observe a distribution shift in R. where the mean of R is not always zero due to inevitable errors during the binarization process (see Figure 2). To address this, we introduce a correction term 6, to the original mean μ,, effectively mitigating the distribution shift. The refined mean is defined as follows:
Prefine p.. + 8p where 5,
m Σ R J.
1
This is equivalent to taking the partial derivative of L1 with respect to μ and setting it to 0, as shown Figure 4. Since μ has been updated to Prefines the original a and are no longer optimal solutions for quantization error L1. To further minimize the quantization error, the optimal solutions for O'efinee and Byeline can be obtained by setting 0C1/Oa = O, leading to the following expressions:
diag (B (W - -rtelem))), B = sign W - -Frreline)
Oreline
m
After refining μ,, a, and B, that this parameter update strategy can be extended to an iterative algorithm.
obtain the W Wrefin W refine Orefine Brefine + Prefine We find
In each iteration, we sequentially update 14, and B to ensure they the optimal solutions under the current quantization error L.. The pseudocode is shown in Algorithm 1, which extends ARB with group mask (a bitmap detailed in Section 3.4). Moreover, error during the ARB process and derive a specific value for the reduced quantization error after т iterations, stated in Theorem 1. The proof is provided in supplementary file.
theoretically analyze the quantization
4

---

Published as a conference paper at ICLR 2025
Algorithm 1 First-Order Alternating Refined Binarization
func ARB' (W. RW. M, T)
fune binary (W, M) 1
Input: W E R=xx full--preee pelepiseeiioon weight
I: J m 2j= (W O MM,
M E Raxm group mask
W W =M
T - total iterations
Output: W E RR
3: a = 1 ش 1( (WOM).)
W.a, a, B binary (W.M)
4: B = sign W C M)
2: for iter = 2..,.... do
5: W = a B +
3:
R W-W W
residual matrix
6: return W, a. B,
4:
6μ (, M)
func refine-alpha (B, W. M,
5:
μ < ò 8p
refine mean reline.alpha( W, M)
1: num = ΣØ (B @ M,
(W., -1)
6:
7:
<- sigu(
refine
2: den = ブド=1 (B MM, +c
+c b avoid
8: W f t-a++
zero-division
9: end for
3: = num den
4: return a
return W
Theorem 1. For any 2 0, Algorithm / achieves quantizatiio error L satisfying Ci = C m((a")² - (a (μ² (1))) s Ct. where a and nf denote the initial scaling factor and mean respectively, a", μ², and E represent the scaling factor, mean, and quantizution error after the T-th iteration respectively.
(5)
To achieve better quantization precision, we extend ARB to second-order binarization and apply it to salient weights, following BiLLM (Huang et al., 2024). The second-order binarized matrix W is
02 B1 + 0g2 1 where B,, E -1]mmm are binary matrices, 01,02 E R are corresponding row-wise scaling factors, and μ E Rn is the row-wise shifting factor. Based the first-order ARB, use Equation 3 to update μu, then sequentially update a₁ and a2 by setting 0L,/0ur = 0 and 0L1/0a2 respectively, leading to the following formulas:
1 d (W- - -reetine a2)))), ă2 diag(B) By (W Pretine B, (7) m
â1 m
The final step is to update the binary matrices B and Bg. The objective of refining B₁ and B is:
B B2 arg min || Prefin ā B1 B2lei...
B,,B₂
Since Bi, B2 E (+1,-1)axm there are only four possible combinations for (ā B B3++gBB Thus, we construct a candidate vector V= {-ā 02, -ū + 02, +ā; 22, +ā; + 22 } E R then use binary search find the combination that is closest to (W Mretine.. The corresponding elements of B, and B2 are then determined accordingly. Detailed pseudocode is provided in supplementary file.
3.2 ARB WITH CALIBRATION DATA (ARB-X)
Although the ARB algorithm can effectively reduce the quantization error Li, we observe that the weight matrix W operates in conjunction with the input data to produce the output. It means that C alone does not fully capture the true impact of quantization. To address this issue, we introduce
calibration data X and define a new quantization eтооr L3 as Lg = WX- - WXI Based on L2 and the ARB algorithm, we propose an extended algorithm, naming ARB-X.
Reformulation. However, incorporating calibration data necessitates a large number of matrix multiplications when computing L2, substantially increasing computational overhead, and often making the combination of calibration data impractical. To address this issue, we reformulate the error computation by decoupling the calibration data and weight matrix as:
C. = RS R) F = Tr(RSR )) where S = Σ, X R= R- -
X E R fxxm denotes the calibration data with batch size B, sequence length L, and embedding dimension m. By compressing the high-dim nigh-dimensional tensor X into a 2D matrix S Rm and precomputing it, can significantly reduce the computational overhead. To quantify the efficiency
5

---

Published as a conference paper at ICLR 2025
improvement of our reformulation, define the speedup ratio n), which denotes the ratio between the time complexity of the original error computation and that the revised method. We present the theoretical result in Theorem 2, with the proof provided in the supplementary file.
Theorem 2. The speedup ratio of the reformulation compared to the original method is
X
(10)
k. -n (
B-L
where n the hidden dimension of W, is the block size, and the number of iterations.
Typically, we set 4,096, to 128, L 2,048, T 15, and 128. Under these circumstances, is proportional to 389, meaning that the reformulated method is approximately 389x faster than the original one. Further details are provided in supplementary file.
Parameter Update. By combining the parameter updating strategy with the reformulated L2. we can derive the parameter update formulas for ARB-X by setting 0C2/0p 0 and 0L2/da
which results in the sequential updates of f and respectively:
(( aB) a diag BS(W - -
/
ST
diag BSB
(11)
More details are provided in supplementary file. It is worth noting that, during this process, the matrix B is not updated. Since B consists of discrete values (i.e. +1 and -I), is not possible update B directly by setting the partial derivative of Ly with respect to B to zero. The pseudocodes for the first-order and second-order ARB-X are provided in supplementary file.
3.3 ARB ALONG ROW-COLUMN AXES (ARB-RC)
Previous binarization methods row-wise scaling factor a" for weight binarization. However, our analyses of the numerical distribution of the weight matrix W in LLMs reveal significant deviations across columns, with some columns exhibiting notably larger values (Figure 3). As result, using single row-wise scaling factor may not effectively capture the distribution characteristics of LLM parameters. Additionally, the weight distribution shows a mean close to zero, making the redistribution to zero-mean less effective in LLM binarization.
To address this, we propose the ARB-RC algorithm, which introduces a column- wise scaling factor a to better handle parameter variations across columns, while eliminating the redistribution parameter Ii to enhance compression in LLMs. The row-column binarization process is performed as follows:
-== n
W
a"
Wal a
B = sign(W):
(12)
Then, can obtain the binarized matrix ==aaaa where removing μ while introducing af reduces parameters but improves model performance. However, introducing of without adopting an alternating parameter update strategy fails to improve performance and can even increase quantization eтror. Thus, it is necessary to combine a with the discussed ARB algorithm. In this approach, we optimize the parameters using the quantization error L1. Although the quantization error La is more aligned with real-world conditions, our analysis shows that incorporating X in the ARB-RC method results in parameter coupling, making optimization difficult (detailed in supplementary file). Thus, based on LL, we can update a" and af by setting 0L,/da" = 0 and 0L 0 respectively:
diag( ((aa T)
a diag W (a ())
a" diag((a (a
(13)
diag((a T
The first-order and second-order pseudocodes of the ARB-RC are provided in supplementary file.
3.4 COLUMN-GROUE BITMAP (CGB)
Inspired by BiLLM (Huang et al.. 2024), we partition the entire set of weights into salient and non-salient columns, and apply higher-bit representation, i.e., second-order binarization, to the salient weights. However, different from BiLLM, not only divide the non-salient weights into sparse and concentrated groups but also divide salient weights in a similar manner. This approach allows for more efficient use of both column bitmap and group bitmap, as shown in Figure 5.
To identify the sensitivity of weights, i.e., salient weights, we follow well-established PTQ methods by utilizing the Hessian matrix standard criterion. The sensitivity is computed as 8; w / HH H k0

---

Published as a conference paper at ICLR 2025
Table 1: Perplexity of RTN, GPTQ, PB-LLM, BiLLM, and our methods on OPT family. The columns
represent the perplexity results on WikiText2 datasets with different model sizes.
Block Size
Weight Bits
Method
1.3B
2.7R
6.7B
13B
30B
66B
Full Precision
16.00
14.62
12.47
10.86
10.13
9.56
9.34
RTN
3.00
13,337.38
15,594,72
5,797.32
3,357.01
1,566.00
6,126.09
GPTQ
128
3.00
16.45
13.61
11.31
10.47
9.71
10.55
RTN
2.00
11,272.65
9,505.76
28,363.14
194,086.78
169.616.47
1,165,864,25
GPTQ
128
2.00
121.64
59.53
20.81
20.05
13.04
46.38
RTN
1.00
17.165.72
36,516.69
11,550,91
69.863,488
6,485,99
184,796.30
GPTQ
128
1.00
8,719.58
11,700.13
6.633.13
1,743,929.88
.88
14.083.15
11,045.36
PR-LLM
128
1.70
239.81
278.27
144.25
74.59
28.30
27.66
BiLLM
128
1.11
69.05
48.61
47.65
18.75
13.86
12.05
ARB-LLMX
128
1.11
45,40
34.37
20.07
15.47
12.36
11.23
ARB-LLMRc
128
1.11
26.63
19.84
14.92
12.92
11.12
10.30
where H represents the Hessian matrix for each layer, and w; represents the original value of each weight element. Weight columns with higher s; are selected salient columns, which are then marked using the salient column bitmap. For more details, please refer to Huang et al. (2024).
salient group partition
For non-salient columns, BiLLM further divides them into sparse and concentrated groups based on their magnitude, marking them using a group bitmap. Although this grouping strategy signif- icantly reduces the quantization error, it can be further refined since some regions of the group bitmap are underutilized. As shown on the left side of Figure 5, the salient columns of the group bitmap remain unused. Thus, to better utilize the space of the group bitmap, we opti- mize the combination of the column bitmap and further categorize
group bitmap (4096x128)
salient column bitmap 10 Hoo
1
group bitmap. Specifically, the salient weights into sparse and concentrated groups, which improve the quantization accuracy of salient weights without increasing bitmap stor- age. Our combination format is defined as follows:
non-salient columns
non-salient column bitmap
non-salient group partition
BiLLM
Ours
Figure 5: Comparison between BiLLM and our combination of column and group bitmaps.
G 1, C T @ G, Gns Cns T @GG. G,
(14)
where G. and Gns represent group bitmaps for salient and non-salient weights, respectively. C, indicates the salient columns, while C.s -C indicates the non-salient columns. We extend the column bitmap along the row axis and then perform element-wise multiplication with the group bitmap to obtain the final partitions. Experiments demonstrate that our Column- Groun Bitmap (CGB) further enhances the quantization performance when applied to ARB algorithms. Additionally, following BiLLM, we adopt the block wise compensation (Frantar et al., 2023; Frantar & Alistarh, 2022) to mitigate quantization errors. For further details, please refer to their papers.
EXPERIMENTS
4.1
SETUP
All the experiments are conducted with PyTorch (Paszke et al., 2019b) and Huggingface (Paszke et al., 2019a) on single NVIDIA A800-80GB GPU. We implement 15 iterations for ARB-LLMy and ARB-LLMgC to ensure the convergence binarization parameters. Following Frantar et al. (2023) and Huang al. (2024), we 128 samples from C4 (Raffel al., 2020) dataset calibration data.
Models and Datasets. We conduct extensive experiments the I.LMM,, ILAMA-22, and I.L.MA- 3 families (Touvron et al., 2023), the OPT family (Zhang et al., 2022), and instruction-tumed I. I.LMs Vicuna (Chiang et al., 2023). To evaluate the effectiveness our proposed ARB-LLMy (ARB-X
+ CGB) and ARB-LLMPc (ARB-RC + CGB), we measure the perplexity of LLM's outputs on WikiText2 (Merity al., 2017). PTB (Mareus et al., 1994), as well as a part of the C4 (Raffel et al., 2020) data. Moreover, we also evaluate the accuracy for 7 zero-shot QA datasets: ARC-c (Clark al., 2018), ARC-e (Clark et al., 2018), BoolQ (Clark et al., 2019), Hellaswag (Zellers et al.. 2019). OBQA (Mihaylov et al., 201 8), PIQA (Bisk et al., 2020), and Winogrande (Sakaguchi et al., 2020).
7

---

Published as a conference paper at ICLR 2025
Table 2: Perplexity of RTN, GPTQ, PB-LLM, BiLLM, and our methods on LLaMA family. The columns represent the perplexity results on WikiText2 dataset with different model sizes. N/A: LLaMA-2 lacks a 30B version, and LLaMA-3 lacks both 13B and 30B versions. #: LLaMA has 65B version, while both LLaMA-2 and LLaMA-3 have 70B versions.
Block
Weight Bits
Model
Method
7B/8B
13B
30B
65B/70B
Size
Full Precision
16.00
5.68
5.09
4.10
3.53
3.00
25.54
11.40
14.89
GPTQ
128
3.00
8.63
5.67
4.87
4.17
RTN
2.00
106,767.34
57,409.93
26,704.36
19.832,87
GPTQ
128
2.00
129.19
20.46
15.29
8.66
LLaMA
RTN
1.00
168,388.00
1,412,020.25
14,681.76
65,253.24
GPTQ
128
1.00
164,471.78
131,505.41
10,339.15
20,986,16
PB-LLM
128
1.70
82.76
44.93
23.72
12.81
BiLLM
128
1.09
49.79
14.58
9.90
8.37
ARB-LLMx
128
1.09
21.81
11.20
8.66
7.27
ARB-LLMgC
128
1.09
14.03
10.18
7.75
6.56
Full Precision
16.00
5.47
4.88
N/A
3.32
RTN
3.00
542.80
10.68
N/A
7.53
GPTQ
3.00
6.44
5.46
N/A
3.88
RTN
2.00
17,788.9 94
51,145.61
N/A
26,066.13
GPTQ
128
2.00
52.22
23.63
N/A
8.18
LLaMA-2
RTN
1.00
157.058.34
47,902.32
N/A
160,389.91
GPTQ
128
1.00
59,758.69
22,926.54
N/A
14,219.35
PB-LLM
128
1.70
66.41
236.40
N/A
28.37
BiLLM
128
1.08
32.31
21.35
N/A
13.32
A ARB-LLMX
128
1.08
21.61
14.86
N/A
7.88
ARB-LI RBB-LLMRC
128
1.08
16.44
11.85
N/A
6.16
Full Precision
16.00
6.14
N/A
N/A
2.86
RTN
3,00
2,194,98
N/A
N/A
13,592,69
GPTQ
128
3.00
18.68
N/A
N/A
6.65
RTN
2.00
1,335,816.13
N/A
N/A
481,927.66
GPTQ
2.00
1,480.43
N/A
82.23
LLaMA-3
RTN
1.00
1,353,698.38
N/A
N/A
375,658.34
GPTQ
1.00
1,121,260.50
N/A
N/A
130,516.50
PB-LLM
128
1.70
73.08
N/A
N/A
22.96
BiLLM
128
1.06
55.80
N/A
N/A
66.30
ARB-LLMx
1.06
31.98
N/A
N/A
14.15
ARB-LLMRC
128
1.06
27.42
N/A
N/A
11.10
Baselines. We mainly compare our ARB series with BiLLM (Huang al., 2024), the SOTA PTQ approach on binary LLMs. Other recent PTQ algorithms, such RTN (round- to-nearest), GPTQ (Frantar et al., 2023), and PB-LLM (Shang et al., 2024) are also selected.
Table Perplexity of GPTQ, PB-LLM, BiLLM, and our methods Vicuna family. The columns represent the perplexity results on WikiText2 datasets with different model sizes.
Block
Weight Bits
Method
13B
Size
Full Precision
16.00
6.34
5.57
4.2 MAIN RESULTS
GPTQ
128
2.00
688.08
37.97
We follow BiLLM report the average bit- width of all methods, where our methods have the same bit-width as BiLLM. Table presents the perplexity comparison of the OPT family across different model sizes. It can be observed
PB-LLM
128
1.70
58.68
2,506.44
BiLLM
128
1.08
39.36
43.39
ARB-LLMX
128
22.79
13.76
ARB-LLMRE
128
1.08
17.60
13.38
that both BLB- and ARB-LLMRC significantly outperform SOTA BiLLM, and reduce the perplexity by up to 68.7 % without increasing weight bit-width. Table 2 presents the perplexity com- parison LLaMA & families, which also suggests the superior performance of our ARB-LLM. is noteworthy that RRB-LLMRC outperforms RTN with 3-bit quantization on some models, such the I.LMMA&& &3 families, LlaMA2-70B model, well OPT family. Similarly, ARB-LLMMR
8

---

Published as a conference paper at ICLR 2025
Table Ablation study on LLaMA-7B, where all ARB methods are equipped with CGB except for ablation (b). Results are measured by perplexity, with final results highlighted in bold.
(a) Effectiveness of two advanced variants
(b) Effectiveness of CGB
Calibration
Row-column update
Method
CGB
WikiText2
C4
Method
WikiText2
C4
update
BiLLM
49.79
46.96
BiLLM
49.79
46,96
ARB-LLMx
26.29
27.11
ARB
22.67
26.44
ARB-LLMX
22.73
ARB-LLME
21.81
22.73
ARB-LLMRc
X
15.85
19.42
ARB-LLMac
X
14.03
17.92
ARB-LLMRc
/
14.03
17.92
(c) Study of decoupling column and group bitmaps
(d) Study of ARB-LLMy calibration set size
Column bitmap
Group bitmap
Calibration set size
Method
WikiText2
C4
Method
WikiText2
C4
ARB-I M.
x
10,942.45
11,032,93
BiLLM
128
49.79
46.96
ARB-LLMRc
369.20
205.56
ARB-LLMx
64
24.79
25.11
ARB 1.LMMcC
X
920.42
572.69
ARB-LLMX
128
21.81
22.73
ARB-LLMac
14.03
17.92
ARB-LLMx
256
21.88
24.28
(e) Study of ARB-LLM iteration number
(I) Study of ARB-LLM group number
Method
#lteration
WikiText2
Method
#Group
WikiText2
)
BiLLM
0
49.79 22.59721.12
BiLLM
2
49.79
46.96
ARB-LLMX
T/3715 1/3/15
21.81
ARB-LLMx
274
22.73 8.56
ARB-LLMac
15.23 14.34 / 14.03
ARB-LLMac
2/4
14,03 12.77
17.92 16.06
Avg. Accuracy
70
70
70
60
60
60
50
50
50
Av9.00
440
7B
13B
30B
65B
7B
13B
70B
8B
70B
LLaMA
LLaMA-2
LLaMA-3
GPTQ (2-bit)
PB-LLM
BiLLM
ARB-LLMX
ARB-LLMRc
Figure 6: Average accuracy of 7 zero-shot QA datasets on LLaMAI families.
also surpasses GPTQ with 3-bit quantization on ОРT-66B model. For instruction-tumed Vicuna comparison shown in Table 3, ARB-LLMx and ARB-LLMBC also show superior performance, surpassing SOTA binary PTQ method BiLLM for large margin. Regarding average accuracy on QA datasets, ARB-LLMy and ARB-LLMRC both significantly outperform previous methods, as shown in Figure 6. More results are provided in the supplementary file.
4.3 ABLATION STUDY
Effectiveness of Advanced Variants. To validate the effectiveness of our advanced variants ARB- LLMx and ARB-LLMgC- we compare them with the vanilla ARB algorithm in Table 4a. First, we observe that the vanilla ARB already significantly outperforms BiLLM. Furthermore, by introducing either the calibration update or the row-column update to the binarization process, performance is further improved. This demonstrates that advanced variants. ARB-LLMx and ARB-LLMRC. can further enhance the performance of binary L.Mss based on ARB.
Effectiveness of CGB. To demonstrate the effectiveness of our column-group bitmap (CGB), we conduct an ablation study in Table 4b. this study, the absence of CGB does not imply the exclusion of partitioning but rather the use of the partitioning strategy used by BiLLM. The results show that CGB further enhances the performance of both ARB-LLMy and ARB-LLMBC.. Notably, even when using BiLLM's partitioning strategy, our methods significantly outperform BiLLM..
Column Bitmap and Group Bitmap. We use column bitmap to differentiate between salient and non-salient weights, and a group bitmap to separate weights based on their magnitude. The combination of column and group bitmaps creates four distinct zones. As shown in Table 4c, we explore the effect of decoupling this combination by using either the column bitmap or the group bitmap individually. It is evident that using the column bitmap group bitmap only will result in a significant performance drop. Omitting both column bitmap and group bitmap entirely (i.e., #group=1), which reduces the method naive binarization, leads to complete failure.

---

Published as a conference paper at ICLR 2025
Calibration Set Size. Similar to other PTQ methods, our ARB-LLM requires small calibration set of just 128 samples. We further incorporate the calibration data into the update of binarization parameters in ARB-LLMx. To explore the effect of calibration set size on performance, we compare results using different set sizes, as shown in Table 4d. It can be observed that using fewer calibration samples (e.g., 64) results in a performance drop, while increasing the calibration set size from 128 to 256 yields similar results. This indicates that our ARB-LLMX requires only small calibration set. Even with just 64 samples, ARB-LLMx significantly outperforms the baseline BiLLM.
ARB Iteration Number. We use 15 iterations for the main results (Table Table 2, Table 3, and Figure 6), as all parameters have fully converged. To explore the impact of different iteration numbers, we compare results using 1, 3, and 15 iterations in Table 4e. As can be seen, regardless of the iteration number, the perplexity of ARB-LLMx and ARB-LLMRC significantly outperforms the baseline BiLLM. Increasing the iteration number further reduces perplexity, yet they can achieve superior results even with just one iteration. Additionally, we visualize the changes in the scaling factor a throughout the alternating iterations to provide further insights in supplementary file.
Group Number. Following BiLLM (Huang et al., 2024), we introduce an additional bitmap for grouping weights, which has been demonstrated to enhance performance. To explore the impact of group size, we expand the group bitmap from a 1-bit to a 2-bit system, increasing the number of groups from 2 to 4. As shown in Table 4f, increasing the number of groups leads to better performance, especially for ARB-LLMx, which outperforms ARB-LLMRC with the same number of groups. Yet, this also results in extra storage (about 0.8 GB for LLaMA-7B). In contrast, using only one group (i.e. the first row of Table 4c) results in total failure. Given the additional storage overhead, the 2-group configuration strikes a good balance between performance and memory efficiency.
4.4 TIME AND MEMORY ANALYSES
Time Comparison. As binary PTQ frame- work, ARB-LLM eliminates the need for fine- tuning. The alternating algorithm requires more computation to align the distribution progressively, yet this overhead is acceptable. In Table 5, ARB- LLMRC with 15 iterations requires only 21 more minutes than BiLLM, while ARB-LLMRe (with- out CGB) requires only 3 more minutes than BiLLM using just iteration. The combination of CGB results in an increase of time overhead, due the percentile search for optimal splitting.
Table 5: Time comparison between BiLLM and our ARB- LLM methods on LLaMA-7B.
Method
CGB
#lter=1
#lter=3 #lter=15
BiLLM
45 min (#Iter=))
ARB-LLMy
x
52 min
59 min
70 min
/
72 min
78 min
88 min
ARB-LLMBC
x 48 min
49 min
53 min
ARB-LLMMBC
67 min
68 min
76 min
Following PB-LLM and BiLLM, we present the memory usage with Raw bitmap/ / CSR compressed bitmap in Table 6. ARB- LLMRc, which replaces the row-wise mean with a column- wise scaling factor, achieves a higher compres- sion ratio along with better performance. Although the refined column-grov group bitmap (CGB) strategy requires more memory due to more scaling factors, the com- bination of ARB-RC and CGB still results in lower storage requirements than BiLLM, while delivering outstanding performance. As shown in Table 6, ARB- L.MRC with or without CGB both require less storage than previous methods. The computation formulas can be found in supplementary file.
Memory Comparison.
Table 6: Memory (GB, Raw bitmap / CSR bitmap) comparison between FP16, PB- LLM, BiLLM, and our ARB-LLM methods.
Method
CGB LLaMA-7B LLaMA-13B
FP16
13.48
26.03
PB-LLM
2
5.33/3.96
BiLLM
2.93/2.19
5,36/3.92
ARB BLMM
2.9372.19
55.663.992
ARB-LLMS
/
2.23
5.95
ARB-L.LMRC X
2.63 2.63/1.89 1.89
4.77 /3.33
ARB-LLMBc
/
2.83 2.09
5.17
CONCLUSION
In this work, we propose ARB-LLM, a series of alternating refined binarization (ARB) methods for LLMs. Through the analyses of the distribution shift between binarized and full-precision weights, we propose an alternating refinement of binarization parameters to progressively align the weight distribution. Moreover, we extend the basic ARB by equipping the calibration data and scaling along row-column axes, resulting in ARB-X and ARB-RC respectively. Additionally, we propose a refined strategy to better combine the salient column bitmap and group bitmap. Our experiments multiple open-source LLM families show that the final models ARB-LLMX and ARB- -RB- can further push the performance boundary from the SOTA binary PTQ methods.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
The research conducted in the paper conforms, in every respect, with the ICLR Code of Ethics.
REPRODUCIBILITY STATEMENT
We have provided implementation details in Sec. 4. We will also release all the code and models.
ACKNOWLEDGMENTS
This work was supported by Shanghai Municipal Science and Technology Major Project (2021SHZDZX0102) and the Fundamental Research Funds for the Central Universities. It was also supported by Lenovo Research (20241 SJTUO1-LR2)))
REFERENCES
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 2013.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. AAAI, 2020.
Mengzhao Chen, Wengi Shao, Peng Xu, Jiahao Wang, Peng Gao, Kaipeng Zhang, Yu Qiao, and Ping Luo. Efficientqat: Efficient quantization-awwe aware training for large language models. arXiv preprint arXiv:2407.11062, 2024.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna Imsys. org (accessed 14 April 2023), 2023.
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty natural yes/no questions. NAACL, 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.0544 2018.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized 1lms. In NeurIPS, 2024a.
Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Salch Ashk- boos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. Spqr: A sparse-quantized represen- tation for near-lossless IIm weight compression. In ICLR, 2024b.
Elias Frantar and Dan Alistarh. Optimal brain compression: framework for accurate quantization and pruning. In NeurlPS, 2022.
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In ICMI, 2023.
Elias Frantar, Salch Ashkboos, Torsten Hoefler, and Dan Alistarh, Gptq: Accurate post-training quantization for generative pre-trained transformers. In ICLR, 2023.
Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Knowledge distillation large language models. In ICLR, 2024.
Yiwen Guo, Anbang Yao, Hao Zhao, and Yurong Chen. Network sketching: Exploiting binary structure in deep cnns. In CVPR, 2017.
11

---

Published as a conference paper at ICLR 2025
Wei Huang, Yangdong Liu, Haotong Qin, Ying Li, Shiming Zhang, Xianglong Liu, Michele Magno, and Xiaojuan Qi. Billm: Pushing the limit of post-trann post-training quantization for llms. In ICML, 2024.
Dongwon Jo, Taesu Kim, Yulhwa Kim, and Jae-Joon Kim. Mixture of scales: Memory-efficient token-adaptive binarization for large language models. arXiv preprint arXiv:2406. 12311, 2024.
Changhun Lee, Jungyu Jin, Taesu Kim, Hyungjun Kim, and Eunhyeok Park. Owq: Outlier-aware weight quantization for efficient fine-tuning and inference of large language models. In AAAI, 2024.
Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. Breeq: Pushing the limit of post-training quantization by block reconstruction. In ICLR, 2021.
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-awa aware weight quantization for on-device Ilm compression and acceleration. In MLSys, 2024.
Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamootthi, and Vikas Chandra. LIm-qat: Drte-free quantization aware training for large language models. In ACL, 2024.
Mitch Marcus, Grace Kim, Mary Ann Marcinkiewicz, Robert MacIntyre, Ann Bies, Mark Ferguson, Karen Katz, and Britta Schasberger. The penn treebank: Annotating predicate argument structure. In IILT, 1994.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In ICLR, 2017.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct clectricity? new dataset for open book question answering. In EMNLP, 2018.
A Paszke, Gross, F Massa, A Lerer, JP Bradbury, G Chanan, T Killeen, Z Lin, N Gimelshein, Antiga, et al. An imperative style, high- pegforerffrmecce deep learning library. In NeurIPS, 2019a.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In NeurlPS, 2019b.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 2020.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xeor-net: Imagenet classification using binary convolutional neural networks. In ECCV, 2016.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: adversarial winograd schema challenge at scale. In AAAI, 2020.
Yuzhang Shang, Zhihang Yuan, Qiang Wu, and Zhen Dong. Ph-llm: Partially binarized large language models. In /CLR, 2024.
Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. simple and effective pruning approach for large language models. In ICLR, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023,
A Vaswani. Attention is all you need. In NeurlPS, 2017.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. ICML, 2023.
Chen Xu, Jianqiang Yao, Zhouchen Lin, Wenwu Ou, Yuanbin Cao, Zhirong Wang, and Hongbin Zha. Alternating multi-bit quantization for recurrent neural networks. In ICI.R, 2018.
12

---

Published as a conference paper at ICLR 2025
Yuzhuang Xu, Xu Han, Zonghan Yang, Shuo Wang, Qingfu Zhu, Zhiyuan Liu, Weidong Liu, and Wanxiang Che. Onebit: Towards extremely low-bit large language models. In NeurIPS, 2024.
Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. In NeurIPS, 2022.
Zhihang Yuan, Yuzhang Shang, Yue Song, Qiang Wu, Yan Yan, and Guangyu Sun. Asvd: Activation- aware singular value decomposition for compressing large language models. arXiv preprint arXiv:2312.05821, 2023.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? ACL, 2019.
Mingyang Zhang, Hao Chen, Chunhua Shen, Zhen Yang, Linlin Ou, Xinyi Yu, and Bohan Zhuang. Loraprune: Pruning meets low-rank parameter-efficient fine-tuning. In ACL, 2024.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.00 200000 2022.
Qihuang Zhong, Liang Ding, Li Shen, Juhua Liu, Bo Du and Dacheng Tao. Revisiting knowledge distillation for autoregressive language models. In ACL, 2024.
13


Reviews:
[{"review_id": "fRSHA0fM8E", "reviewer": "Reviewer_mmbQ", "summary": "This paper proposes a novel post-training quantization method alternating Refined Binarization for quantizing LLM using binary weight. The existing methods suffering the mis-alignment from the distribution of the binarized weight with the full precision counterparts. The existing method also didn't address the column-wise deviation. The proposed framework progressively refines binarization parameters to minimize the quantization error. The proposed framework has 4 components. 1. alternating refined binarization which iteractively update the binarized weight. 2. The proposed method is also improved by calibration data. 3. A more accurate binarization can be achieved by column-wise scaling factor. 4. A column-group bitmap strategy was applied in the paper to improve quantization by maximizing bitmap utilization through salient and non-salient division. The experiment results show impressive results.", "questions": "Does the proposed method perform consistently on 7 zero-shot QA datasets?\n\nIs the binarized model able to get reasonable performance on long context, math and reasoning dataset?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The proposed ARB-LLM method is innovative, which also addresses a critical problem in binary quantization.\n2. The experiments on various LLM families demonstrate the superiority of the proposed methods.\n3. The paper also provides thorough ablation studies the effectiveness of the individual components of ARB-LLM.", "weaknesses": "1. The main weakness of the paper is the evaluation seems limited to perplexity and average accuracy. The paper would be more interesting if there were trade-offs on more tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AR9vsGmbNP", "reviewer": "Reviewer_H9m3", "summary": "The paper describes an Alternating Refined Binarization (ARB) framework for performing post-training quantization (PTQ) of 1-bit for large language models (LLM). The framework is based on the observation that there is a distribution gap between the binarized weights and the floating-point weights. Through the iterative refinement process of ARB, the quantization error can be reduced. The authors also provide two variants that use calibration data and address column deviation. They conduct experiments on OPT and LLAMA models, achieving better results compared to other binary PTQ methods such as BiLLM.", "questions": "1. Can the authors provide any runtime inference metrics to showcase the practical feasibility of the proposed implementation?\n\n2. Could the authors introduce the meaning of each term in equation (6) so that the reader can understand without referring to prior papers?\n\n3. Can the authors clarify the actual memory consumption and compare it to PTQ methods under the same memory budget?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper identifies and addresses the distribution shift between floating-point weights and binary weights through an iterative refinement method. The study shows that through the ARB process, the quantization error decreases.\n\nThe paper further extends the method by adding a calibration dataset and column-wise scales, resulting in better performance compared to BiLLM models.\n\nThe paper offers a comprehensive ablation study and time/memory analysis, showing the trade-offs of each component.", "weaknesses": "The authors claim that the weight bit is 1.11 bits, but based on the memory comparison, the model takes approximately 3GB of memory on the Llama-7B model, which is similar in size to GPTQ-3bit of the same model, with inferior performance. Can the authors clarify this?\n\nIt seems the approach works better on the older OPT model than the Llama family, with Llama3's performance, in particular, being much worse than floating-point. Can the authors showcase the performance on more recent models like Phi/Llama3.2 to address this concern?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N0Wjn0lm6d", "reviewer": "Reviewer_muCn", "summary": "Authors propose to ALTERNATING REFINED BINARIZATION (ARB) which is based on BiLLM paper. They introduce correction term δµ to the original mean µ, effectively mitigating the distribution shift. As a result α, B and µ have to be refined.\n\nThey further extend ARB to ARB-X and ARB-RC, where \nARB-X is extension of ARB with calibration data and loss function which is based on papers Optimal Brain Compression and GPTQ.\n\nARB-RC introduces a column-wise scaling factor αc to better handle parameter variations across columns, while eliminating the redistribution parameter µ to enhance compression in LLM.\n\nIn addition they combine ARB-X and ARB-RC with weight partition strategy based on columngroup bitmap (CGB).\nAnd call final versions as ARB-LLMX and ARB-LLMRC respectively, which outperform state-of-the-art (SOTA) binarization methods for LLMs.\n\nARB-LLMRC outperforms FP16 models of the same size (in GB).", "questions": "Q1: Is it ARB second-order binarization is called in the paper \" higher-bit representation, i.e., second-order binarizatio\".  Is effectively 2bits quantization? \n\nQ2: Please explain why BiLLM accuracy in Table 2 does not match the results in the original BiLLM paper (shown in Table 3 LLaMA and LLaMA2)", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Authors proposed several improvements of BiLLM method and show that ARB-LLMRC outperforms FP16 models of the same size (in GB) and outperforms BiLLM (it is SOTA of LLM post training binarization)\n\nExtensive ablation study of the proposed methods showed effectiveness of Effectiveness of ARB, ARB-LLMX, ARB-LLMRC its combination with CGB and calibration data size, iteration number and group number.\n\nConducted extensive experiments on the LLaMA, LLaMA-2, and LLaMA3 families. \n\nThey did a thorough analysis of time and memory consumption for proposed methods.\n\nThey will release all the code and models.", "weaknesses": "Q: On Figure 1, authors show Pareto curve and demonstrate that  ARB-LLM-RC outperforms the same-size FP16 models and previously published binarized models. So authors show an improvements over previous models binarization, but it does not show the whole picture in terms of pareto curve: they only compare binarized models vs fp16 (on Figure 1). But it will be informative to show also 4bits and 8bits models. For example paper \"The case for 4-bit precision: k-bit Inference Scaling Laws\" shows that in terms of pareto curve, for the same model size, 4bits has the best accuracy. In addition to 4bits and 8bits it would be great to plot 3bits too on pareto curve (given that 3bits is already presented in multiple Tables in this paper).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)