PB-LLM: Partially Binarized Large Language Models
Authors: Zhihang Yuan
Yuzhang Shang
Zhen Dong
Venue: ICLR 2024 poster (2024)
Keywords: Large Language Model
Network Compression
Primary area: generative models
TL;DR: Frist work using network binarization for large language model compression.

Abstract: This paper explores network binarization, a radical form of quantization, compressing model weights to a single bit, specifically for Large Language Models (LLMs) compression. 
Due to previous binarization methods collapsing LLMs, we propose a novel approach, Partially-Binarized LLM (PB-LLM), which can achieve extreme low-bit quantization while maintaining the linguistic reasoning capacity of quantized LLMs. 
Specifically, our exploration first uncovers the ineffectiveness of naïve applications of existing binarization algorithms and highlights the imperative role of salient weights in achieving low-bit quantization. 
Thus, PB-LLM filters a small ratio of salient weights during binarization, allocating them to higher-bit storage, i.e., partially-binarization. 
PB-LLM is extended to recover the capacities of quantized LMMs, by analyzing from the perspective of post-training quantization (PTQ) and quantization-aware training (QAT). 
Under PTQ, combining the concepts from GPTQ, we reconstruct the binarized weight matrix guided by the Hessian matrix and successfully recover the reasoning capacity of PB-LLM in low-bit. 
Under QAT, we freeze the salient weights during training, explore the derivation of optimal scaling factors crucial for minimizing the quantization error, and propose a scaling mechanism based on this derived scaling strategy for residual binarized weights. 
Those explorations and the developed methodologies significantly contribute to rejuvenating the performance of low-bit quantized LLMs and present substantial advancements in the field of network binarization for LLMs. 
Code is available at https://github.com/hahnyuan/PB-LLM.

PB-LLM: PARTIALLY BINARIZED LARGE LANGUAGE MODELS
Zhihang Yuan Houmo Al
Zhen Dong UC Berkeley
ABSTRACT
This paper explores network binarization, a radical form of quantization, compress- ing model weights single bit, specifically for Large Language Models (LLMs) compression. Due to previous binarization methods collapsing LLMs, we propose a novel approach, Bartially-Binarize L.LM LLIM, which can achieve extreme low-bit quantization while maintaining the linguistic reasoning capacity of quan- tized LLMs. Specifically, our exploration first uncovers the ineffectiveness of naive applications of existing binarization algorithms and highlights the imperative role salient weights in achieving low-bit quantization. Thus, PB-LIN LLM filters a small ratio of salient weights during binarization, allocating them to higher- bit storage, i.e. partially-binan binarizationn. PB-LLM is extended to recover the capacities of quan- tized LMMs, by analyzing from the perspective of post-training quantization (PTQ) and quantization-awan training (QAT). Under PTT, combining the concepts from GPTQ, we reconstruct the binarized weight matrix guided by the Hessian matrix and successfully recover the reasoning capacity of PBBLLM in low-bit. Under QAT, we freeze the salient weights during training, explore the derivation of optimal scaling factors crucial for minimizing the quantization error, and propose scaling mechanism based on this derived scaling strategy for residual binarized weights. Those explorations and the developed methodologies significantly contribute rejuvenating the performance advancements in the field of network binarization for LLMs. The code is available at PB-LLM.
low-bit quantized LLMs and present substantial
INTRODUCTION
Recently, large language models (LLMs) have gained significant traction in artificial intelligence. It can be attributed to the success of models such ChatGPT [Brown et al., 2020, Ouyang et al., 2022]. Following its lead, other LLMs such as OPT [Zhang et al., 2022], BLOOM [Scao et al., 2022], and LLaMA [Touvron al., 2023] have emerged, proving that an increase in model size typically results in enhanced capabilities. As a result, models with tens to hundreds of billions of parameters have become the norm. However, their vast size poses considerable deployment challenges memory- constrained devices. A model such as the LLAMA-65D (with 65 billion parameters) requires at least 130GB of memory for inference- number that often exceeds the capacity of single GPU or server.
Many methods have been proposed to reduce the memory consumption of LLMs [Yuan et al., 2024]. Those methods can be categorized into weight quantization [Dettmers et al., 2022], network pruning [Frantar and Alistarh, 2023], and low-rank factorization [Zhang et al., 2023]. Among these compression paradigms. weight quantization is particularly prominent and widely adopted for LLMs. Since it preserves the original model architecture and leverages well-trained L.MSs full-precision checkpoints, the compression process is greatly simplified [Zhu et al., 2023]. However, state-of-the-art LLM quantization methods show marked decline in quality beyond 4 bits [Liu et al., 2023a].
More aggressive compression methods are required to push the LLM quantization into the lower bit range. The network binarization technique stands out, reducing the bit-width of weights to just one bit [Helwegen et al., 2019, Rusci et al., 2020, Qin et al., 2020a; 2023]. The binarized models take little storage and memory, and accelerate the inference by efficient bitwise operations. Compared to other aggressive compression technologies like high-sparsity pruning, network binarization has potent topological generics, it only applies parameters. Binarization is widely studied in

---

Multi-ftedd Self-Attention
Activation
Forward
Feed-
Partially-Binarizze Weight Matrix
Pe (5A%)
6.615
D. 005
Network
FP
OPT1.38
Partially- Binarized Weight FC
Partially- Binarized Weight FC K
Partially- Binarized Weight FC V
Activation
+07
10.
H0.3
+0.3
43
03
+0.
023
2540
Partially- Binarized Weight FC-1
10.
KV cache
KV cache
Partially
Binarize
1
12
000
Random
Activation
Guess
Partially- Binarized Weight FC-1
Activation
Column
Scaling
Partially-
0.01
Binarized
Activation
0.380
Weight
224
327
Epoch
(a) One basic block the Partially-Binn Binarized LLM.
(b) Performance on BooIQ.
Figure 1: (a) We introduce Seriaaly-Binarrzedd Large Language Model (PB-I.M)) where a small subset of the weights of the LLM are frozen and preserved with higher hit precision, while the remaining weights are binarized utilizing an optimal scaling factor strategy; (b) By using FB-LLM, an extremely low-bit LLM can be acquired efficiently (i.e., quantization-awan training converges quickly) while maintaining language reasoning capabilities.
academic research standalone compression technique, rather than simply 1-bit specialization of quantization. Some So'T binarization algorithms have even achieved full-precision performance on large-scale tasks, e.g., ReActNet [Liu et al., 2020a] for ImageNet classification [Deng et al., 2009]. It is theoretically possible to significantly lower the LLM quantization if we generalize the idea of binarizing the weights of LLMs.
In this paper, we explore network binarization specifically for LLM quantization and propose Partially- binarized LLMs (abbreviated as PB-LLM). This methodology aims to achieve extreme quantization to the lowest possible bit, while maintaining the language reasoning capacity inherent in LLMs. The explorations indicate that simple adaptations of existing binarization algorithms do not work well for LLM quantization. As result of this realization, attention is directed towards the salient- salient-weight property of LLM quantization. weights must be fully exploited. We investigate the salient weights in aspects of their detection criteria and granularity, as well as the storage costs. Then, we propose the partially binarized matrix, storing the salient weights in higher bits. After establishing the foundation of PB-LLM, the exploration extends to regain the lost reasoning capacity of the quantized LLMs, under the frameworks of post- training quantization (PTQ) and quantization-awar by the concepts of GPTQ [Frantar et al., 2022], we reconstruct the P3-LLM matrix guided by the Hessian matrix and successfully recover the reasoning capacity of PB-LLM in low-bit. In the view of QAT, salient weights are frozen throughout the binarization process for efficient training. In addition, from the perspective of quantization error minimization, we explore how binarized LLMs should be scaled based on the ideal scaling factor. We scale the binarized weight based on the derived scaling strategy shown in Fig. la. Low-bit quantized L.LMs can significantly improve their performance with such explorations. Benefited from explorations of PTQ and QAT, FB-LLM can efficiently obtain an extremely low-bit LLM with comparable reasoning capacity (see Fig. lb). The methodologies
order to achieve the desired extreme low-bit quantization, salient
training (QAT). In the view of PTQ, inspired
applied and the insights gained within this study stand to contribute substantially to the advancement of knowledge and development in the field network binarization for LLMs.
RELATED WORK
2.1
NETWORK BINARIZATION.
Binarization uses the sign function to binarize weights and activations to +1. To eliminate the vanishing gradient issue caused by the sign function in the binarization, the ttraight estimator (STE) [Bengio et al., 2013] utilized for the network backpropagation. Based on this archetype, copious studies contribute improving the performance of BNNs. Binarization techniques can
2

---

be broadly classified into three categories: the enhancement of training objectives, the reduction of gradient mismatch, and the minimization of quantization errors [Qin et al., 2020b; 2023, Yuan and Agaian, 2023]. To illustrate: Gradient Mismatch: Liu et al. [2020b] introduce double residual connections paired with full-precision downsampling layers. This approach addresses the gradient vanishing problem that arises due to binarization. Training Objectives: Martinez al. [2020], Shang et al. [2022a;b; 2021] focus on optimizing the loss function during training. They suggest aligning the spatial attention maps derived from both binary and real-valued convolutions. Quantization Error Minimization: Rastegari et al. [2016] identify that the disparity in quantization between full-precision and binarized weights can impede the representational abilities of BNNs. As solution, they introduce scaling vecor-determinne by the LI norm-for for both weights and activation functions.
While binarization has proven successful in computer vision, its exploration in natural language processing remains limited. Existing methods [Bai et al., 2020, Qin et al., 2022, Liu et al., 2022; 2023b] primarily target smaller language models (e.g.. BERT-base [Devlin et al., 2018] with 110M parameters) potentially hindering their generalization to larger ones (e.g. LLAMA-7B [Touvron et al., 2023] with 7B parameters). We investigate binarization for LLMs comprehensively in this paper and propose PB-LLM, which is an attempt to compress LLMs using binarization.
2.2 LARGE LANGUAGE MODEL QUANTIZATION.
Quantization, prominent method in model compression, addresses the storage and computational overhead of deep learning models. Recent research efforts successfully apply quantization to compress Large Language Models (LLMs), including Quantization-Aware Training (QAT) and Post-Training Quantization (PTQ).
In the domain of QAT, innovative strategies like LLM-QAT [Liu et al., 2023a] address challenges in acquiring training data for LLMs by leveraging pre-trained models for data-free distillation. Addi- tionally, techniques such as QLORA [Dettmers et al., 2023a] focus on parameter-efficiet fine-tuning (PEFT), expediting model compression and inference acceleration. In PTQ. approaches range from quantizing only the weights of LLMs to jointly quantizing both weights and activations. Methods like GPTQ [Frantar et al., 2022] and QuIP [Chee et al., 2023] optimize matrix multiplications and propose novel layer-wise quantization techniques achieving high compression rates. Squeezel.L.M [Kim et al., 2023] and SpQR [Dettmers et al., 2023b] identify weights that lead to particularly large quantization errors and subsequently storing them with higher precision to mitigate the accuracy degradation caused by weight quantization. AWQ [Lin et al., 2023] and OWQ [Lee et al., 2023] contend that when quantizing weights, it is crucial to account for the impact of activation outliers on weights. Norm Tweaking [Li et al., 2023] addresses the issue of activation value deviation by training LayerNorm. For activation quantization, ZeroQuant [Yao et al., 2022] proposes a fine-grained quantization method that can be applied to both weights and activations. Methods like SmoothQuant [Xiao et al., 2022] and Outlier Suppression [Wei et al., 2022; 2023] shift the quantization challenge from activations to weights by proposing mathematically equivalent per-channel scaling transformation. Omni- Quant [Shao et al., 2023] further enhances performance by training the quantization parameters. RPTQ [Yuan al., 2023a] proposed proposes performance improvement through grouped quanti- zation after clustering similar channels. In this paper, our primary focus lies in the binarization of weights exclusively, employing both PTQ and QAT methodologies.
3 PARTIALLY BINARIZING LARGE LANGUAGE MODELS (PB-LLM)
In this section, we elaborate on the methodology of Partially Binarizing Large Language Models, named PB-LLM. To begin, a review of the foundational framework of binarized neural networks is presented, showcasing its applicability and limitation to LLM quantization. Subsequently, a novel format for the quantized matrix is formulated, specifically tailored for the binarization of L.LMs. Taking advantage of the proposed partially- binnarize weight matrix, we delve into its potential in the realms of post-training quantization and training-aware training for LLMs, to break the trade-off between bit-width and performance. It is crucial to note that, due to constraints in computational resources, the methodology exploration predominantly utilizes O.T-1.3B [Zhang et al., 2022] to perform the majority of experiments. Given the space constraints, this section primarily focuses on key aspects of the methodology. For detailed discussions, exact result values, and specific implementation details in codes, readers are referred to the supplemental materials.
3

---

3.1 PRELIMINARY: NETWORK BINARIZATION
To begin with, we briefly review the general concept of network binarization and binarized neural networks (BNNs) in [Courbariaux al., 2016, Hubara al., 2016]. As most optimizable quantized structures of LLMs are linear layers (see Fig. la) in LLMs, we use a one-layer Perceptron to show the training and inference processes of the BNN. The one-layer neural network is defined as f(x = (a), where a E R4 is the input activation and W RR4 + Rd stands for the weight matrix, with d, and d, representing the sizes of the input and output of the layer, respectively.
The goal of network binarization is to represent floating- (FP) weights, denoted as Wp, and/or FP activations ar as 1-bit (i.e... =1) values [Qin et al., 2020b]. Networks utilizing this representation are referred to as BNNs. BNNs diverge from FP neural networks in their forward operations and in the approximation of backward gradients. In the forward propagation, the sign function is used for binarizing FP values of weights:
+1 -1
x 2 0
Forward: sign(x)= =
< 0.
Specifically, in the training process of binarized network, the BNN maintains FP latent weights W F for gradient updates, and the updated weight matrix W is binarized into the binary weight matrix WB via the binarize function si sign(-). i.e. WB = sign(W F). Then the intermediate activation map (full-precision) of this layer is produced by AF = W AA For inference efficiency, BNNs with 1-bit weights significantly reduce the memory cost of inference. Theoretically, BNNs can binarize both weights and activations to 1 1-bit, providing 32x compression in memory cost and 64x acceleration in inference speed, by replacing FP multiplications in conventional floating poinnt networks with Xnor-Bitcoun operations. However, recent studies highlight that the weights of LLMs as the main contributor to memory overhead [Kim et al., 2023], and thus we primarily aim to curtail memory costs. Therefore, in this pivotal exploration of binarized LLMs, our attention is specifically centered on weight binarization, foregoing the simultancous binarization of weights and activations.
In the backward propagation, the main challenge is that the pervasive sign functions are theoretically non- differentiable, and thus extremely destroy the gradient chain in the backward propagation. To address this prob- lem, researchers widely exploit the straight-through esti- mator (STE) [Bengio et al., 2013] to numerically approx- imate the derivative of the whole BNN [Qin et al., 2020b], i.e,,
Book)
Random
FP BNN
Mcan
0,
XNOR
BI-Real
FDA
HellaSwag
OBOA
Backward:
ac
dr =
Baign(a)
I
S
[
1,
ARC Challenge
WinoGrande
which makes the optimization of BNN accessible.
ARC-Easy
We first investigate the possibility of implementing bina- rization to LLM quantization. Specifically, following the binarization benchmark in BiBench [Qin et al., 2023], generalize some representative binarization methods into LLM quantization scenarios. BNN [Hubara et al., 2016], XNOR [Rastegari al., 2016], Bi-Real [Liu al., 2020b], ReCU [Xu et al., 2021a] and FDA [Xu et al., 2021b] are re-implemented to quantize LLMs, particularly to OPT [Zhang et al., 2022]. Training details are illustrated in the Sec. 4. The results evaluated on seven zero-shot common sense reasoning tasks are shown in Fig. 2. We can see that the L.MSs binarized via the exist- ing popular binarization algorithms perform worse than random guesses, showing that the existing binarization methods are not suitable for L.M binarization.
Figure 2: We implement five renowned bi- narization methods on LLMs and assess the resultant binarized LLMs across seven zero- shot common sense reasoning tasks. Ran- dom represents the hypothetical worst basc- line, indicating random guesses, while FP stands as the optimal baseline, representing full-precision OPT-1.3B. The exact values corresponding to this radar graph are detailed in the Appendix.
3.2 PARTIALLY BINARIZED WEIGHT MATRIX
In the low-bit quantization of Transformers, a significant challenge is managing the salient weights, as they can unnecessarily extend the quantization range [Kovaleva et al., 2021]. Several owteee-
4

---

LLM compression methods have been explored tackle this issue [Dettmers et al., 2022, Wei et al., 2022, Kim et al., 2023, Lin et al., 2023, Yuan et al., 2023b]. Notably, SqueezeLLM [Kim et al., 2023] provides generalized methodology for handling outliers in weight values during 4-bit LLM post-training quantization. Concurrently, AWQ [Lin et al., 2023] demonstrates that preserving only 1% of significant weights benefit 4-bit L.M quantization. Motivated by existing research, this study also seeks to optimize the treatment of salient weights while binarizing most of weights. We present Partially-Binarizz LLMs (PB-LLM), method involving the selective binarization of the LLMs' weight matrix, wherein a minor fraction of weights is kept in high bits for enhanced language capacity.
3.2.1 SALIENT WEIGHT: CRITERIA, GRANULARITY, AND COST
Beyond the most straightforward method of choosing salient weights based on magnitude Weement-wise-we conduct a thorough investigation into salient weight detection from two perspec- tives: criteria and granularity. For criteria, we compare Magnitude- and Hessian-based methods, and for granularity, we explore both wise and column- wise approaches. In addition, we discuss the cost of storing matrix weights in mixeddprrcision manner.
Criteria: Magnitude vs. Hessian. Beyond the identification salient weights through magnitude, alternative criteria have also been examined. The Hessian metric emerges crucial factor in LLM quantization, as elucidated in [Dong et al., 2019, Frantar et al., 2022, Frantar and Alistarh, 2023], particularly in relation to post-training quantization for LLMs (details regarding the Hessian criteria for PTQ can be found in Sec. 3.3). However, we observe that the selection of salient weights, whether by magnitude or Hessian, does not significantly impact the efficacy of LLM partial binarization. especially under the framework of QAT. Consequently, magnitude is elected as the preferred criterion for the identification of salient weights in both PTQ and QAT, primarily due its simplicity and efficacy in distinguishing critical weight components.
Granularity: Element-wise vs. Column-wise, Our investigations reveal that adopting column-wise approach for selecting salient weights has the potential impair the performance of binarization. Visualization of the salient weights' distribution within the matrix, as depicted in Fig. 3 (where the white dots represent the filtered salient weights), disclosed a random and uniform scattering of these weights. Given the absence of any discernable column- wise pattern in the distribution of salient weights, a column- wise filtration method is deemed unsuitable. This scattered and uniform distribution necessitates an element-wise approach for effective filtration in the binarization process.
Figure 3: Distribution of 5% salient weight.
Salient Weight Storing Cost. The additional overhead for storing the salient weights is acceptable. The overall bit number, Naa must adhere to the following condition:
hinary weights
for salient weights
index storing, could he optimized
Nba S
Tbinary
+N -Nalienttbbr bit * (1 Tbinary)
Here, Pbinary denotes the ratio of the binarized weights, Nsatient-Bae bit represents the number of bits allocated for storing salient weights (e.g.. 8 bits), and the additional 1 bit is allocated for using the bitmap mechanism [Chan and loannidis, 1998] for index saving. It's important to note that employing bitmap for index storage not the most efficient method and can be optimized further using sparse matrix storage methods such as Compressed Sparse Row (CSR) or Compressed Sparse Column (CSC) [Borštnik et al., 2014]; hence the use of < instead of = Eq. 3. The relationship between the ratio of salient weights and the overall bit number is illustrated in Fig. 4, depicting that a lower ratio corresponds to a reduced overall bit number. For example, retaining 10% of weights in 8 bits and binarizing the remaining 90% equates to, at most, a 2.7-bit quantization.
10
Figure Variation overall bit number Nose with the ratio of the salient weights Tbinary, where salient weights are stored in 8-bit.
5

---

Table Perplexity of C4 OPT-1.3B quantized with RTN (without GPTQ) and PB-GPTQ. Magnitude criteria or Hessian criteria used for detecting salient weights.
Salient Fraction
50%
20%
10%
5%
RTN Magnitude
24.5675
5892.0898
4889.0385
8023.1132
RTN Hessian
20.2512
8.092852
7508.7788
6173.1611
PB-GPIQ Magnitude
18.3674
46,4093
895.0322
2880,6157
PB-GPTQ Hessian
17.7567
42.1157
165.6767
528.4877
PB-GPTQ Magnitude g=128
18.0293
57.2164
1230.8537
2662.7114
PB-SPTQ Hessian g=128
17.6000
45.9811
157.8825
646.3616
3.3 POST-TRAINING QUANTIZATION FOR PB-LLMS
After defining the partially-binarizn matrix format, the next step is to recover the performance (i.e., the reasoning capacity in the literature of LLMs) of the quantized PB-ILM. In this section, we explore the weight binarization with post-training quantization (PTQ) methods. PTQ methods hold a prominent position in the realm of quantization techniques for LLMs due to their ease of implementation. They enable direct quantization of pre-trained LLMs without the need for a training dataset and additional training overhead. Therefore, first explore the weight binarization within the PTQ framework.
GPTQ [Frantar et al., 2022] is the most efficient and effective method for weight quantization [Zhu et al., 2023], capable of quantizing LLMs to 4-bit or even 2-bit. Therefore, we generalize the idea of GPTQ the partial-binarization setting. Specifically, GPTQ quantizes the weights in LLM layer-by-layer to minimize the layer-wise quantization error:
arg arg ||X - 2 W
(4)
GPTQ quantizes a weight wa to wig. calculates the compensation d-g for remaining weights w--q- and then applies the compensation factor to the remaining weights:
8_ w. 10g (H
)..
W-g w w-q 8 0-9
19
where the is the Hessian matrix of the layer-wise wise quantization error with respect to the weights and Wy is the g-th value in flattened weight matrix W. In GPTQ, weights are quantized iteratively and the remaining weights are updated until all weights have been quantized.
We propose to use GPTQ to iteratively binarize the un-salient weights and quantize the salient weights to higher bit, and then apply the compensation to the remaining weights. Specifically, we first detect the salient weights Weal and un-salient (o-bb--biarrized) weights Wunsal in the weight matrix W = W + Wunsal Drawing inspiration from SparseGPT [Frantar and Alistarh, 2023], we calculate the saliency metric, represented v, / RR for the purpose of detecting salient weights using Hessian criterion. The un-salient weights will be binarized to Wunsal and the salient weights will be quantized to higher bit Woe. We use asymmetric per-channel quantization for both salient and un-salient weights. For un-salient weight, we the per-channel mean as zero point and calculate the optimal scaling factor a for the un-salient weights using the method in Sec. 3.4.2. We use MinMax metric to calibrate the scaling factor and zero point for salient weights.
In the quantization process, we iteratively quantize the columns in the weight matrix W. For each column, we binarize the un-salient weights and quantize the salient weights, and then calculate the compensation for remaining weights, and then apply the compensation factor to the remaining columns of weights. This process is repeated until all the weights are quantized. The proposed method is denoted as PB-GPTO. We also explore the fine- tineggrined PB-GPTO, which quantizes the weights in group-wis wise manner. Specifically. the weight matrix is split into several groups, each group contains g columns. In each group, we detect the salient weights and un-salient weights, and then calibrate to set the scaling factor and zero point using the weights in this group.
The results are listed in Tab. 1. PB-GPTQ is significantly better than RTN. We note that the Hessian- based TB-GPTQ exhibits a superior performance compared to the Magnitude criterion PB GPTQ. The group-wise PB GPTQ performs better worse than the non-group-wisg PB GPQ. but the difference is not significant. Our analysis suggests that the disparity in scaling factors is not the primary determinant of binarization performance; hence, the introduction of group-wise methodology does not yield an enhancement in binarization performance. Subsequently, our next endeavor will involve the application of QAT to reduce the error introduced by weight binarization.

---

3.4 QUANTIZATION-AWAN TRAINING FOR PB- PB-LLMS
In order to further enhance the reasoning capacity of the Partially- Large Language Models (PB-LIM), we extend our exploration by employing Quantization-wware Training (QAT) to meticu- lously train the quantized models. Because LLM training is difficult, we desire that PB-LLM training could be as efficient as possible. To realize efficient training for PB-LIM, we propose the Salient Weights Frozen and Optimal Scaling Factor for Binary Weights, targeting the salient weights and binarized weights, respectively.
3.4.1 SALIENT WEIGHTS FROZEN
To leverage the value of pretrained weights, we propose freezing the salient weights, determined by weight magnitude, prior to the weight binarization process. As illustrated in Fig. la, we initially filter out number of weights from pre-trained weight matrix- e.g., 2% by magnitude-at the beginning of quantization-awwe maintaining their fixed state throughout the training process. Ex- amination of training efficiency (refer to Fig.5) suggests that these salient weights play crucial role in LLM capacity. Maintaining the high bit representation of certain weights, aids the training quantized LLMs and reduces their optimization difficulty.
Loss (we Outlee Frazen)
Loss (w. 2% Outlier Frozon)
training,
Figure 5: Training Loss Curves: When only 2% of weights are re- tained in their un- ne-binarized state, the training loss converges swiftly.
3.4.2 OPTIMAL SCALING FACTOR FOR BINARY WEIGHTS.
AWQ [Lin et al., 2023] enhances the weight-only quantization method for LLMs by optimizing scaling factors mitigate the quantization eгтоr of quantized weights. Specifically, AWQ demonstrates that searching for empirically optimal scaling factors proves to be an effective strategy for reducing quantization errors and recovering the performance of the quantized models. Fortunately, in the context of LLM binarization, we have better choice for scaling the binarized weights. There's no need to search for optimal scaling factors as they can be analytically derived. Specifically, we apply a column- wise scaling factor to binarized weights to reduce the binarization error, i.e., enforcing Wr = aw g- The optimal values of scaling factor a for the WB E {-1, 1] can be calculated by minimizing the L2 error:
a' = arg min J(a), in which T(a) = ||w ow
(6)
aeR.
Following XeOR-Net [Rastegari et al., 2016], by expanding the below equation, we have
[aa) = a² w Ww 2ow7 WB + w WwF
For the vector with w E RR" we follow the traditional meth- ods of binarizing weights [Hubara et al., 2016] by taking the sign of real-valued weights:
2073751
P5-LMM Outter
w B sign(w()) { +1,
w 2 0: w
-1,
< 0.
241.7581
5753
In that case, B WB Twwer where nwr is number of ele- ments in Wr, and a² can be solved as:
47,3225
16 16,0007
Figure Perplexity (PPL) on C4:
When 50% of the weights are main- (9) tained in their un- oinaiinarzzed state (equiv- alent to around 5-bit quantization), the untrained P3- LLL does not experience of salient-frozen and optimal-scaling mechanisms: directly a total loss of reasoning capabilities.
WB
||r]l1
Tw
TWr
A counterintuitive outcome emerges from the incorporation
deploying those two mechanisms to pre-trained LLM even without any retraining or fine-tuning, still results in commendable performance. For instance, applying these techniques to OPT-1.3B with 50% salient weights (see Fig. 6) reveals that the partially-biarrized OPT-1.3B retains small amount of language capacity, corroborating the importance of a small number of salient weights in LLM quantization. Consequently, implementing just these two techniques OOutlirr Frozen and Optimal Scaling Factor for Binary Weights- pre-trained LLMs serves as an efficient starting point for training PB-LIM.
7

---

Book) (Clark 2019]
(8isk 20201
HellaSwag [Zeleess al,, 2019]
(d) WinoCrrnde [Sakaguchi 2021
60m
ARC-E (CTark
ARC-C [Clark
|Mihaylov
Seven Tasks
2019]
[Bisk
(k) HellaSwag al., 2019]
WinoCimode [Sakaguchi 2021]
10ch
ARC-E [Clark 2018]
ARC-C [Clark 2018]
[Milaylov
Avcrage Seven Tasks
Figure 7: QAT training results with 30% salient weights PB-LLM (upper two lines): As fine-tuning epochs increase, quantized models swiftly regain their reasoning capacities, demonstrating resilience and adaptability PB-LM sustaining cognitive functionalities within models, despite substantial quantization; QAT training results with 5% salient weights PB-LIM (bottom two lines): Existing LLM QAT methods exhibit an absolute failure when subjected to extremely-low bit conditions. In contrast, P LLM triumphs in restoring the reasoning capacities of low-hit quantized LLMSs. This underlines the efficacy of PR-LLM in balancing quantization and performance, preserving the essential reasoning abilities LLMs even under rigorous bit reduction.
Both of the above-prow proposed mechanisms are very effective when used during quantization-ave aware training of PB-LLM. The consequential outcomes are delineated in Fags.7a-7p. Observations from the presented results elucidate that optimizing using the partially-binarizedd quantization format is notably more straightforward compared to single-bit quantization. This empirical evidence corroborates the discussion regarding the rapid convergence property found in 3.c.3.4.1, highlighting the efficacy and adaptability of our proposed methodology in optimizing LLMs within the constraints of partial binarization. From the perspective of QAT, PB-LLM emerges to existing LLM QAT methods. For instance, while models like LLM-QAT [Liu et al., 2023a] necessitate up to 100K iterations for adequate training, PB- LLM remarkably achieves recovery the performance of quantized LLMs in merely around 1-10K iterations.
efficient in training compared
4 EXPERIMENTS
Besides the exploration with OPT-1.3B in Sec. 3, we assess the effectiveness of PB-LLM by conduct- ing experiments on ILaMA-7B [Touvron et al., 2023] and presenting results on various tasks.
4.1 EXPERIMENTAL SETUP
Dataset. In this study, the FB-LLM is trained using the RedPajama-simple-1B 1B dataset, the dataset for LLaMA training is not openly accessible. This dataset, RedPajama- IT, is structured to closely resemble the LLaMa paper and serves transparent, open-source alternative to LLM training
8

---

Table 2: Zero-shot performance Common Sense Reasoning tasks within 4-bit setting. Reported results of previous works are documented in their papers. PB-LLM 30% denotes the preservation of 30% salient weights. and FLL 10% implies the preservation of 10% salient weights.
Method
BoolQ
PIQA
HellaSwag
WinoGrande
ARC-E
ARC-C
OBQA
Avg
FP LMA-BB
76.8
79.3
76.1
70.0
73.0
48.0
57.6
68.7
RTN
71.2
77.3
72.7
66.9
68.8
46.4
52.8
65.2
SuaothQuant
67.7
76.0
69.4
66.7
66.9
43.0
50.6
63.0
LLM-QAT
75.5
78.3
74.0
69.0
70.0
45.0
55.4
66.6
PB PB-GPTQ 10%
62.3
55.9
27.7
49.3
29.3
20.1
10.6
36.5
PR-GPTQ 30%
73.5
74.9
47.5
64.9
61.3
32.4
25.2
54.2
PB-LIM 10%
68.9
67.8
68.1
67.4
58.7
42.9
50.6
60.6
PB-LLM 30%
75.7
78.0
74.3
69.7
69.0
45.6
55.8
66.9
dataset. It amalgamates data from diverse sources including Commoncrawl, C4, GitHub, Wikipedia, Gutenberg Books3, ArXiv, and Stackexchange. RedPaajma-simplee-B IB, representing 0.1% subset of RedPajama- IT, is substantially smaller than the typical datasets used for training other LLMs, making it convenient choice for experiments.
Training Details. In the training process of our quantized network, we commence with a pre- trained model for initialization. The optimization of the model is facilitated through the AdamW optimizer [Loshchilov and Hutter, 2017], applied with zero weight decay. We assign a batch size of I each GPU and implement a learning rate of 2c-5, adhering to a cosine learning rate decay strategy. We only fin-tunee our PB-LLM for 10K iterations.
Evaluated Tasks. To eliminate the variance of evaluated performance, we evaluate the binarized LLMs on seven zero-shot common sense reasoning tasks, i.e., BoolQ [Clark et al., 2019], PIQA [Bisk al., 2020], HellaSwag [Zellers et al., 2019], WinoGrande [Sakaguchi et al., 2021], ARC-Easy, ARC-Challenge [Clark et al., 2018], OBQA [Mihaylov et al., 2018]. We also along eavulated the quantized moelds' perplexity scores on WikiText2 [Merity et al., 2016] and C4 [Raffel et al., 2020].
4.2 RESULTS ON LLAMA
Experiments were conducted on LLaMA-7B. The results of employing PB-GPTQ and PB-LIM are illustrated in Tabs. 2 and 3. When employing PTQ. PB PB-GPTQ exhibited commendable performance, particularly when the salient weight exceeded 30%. Nevertheless, a noteworthy decline in the perfor- mance of the quantized network was observed when the salient weight was reduced to 10%. On the other hand, employing QAT resulted in a notable improve- ment in the performance. A comparison within 4-bit quantization setting between PB 77--LN -LMM 30%
Table 3: Perplexity of C4, wikitext2 and PTB on LLaMA-7b quantized with PTQ methods.
C4
WIKI
PTB
7.3435
5.6770
41.1509
GPTQ 4b
8.6977
8.1368
57,9951
SparseGPT 50%
15.5949
12.829483
505.1396
GP- 50%
8.1466
6.3089
54,8674
PR-CPTO 20%
20.6057
17.1929
280.4353
PB-GPTQ 10%
72.1115
85.7838
708.4120
PR-CPTO 5%
401.6475
619.1054
1687.1815
and LLM-QAY in Tab. reveals superior performance by our method. It is notable that PB-LLM is only fine-tuned for 10K iterations, whereas LLM-QAT underwent 100K iterations of training, show- ing its fast convergence property (refer to Sec. 3.4). The results under PB-LLM 10% represent the outcomes of PB-LLM where 10% of salient weights are preserved. This demonstrates the potential for advancing LLM quantization towards a fully 1-bit state.
5 CONCLUSION
In conclusion, this work is the first to implement network binarization for LLM quantification, intro- ducing the novel Partially-binarizze LLM (PB-LLM) methodology. This approach is meticulously designed to maintain linguistic reasoning capabilities of LLMs, even under extreme low-bit quantiza- tion. The research unearthed the significant role of salient weights in achieving extreme quantization and proposed innovative strategies like optimal scaling for effective binarization. This framework extended to recover the capacities of quantized L.Ms, by analyzing from the perspective of post-training quantization (PTQ) and quantization-awarr training (QAT). The methodology is significant stride in the realm of network binarization for LLMs.

---

REFERENCES
Tom Brown, Benjamin Mann, Nick Ryder, Melamie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedhack. Advances in Neural Information Processing Systems, 35:27730-27744 2022.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models, arXiv preprint arXiv:2205, 01068, 2022.
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, al. Bloom: 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.08 100, 2022.
Hugo Touvron. Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 23002 13971, 2023.
Zhihang Yuan, Yuzhang Shang. Yang Zhou, Zhen Dong, Chenhao Xue, Bingzhe Wa, Zhikai Li, Qingyi Gu, Yong Jae Lee, Yan Yan, et al. Lim inference unveiled: Survey and roofline model insights. arXiv preprint arXiv:2402. 16363, 2024.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm. int8 (:: 8-bit matrix multiplication for transformers at scale. arXiv preprint arXiv:2208.0733: 2022.
Elias Frantar and Dan Alistarh. Sparsegpt: ICML, 2023.
Massive language models can be accurately pruned in one-shot.
Mingyang Zhang, Chunhua Shen, Zhen Yang, Linlin Ou, Xinyi Yu, Bohan Zhuang, et al. Pruning meets low-rank parameter-efficient fine-tuning, arXiv preprint arXiv:2305. 18403, 2023.
Xunyu Zhu, Jian Li, Yong Liu, Can Ma. and Weiping Wang. A survey model compression for large language models. arXiv preprint arXiv:2308.0765 2023.
Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernic Chang, Pierre Stock, Yashar Mchdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. Llm-qat: Data-free quantization aware training for large language models. arxiv preprint arXiv:2305. 17888, 2023a.
Koen Helwegen, James Widdicombe, Lukas Geiger, Zechun Liu, Kwang- Ting Cheng, and Roeland Nusselder. Latent weights do not exist: Rethinking binarized neural network optimization. Advances in neural information processing systems, 2019.
Manuele Rusci, Alessandro Capotondi, and Luca Benini. Memory-driven mixed low precision quantization for enabling deep network inference on microcontrollers. MLSys, 2020.
Haotong Qin, Ruihao Gong, Xianglong Liu, Mingzhu Shen, Ziran Wei, Fengwei Yu, and Jingkuan Song. Forward and backward information retention for accurate binary neural networks. In CVPR, 2020a.
Haotong Qin, Mingyuan Zhang, Yifu Ding, Aoyu Li, Zhongang Cai, Ziwei Liu, Fisher Yu, and Xianglong Liu. Bibench: Benchmarking and analyzing network binarization. ICMI, 2023.
Zechun Liu, Zhiqiang Shen, Marios Savvides, and Kwang- Ting Cheng. Reactnet: Towards precise binary neural network with generalized activation functions. In ECCV, 2020a.
Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre- trained transformers. arXiv preprint 221.322200.7723., 2022.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. 108.v:130..3433 2013.
Haotong Qin, Ruihao Gong, Xianglong Liu, Xiao Bai, Jingkuan Song, and Nicu Sehe. Binary neural networks: A survey. Pattern Recognition, 105:10728, 2020h.
10

---

Chunyu Yuan and Sos Agaian. A comprehensive review of binary neural network. Artificial Intelligence Review, pages 1-65, 2023.
Zechun Liu, Wenhan Luo, Baoyuan Wu, Xin Yang. Wei Liu, and Awang-Ting Cheng. Bi-real net: Binarizing deep network towards real-network performance. IJCV, 2020b.
Brais Martinez, Jing Yang. Adrian Bulat, and Georgios Tzimiropoulos. Training binary neural networks with real-lo-binary convolutions. arXiv preprint (rXXi:2003.//5333 2020.
Yuzhang Shang, Dan Xu, Ziliang Zong, Liqiang Nie, and Yan Yan, Network hinarization via contrastive learning. In ECCV, 2022a.
Yuzhang Shang, Dan Xu, Bin Duan, Ziliang Zong, Liqiang Nie, and Yan Yan. Lipschitz continuity retained binary neural network. In European conference on computer vision, pages 603-619. Springer, 2022b.
Yuzhang Shang, Bin Duan, Ziliang Zong, Liqiang Nie, and Yan Yan. Lipschitz continuity guided knowledge distillation. In ICCV, 2021.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet elassification using binary convolutional neural networks. In ECCV, 2016.
Haoli Bai. Wei Zhang, Lu Hou, Lifeng Shang. Jing Jin, Xin Jiang, Qun Liu, Michael Lyu, and Irwin King. Binarybert: Pushing the limit of bert quantization. arXiv preprint arXi::2012.1500. 2020.
Hantong Qin, Yifu Ding, Mingyuan Zhang, Qinghua Yan, Aishan Liu, Qingqing Dang, Ziwei Liu, and Xianglong Liu. Bibert: Accurate fully binarized bert. arXiv preprint arXiv:2203.6638 2022.
Zechun Liu, Barlas Oguz, Aasish Pappu, Lin Xiao, Scott Yih, Meng Li, Raghuraman Krishnamoorthi, and Yashar Mehdad. Bit: Robustly binarized musti-distilled transformer. Advances in neural information processing systems, 35: 35:14303-14316, 2022.
Zechun Liu, Barlas Oguz, Aasish Pappu, Yangyang Shi, and Raghuraman Krishnamoorthi. Binary and ternary natural language generation. ACL, 2023b.
Jacob Devlin, Ming- Bing-Weiie Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04005. 2018.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized Ilms. arXiv preprint arXiv:2305. /4314 2023a.
Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher De Sa. Quip: 2-bit quantization of large language models with guarantees. arXiv preprint arXiv:2307.13304 2023.
Schoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong. Xiuyu Li, Sheng Shen, Michael W Mahoney, and Kurt Keutzer. Squeezellm: Dense-and-sparse quantization. arXiv preprint arXiv:2306.00 076699, 2023.
Tim Detimers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Salch Ashkboos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. Spqr: A sparse-quantized representation for ncar-lossless Ilm weight compression. arXiv preprint Xrx::x2366.002 03300787 2023b.
Lin, Jiaming Tang, Haotian Tang, Shang Yang. Xingyu Dang, and Song Han, Awq: Activation aware weight quantization for Ilm compression and acceleration. arXiv preprint arXiv:2306.0099 2023.
Changhun Lee, Jungyu Jin, Taesu Kim, Hyungjun Kim. and Eunhyeok Park. Owq: Lessons learned from activation outliers for weight quantization large language models. arXiv preprint arXiv:2306.02272. 2023.
Liang Li, Qingyuan Li. Bo Zhang, and Xiangxiang Chu. Norm tweaking: High-performance low-bit quantization of large language models. arXiv preprint arXiv:2309.02788 2023.
Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Ze- roquant: Efficient and affordable post-training quantization for large-scale transformers. arXiv preprint arXiv:2206.0186) 2022.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. arXiv preprint arXiv:2211. 10438, 2022.
Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Fengwei Yu, and Xianglong Liu. Outlier suppression: Pushing the limit of low-bit transformer language models. NeurlPS, 2022.
11

---

Xiuying Wei. Yunchen Zhang, Yuhang Li, Xiangguo Zhang, Ruihao Gong, Jinyang Guo, and Xianglong Liu. Outlier suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling, arXiv preprint wrXiv:2304.09145, 2023.
Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantization for large language models. CoRR, abs/2308. 13137, 2023.
Zhihang Yuan, Lin Niu, Jiawei Liu, Wenyu Liu, Xinggang Wang. Yuzhang Shang, Guangyu Sun, Qiang Wu, Jiaxiang Wu, and Bingzhe Wu. Rptq: Reorder- based post-training quantization for large language models. arXiv preprint arXiv:2304.0 01089, 2023a.
Matthicu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. In NeurlPS, 2016.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El- Yaniv, and Yoshua Bengio. Binarized neural networks. In NeurlPS, 2016.
Zihan Xu, Mingbao Lin, Jianzhuang Liu, Jie Chen, Ling Shao, Yue Gao, Yonghong Tian, and Rongrong Ji. Recu: Reviving the dead weights in binary neural networks. In ICCV, 2021a.
Yixing Xu, Kai Han, Chang Xu. Yehui Tang, Chunjing Xu, and Yunhe Wang. Learning frequency domain approximation for binary neural networks. In NeurIPS, 2021b.
Olga Kovaleva, Saurabh Kulshreshtha, Anna Rogers, and Anna Rumshisky, Bert busters: Outlier dimensions that disrupt transformers. arXiv preprint arXiv:2105.06990 2021.
Zhihang Yuan, Yuzhang Shang, Yue Song, Qiang Wu, Yan Yan, and Guangyu Sun. Asvd: Activation-ave aware singular value dccomposition for compressing large language models. arXiv preprint arXiv:2312.05821, 2023b.
Zhen Dong, Zhewei Yao, Amir Gholami, Michael w Mahoney, and Kurt Keutzer. Hawq: Hessian aware quantization of neural networks with mixed-precision. In ICCV, 2019.
Chee-Yong Chan and Yannis loannidis. Bitmap index design and evaluation. In SIGMOD, 1998.
Urban Borštnik, Joost Vandelondele, Valéry Weber, and Jürg Hutter. Sparse matrix multiplication: The distributed block-compressed sparse library. Parallel Computing, 2014.
Christopher Clark. Kenton Lee, Ming-Wos Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint 1905:1905510044 2019.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqa: Reasoning about physical commonsense in natural language. In AAAI, 2020.
Rowan Zellers, Ari Holtzman. Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint arXiv:1905.07830 2019.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge scale. Communications of the ACM, 2021.
Peter Clark. Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.0522 1803.05457, 2018
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.00 027899 2018.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, arXiv preprint 171.00171 2017.
Stephen Merity, Caiming Xiong. James Bradbury, and Richard Socher. Pointer sentinel mixture models, arXiv preprint arXiv:1609.07843 2016.
Colin Raffel, Noam Shazcer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li. and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. JMLR, 2020.
12

---

Salient Weights [Low Rabio)
Safient Weight Detaction Regsltse-baned
Partal: Binartaed LU!
Pratra ned LLM
Utisallent
We ghts Matiol
Quantzattion
using
Salient Weight Detection Bhooa ar hased?
Partial: Binanzed
Pretziire LUN
Selection
Un-oalient Weights Hligh Ratio)
Figure 8: Flowchart of PB-LLM.
A SUPPLEMENTAL MATERIALS
A.1 EXISITING BINARIZATION METHODS ON LLM QUANTIZATION
Method
BoolQ
PIQA
HellaSwag
WinoGrande
ARC-Easy
ARC-Challenge
OBQA
Mean
Random Performance
0.5
0.5
0.25
0.5
0.25
0.25
0.25
0.36
FP
0.595
0.63
0.415
0,595
0.54
0.22
0.25
0.46
BNN
0.38
0.345
0.235
0.46
0,195
0.165
0.15
0.30
XNOR
0.37
0.525
0.265
0.49
0.195
0.165
0.16
0.31
Bi-Real
0.395
0.25
0.505
0.235
0.185
0.165
0.32
ReCU
0.515
0.24
0.51
0.255
0.32
FDA
0.39
0.485
0.265
0.49
0.265
0.19
0.17
0.32
Table 4: Table corresponds to Figure 2 in the main paper: We implement five renowned binarization methods on LLMs and assess the resultant binarized LLMs across seven zero-shot common sense reasoning tasks.
We first investigate the possibility implementing binarization to LLM quantization. Specifically, following the binarization benchmark in BiBench [Qin al., 2023], generalize some representative binarization methods into LLM quantization scenarios. BNN [Hubara et al., 2016], XNOR [Rastegari al., 2016), Bi-Real [Liu al., 2020b], ReCU [Xu al., 202 and FDA [Xu al., 2021b] are re-implemented to quantize LLMs, particularly to OPT [Zhang et al., 2022]. Training details are illustrated in the Sec. The results evaluated on seven zero-shot common sense reasoning tasks are shown in the above table. We can see that the LLMs binarized via the existing popular binarization algorithms perform worse than random guesses, showing that the existing binarization methods are not suitable for LLM binarization.
A.2 CODES
Codes be found anomalously in PB-LLM.
13

---

Criteria
WIKI
PTB
C4
Hessian
37.07
47.07
38.76
Magnitude
37.50
44.42
38.64
Table 5: Salient Weight Selection
A.3 FLOWCHART OF PB-LIM
A.4 MORE EXPERIMENTS ON SALIENT WEIGHT SELECTION
In experimental analysis, employed the OPT-1.3b network and conducted PB-LLMM (QAT) quantization under both Hessian and Magnitude criteria, targeting a 10% salient weight. The performance results, measured in perplexity (PPL) Tbale 5. We observed that the PPL values were comparable between the two criteria. However, it's worth noting that the Hessian criteria involve an additional calibration step (collecting calibration data to compute the Hessian), adding complexity to the process. In contrast, the Magnitude criteria offer simpler approach, eliminating need for this extra calibration step. Given the similar PPL results and the desire to streamline the quantization process in QAT, opted for the Magnitude Metric to identify salient weights during QAT. This decision was motivated by the goal of simplifying the process while maintaining comparable performance.
14


Reviews:
[{"review_id": "T5IFyC96el", "reviewer": "Reviewer_92rD", "summary": "The paper examines the application of network binarization to compress Large Language Models (LLMs), an approach that reduces model weights to a single bit. The authors have developed methodologies that significantly improve the performance of binarized LLMs, thereby contributing valuable insights to the field of LLM compression through network binarization.", "questions": "See weaknesses", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. Compressing LLMs is an important question for today’s AI research, and the authors first introduce binarization into LLM compression pushing the quantized LLM into ultra-low bits. \n2. The authors present a thorough exploration of network binarization techniques. They effectively demonstrate the feasibility and potential of partially-binarized LLMs using post-training quantization and quantization-aware training methodologies.\n3. The inclusion of source code with the submission is commendable, enabling reproducibility and verification of the reported results, which are impressive.", "weaknesses": "1. While the incorporation of Quantization-Aware Training (QAT) in LLM compression is an interesting proposal, its practicality is uncertain given the substantial costs associated with training LLMs. Could the authors elaborate on the overhead implications of QAT for LLMs? \n2. In regards to Table 2, it is unclear whether GPTQ-PB represents the method proposed by the authors. Could you clarify the distinction between GPTQ-PB and PB-LLM within the context of your study?\n3. The application of optimal scaling techniques appears to be confined to the specific case presented. Could these techniques be generalized to other bit configurations, and if so, how might this affect the compression performance?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0oBMzfPunQ", "reviewer": "Reviewer_tXaS", "summary": "This paper presents a methodology termed PB-LLM for the partially weight-only binary quantization for Large Language Models (LLMs). Particularly, it outlines the limitations encountered when applying previous BNN works to LLMs, while proposing new methods from Post Training Quantization (PTQ) and Quantization Aware Training (QAT) for LLM low-bit weight quantization. Within PTQ, the paper utilizes Hessian information to identify element-wise salient weights, upon which 8-bit quantization is applied to preserve reasoning capacity in a pre-trained LLM, while binary (1-bit) quantization is applied to the un-salient weights. After applying mixed-quantization, it employs GPTQ to un-salient weights for further optimization (termed as GPTQ-PB). In QAT, the paper applies QAT during the LLM fine-tuning stage, utilizing 1) optimal scaling factor and 2) salient weight freezing methodology to bring about an efficient yet higher-performing QAT methodology (termed as PB-LLM). The methodologies studied and proposed in this work provide insightful contributions to the field of LLM binarization, showcasing the potential for further advancements in LLM binarization.", "questions": "1. What is the rationale behind utilizing a Hessian-based approach to identify salient weights in PTQ, while employing a magnitude-based approach to identify salient weights in QAT?\n\n2. Is there a specific reason why only the PB methodology and GPTQ were applied in PTQ? I am curious about the performance of AWQ-PB in comparison.\n\n3. Is there a plan to compare the task-specific fine-tuning in QAT with the 2-bit QAT-KD methodology (QuantGPT [R2], TSLD [R3])?\n\n4. The optimal scaling factor and salient weight freezing seem to primarily aim at reducing harsh quantization errors before QAT. Is there insight into how these methods improve the QAT training process as depicted in Figure 5?\n\n[R2] Tao et al, \" Compression of Generative Pre-trained Language Models via Quantization\" ACL 2022.  \n[R3] Kim et al, \"Token-scaled logit distillation for ternary weight generative language models\" NeurIPS 2023", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper provides a well-structured presentation of the preliminaries of binary quantization to the introduction of the proposed method, which made it easy to follow.\n\n- Building on previous research such as AWQ and SparseGPT, this paper proposes a partial binarization technique through salient weight protection based on the hessian information and effectively demonstrates its efficacy in PTQ with ablation study (as shown in Table 1).\n\n- The paper showcases empirical improvements in QAT optimization (higher accuracy with fewer training steps than LLM-QAT) across the CSQA tasks.", "weaknesses": "Major Concerns\n- Lack of novelty: The authors propose the optimal scaling factor as their primary contribution, but the core idea itself seems to have already been proposed in the previous work. For example, [R1] proposed an optimal ternary function (eq. (3) of [R1]), but it can be trivially reduced to the binary function when the threshold is zero; then the equation seems to be identical to the equation (8) and (9) proposed in this paper.\n\n[R1] Li te tal., Ternary Weight Networks\n\n\n- Lack of Evaluation Task: This paper evaluates the reasoning capability of LLM only through the accuracy of the CSQA task. In the CSQA task, tasks such as OBQA and ARC challenge were used for OPT-1.3B, where the FP performance did not reach even the random baseline (25%). It raises questions about the suitability of these tasks for demonstrating the effectiveness of fine-tuning, and, hence, the superiority of PB-LLM. To show the effects of fine-tuning more clearly, it would be advisable to carefully select reasoning tasks that are appropriate for the model capacity. Reporting performance not just on CSQA, but also on multi-task accuracy like MMLU would be also beneficial for highlighting PB-LLM's efficacy.\n\n- Inconsistent Salient Weight Methodology between PTQ and QAT: The absence of a consistent methodology for salient weight protection between PTQ and QAT is concerning. While the effectiveness of using Hessian criteria for identifying salient weights in PTQ is demonstrated through performance comparisons, the rationale for using magnitude criteria to identify salient weights in QAT seems to be missing. Understanding the disparity in the approach to salient weight protection across PTQ and QAT is crucial for a holistic appreciation of the proposed method.\n\n- Insufficient evidence on PB-LLM efficiency: To claim that PB-LLM is more efficient in terms of training iteration number compared to LLM-QAT, a more thorough comparison seems necessary. Specifically, it needs to be clear whether the LLM-QAT, being compared with PB-LLM, has been fine-tuned on the same dataset as PB-LLM. Detailed experimental setup information regarding the LLM-QAT is required. Moreover, verification is needed on whether the results through PB-LLM QAT have fully enhanced the reasoning capacity of the pre-trained model. Essentially, it appears that the reasoning accuracy of the target model (OPT-1.3B) obtained through FP fine-tuning should be presented as the upper bound in Figure 7. Additionally, there seems to be a lack of information in Table 2 regarding whether FP LLaMA-7B performance is pre-trained or fine-tuned.\n\n\nMinor Concerns\n- Typo: Sec 3.3 bianrize -> binarize\n- Consistent notation should be used in Sec 4.1 -> LLaMA, LLaMa -> LLaMA\n- There may be an incorrect reference link in Sec 4.1, \"showing its fast convergence property (refer to 3.2)\" Should it possibly be corrected to \"refer to 3.4?\"\n- There are spacing issues in the Figure 7 caption, \"LLMin\" should be \"LLM in\" and \"PM-LLMtriumphs\" should be \"PB-LLM triumphs\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "djmyVVuIaL", "reviewer": "Reviewer_9nFP", "summary": "This paper proposes a Partially-Binarized LLM (PB-LLM), which leverages a PB mechanism to binarize LLMs for more efficient GPU memory usage. The main motivation behind PB is the recognition that a fraction of 'salient weights' exists within the LLM weights, which are essential and restrict full binarization. PB-LLM preserves the precision of these salient weights while focusing on binarizing the non-salient weights. The paper also investigates the integration of PB-LLM with post-training quantization and quantization-aware training schemes and establishes a connection between PB-LLM and GPTQ. Experimental results are presented to demonstrate the effectiveness of PB-LLM.", "questions": "- How does PB-LLM's performance vary among various sizes of Llama models, e.g., 7B-65B?\n- How does PB-LLM perform for aligned/instruction fine-tuned models, e.g., Alpaca and/or Vicuna? To what extent will the binarization scheme affect the model's performance say on the Hugging Face leaderboard [1]?\n- What is the actual GPU memory saving look like for PB-LLM and all considered baselines?\n\n[1] https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-written and well-motivated.\n- The proposed PB-LLM scheme is easy to follow and straightforward to understand.\n- Exploring to improve the memory and/or storage efficiency using quantization (and algorithmic approximation in general) of LLMs is a promising research direction.", "weaknesses": "- Only Llama 7B is studied as the LLM for PB-LLM and all other baselines. Thus, it's not clear how the PB-LLM method performs on larger-scale models.\n- Only pre-trained base models are experimented with those models, however, are usually not deployed directly as applications. \n- The LLM quantization scheme is motivated using an angle of GPU memory efficiency. However, the actual GPU memory usage before and after binarization/quantization is not studied in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NqTeCSfDES", "reviewer": "Reviewer_JdJY", "summary": "This paper introduces PB-LLM, an extreme quantization method that partially binarizes model parameters based on their relevance to linguistic reasoning tasks. PB-LLM selectively identifies and designates a small fraction of salient weights for binarization, assigning them to higher-bit storage, and essentially implementing partial binarization. PB-LLM is extended to recover quantized LMMs' capabilities through post-training quantization (PTQ) and quantization-aware training (QAT). In PTQ, the Hessian matrix guides the reconstruction of binarized weights to restore reasoning capacity at low bits. In QAT, we freeze salient weights during training, derive optimal scaling factors to minimize quantization errors and propose a scaling mechanism for enhanced residual binarized weights. The result shows that PB-LLM achieves competitive results without losing performance gain.", "questions": "1. The readability of the paper can be improved by including a flowchart or block diagram of the proposed method by illustrating different stages of transforming a PB-LLM model. \n2. The paper reports primarily the accuracy to evaluate the proposed approach. The addition of the final model size after partial binarization can further solidify the claim.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ Presents a practical approach that analytically identifies non-salient weights and applies binarization selectively for large models.\n+ Achieves LLM binarization without sacrificing performance gains.", "weaknesses": "- The proposed approach section is comprehensive, but its complexity makes it challenging to navigate and comprehend throughout the entire section.\n- The analysis of evaluation is limited to a single task. It would be valuable to explore the potential limitations of PB-LLM in achieving comparable performance across various tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)