Achieving binary weight and activation for LLMs using Post-Training Quantization
Authors: Siqing Song, Chuang Wang, Rui-Qi Wang, Yi Yang, Xu-Yao Zhang
Accepted at: acl25

Achieving Binary Weight and Activation for LLMs Using Post-Training Quantization

Siqing Song1,2, Chuang Wang1,2*, Ruiqi Wang3, Yi Yang3, Xu-Yao Zhang1,2

1MAIS, Institute of Automation, Chinese Academy of Sciences, China
2School of Artificial Intelligence, University of Chinese Academy of Sciences, China
3Central Media Technology Institute, Huawei Technologies Ltd.

songsiqing2023@ia.ac.cn, {chuang.wang.xyz}@nlpr.ia.ac.cn, {wangruiqi12, yangyi16}@huawei.com

Abstract

Quantizing large language models (LLMs) to 1-bit precision significantly reduces computational costs, but existing quantization techniques suffer from noticeable performance degradation when using weight and activation precisions below 4 bits (W4A4). In this paper, we propose a post-training quantization framework with W(1+1)A(1×4) configuration, where weights are quantized to 1 bit with an additional 1 bit for fine-grain grouping and activations are quantized to 1 bit with a 4-fold increase in the number of channels. For weight quantization, we propose utilizing Hessian-aware fine-grained grouping along with an EM-based quantization scheme. For activation quantization, we decompose INT4-quantized activations into a 4 × INT1 format equivalently and simultaneously smooth the scaling factors based on quantization errors, which further reduces the quantization errors in activations. Our method surpasses state-of-the-art (SOTA) LLM quantization baselines on W2A4 across multiple tasks, pushing the boundaries of existing LLM quantization methods toward fully binarized models.

1 Introduction

The enormous computational and memory overhead limits the widespread adoption of large-language models (LLMs). Model quantization methods (Nagel et al., 2021; Huang et al., 2024b) is a major approach to alleviate these challenges. Recent studies (Zhao et al., 2024) on post-training quantization techniques (PTQ) can attain nearly lossless quantization under the W4A4 configuration. However, lower-bit quantization of LLMs remains challenging. Existing methods (Zhao et al., 2024; Ashkboos et al., 2024) still suffer from the performance collapse issue when the number of bits for both weight and activation is less than 4, as shown in Figure 1.

Most of the PTQ LLM quantization work adopts the round-to-nearest (RTN) method (Nagel et al., 2021) as the implementation scheme in practical operations, focusing primarily on how to preprocess the data for RTN quantization. For example, LLM.int8 (Dettmers et al., 2022) and Atom (Zhao et al., 2024) handle outliers with high precision, effectively increasing the numerator of the scaling factor for some key values to reduce quantization error. Another line of works, including SmoothQuant (Xiao et al., 2023), QuaRot (Ashkboos et al., 2024), and DuQuant (Lin et al., 2024) smooth the distribution of data to be quantized via matrix rotation. Although these works can achieve satisfactory 4 bit quantization, they struggle to further reduce the quantization bit-width, which implies that the simple RTN quantization method is not sufficient for very low-bit quantization with high performance.

Another line of works adopt vector quantiza-

![Line plot showing perplexity vs. quantization settings for GPTQ, QuaRot, Atom, and Ours methods](page_795_370_563_370.png)

Figure 1: The perplexity (\( \downarrow \)) of GPTQ, QuaRot, Atom, and our method on the LLAMA1-7B model under various quantization bit-width settings reveals that GPTQ, QuaRot, and Atom exhibit significant performance degradation when weights are quantized to 1 bit, whereas our method still demonstrates language generation capabilities close to those of the original model.
tion technique (Liu et al., 2024a; van Baalen et al., 2024). These approaches quantized the model weights via grouping the weights and assigning the weights in the same group to the same value. Those methods usually yield more accurate quantization models with smaller size, but cannot be speed-up the inference speed, as they have to dequantization the weight to recover float point format before computation.

To address these issues, we propose a quantization framework, that can simultaneously achieve small model size, fast computation, and high-quality inference outputs. It consists of three parts: 1-bit weight quantization with another bit for fine-grain grouping, and \( 1 \times 4 \) bits activation quantization using 4 times 1 bit channel to represent 4 bit quantization. By this way, we are able to compute the inner loop vector product in pure binary operations drastically boosting the inference speed as well as reduce the model size. The main contributions of this work are as follows:

• We propose a W(1+1)A(1×4) post-training quantization framework, with the boolean operations for the inner-loop multiplication-summation operations.

• We present an EM-based algorithm for searching boolean weight quantization with fine-grouping, noticeably improved the performance compared with the usual RTN approach.

• Our method outperforms existing PTQ methods for 2 or 1 bit-widths. On the WikiText2 benchmark, our method achieves perplexities of 8.58 and 8.89 on LLaMA-7B and LLaMA2-7B, respectively, using only the W2A4 quantization setting. This significantly surpasses the existing state-of-the-art methods and is comparable to the performance of 5.68 and 5.47 achieved by the original full-precision models.

2 Related Works

Considerable research efforts have been devoted to the low-resource computation of LLMs (Dettmers et al., 2024; Chen et al., 2024). Existing works have achieved near-lossless quantization at W4A4 through methods such as mixed-precision quantization (Zhao et al., 2024) and outlier smoothing using rotation factors (Ashkboos et al., 2024; Liu et al., 2024b). However, these methods encounter difficulties when attempting to push towards even lower-bit quantization below 4 bits.

Recently, researchers start to seek binarization methods (Courbariaux and Bengio, 2016; Qin et al., 2022) for LLMs. One line of works are based on Quantization-Aware Training (QAT) method. For example, BitNet b1.58 (Wang et al., 2023), BitNet a4.8 (Wang et al., 2024), OneBit (Xu et al., 2024), and FBI-LLM (Ma et al., 2024) have designed 1-bit Transformer architectures specifically for LLMs, replacing original linear layers with specific quantized version. These methods can reduce the quantization bit-width of weights to 1 bit and demonstrate significant advantages over baseline models. However, these QAT methods require substantial computational resources, making it impractical for the quantization of LLMs.

Another line of works explored the possibility of binarizing LLMs within the PTQ framework. BiLLM (Huang et al., 2024a) and STBLLM (Dong et al., 2024), inspired by the distribution characteristics of weight values and the Hessian matrix, adopt binary residual approximations for significant weights and optimal partitioning for group binarization of non-significant weights, which pushes the quantization boundary of LLM weights down to 1 bit. However, these works neglect the quantization of activation values, which obstructs the computational acceleration of the quantized model and ultimately results in relatively slow inference speed.

Most recently, people start to explore fully binarized models for both weight and activation. The works, ABQ-LLM (Zeng et al., 2024) and QBB (Bulat et al., 2024), propose the idea of decomposing a high-bit matrix into a set of binary matrices for accelerated computation. These methods hold promise for achieving INT1 computational acceleration but are constrained by the limitations of quantization bit-width and performance, preventing further improvements.

Compared with state-of-the-art (SOTA) W4A4 quantization efforts, our W(1+1)A(1×4) quantized model can further reduce computational and memory overhead, which further pushes the boundaries of LLM quantization and enriches the research on LLM binarization.

3 Method

In this section, we introduce our binarized weight and activation (BWA) approach, a paradigm de-
signed to simultaneously accelerate inference speed and reduce memory footprint in large language models. We begin by outlining the architecture, and then describe an EM-based method for computing binarized weights with two-level grouping structure, and conclude by discussing the strategies in these work and other relevant methods.

3.1 Binarized weight and activation (BWA) attention

Overall structure of BWA attention  Our BWA framework is a quantization-aware modification of the standard attention module of a LLAMA-like model.

Following the dataflow of the activations in Figure 2, we introduce the overall structure. The input activation is a \( C \)-dimensional FP16 vector. After the RMSNorm layer, we quantize the activation first from FP16 to INT4 using the standard Round-To-Nearest (RTN) method, then further transform it into four boolean variables. Next, the boolean activations are fed into a binarized fully-connected layers to compute key (K), query (Q), and value (V) matrices. The inner-loop computation is boolean, but the outputs recovers FP16 for query branch, and INT4 for key and value branches, i.e., using 4 bit for KV cache. After the attention, we transform the activation vectors to boolean again so that all FC layer in the subsequent projection operations are also binarized.

At the core of quantization task lies the binarization of fully connected (FC) layers, which account for approximately 90% of memory bandwidth and computation. In contrast, other components such as normalization and attention score matrices contribute to the remaining 10%. Consequently, our work primarily focuses on the binarization of FC layers, which are utilized in computing K, Q, V matrices, as well as all projection layers. Since the most computationally expensive operation in these layers is the matrix-vector multiplication within the inner loop, we pay most effort to design a computational efficiency FC layer, where the inner-loop only have binary operations.

Binarized fully-connected layer  The core modification of the proposed BWA attention module is the binarization of the linear layer. To optimize the tradeoff between model accuracy and efficiency, we introduce a two-level weight grouping strategy for weight binarization, and a binarized decomposition to further quantize a 4 bit activations into four 1 bit boolean variables. Consequently, in the core inner-loop computation of the fully connected layers, only boolean operations are involved, drastically simplifying and boosting the computation.

Specifically, a FC layer computes a single token \( y = W x \), where \( W \in \mathbb{R}^{C_{\text{in}} \times C_{\text{out}}} \), \( x \) is the \( C_{\text{in}} \)-dimensional vector represented as an input token, and \( y \) is the output token vector.

1) Channel-wise grouping  Both weights and activations have a width dynamical range, which are hard to quantize using the same scaling parameter. We sort the input channels by the average activation scales of test samples \( X \), i.e., sort according to \( \operatorname{diag}(X X^T) \) in an ascending order. Then, following the sorted order, we group the channels with similar activation scale. In our implementation, we divide the \( C_{\text{out}} = 4096 \) channels into 32 groups. Each group has \( B = 128 \) channels

\[
y_j = \sum_{\ell=0}^{\lceil C_{\text{out}}/B \rceil - 1} \sum_{i=0}^{B-1} W_{j,(B\ell+i)} x_{B\ell+i}, \quad j = 1, 2, \ldots, C_{\text{out}}.
\]

Note that channel-wise group is implemented by a proper permutation of rows and columns of the weight matrix. Therefore, channel-wise grouping does not introduce any additional computational or memory cost.

2) Element-wise grouping  Directly binarization of \( W \) to 1 bit is not enough resulting severe model degeneration forbidding practical usage, especially combined the target to reduce the activation to 4 bit or lower. Similar to BiLLM, we use additional element-level group to further group weight, and quantize them separately.

Define the set \( D_{\ell,1} \subseteq \{0, 1, 2, \ldots, B-1\} \), and its complementary \( D_{\ell,0} = \{0, 1, 2, \ldots, B-1\} \setminus \)
\( D_{\ell,1} \). We quantized the two group separately using 1-bit for each weight element

\[
W_{j,(B\ell+i)} \approx \hat{W}_{j,(B\ell+i)} = \alpha_{j,\ell,s} q_{j,(B\ell+i)} + \beta_{j,\ell,s},
\]
for \( s = 1 \) if \( i \in D_\ell \) otherwise \( s = 0 \), in which \( \alpha_{j,\ell,s} \) and \( \beta_{j,\ell,s} \) are two quantization parameters for each fine-grained group. In the standard LLAMA-TB model, each column-wise group contains has 128 channels. They are further categorized into two element-wise group, of which the sizes may not equal. The element-wise group are represented using a bit map. Together another bit \( q_{j,i} \) representing weight sign. The weight matrix uses \( 1+1 \) bit per element.

This fine-grained level grouping can effectively the accuracy at the expense of increase additional computational cost. On the other hand, with proper usage of bitmap operations, the cost can be suppressed to be marginal. The element-wise grouping has been used in the works (Huang et al., 2024a; Dong et al., 2024), we improve this strategy by leveraging EM-based parameter searching with Hessian metric information to determine the optimal weighted split point in Section 3.2.

3) Activation binarization. The input activation is quantized before feeding into the FC layer from FP16 to INT4 using the standard RTN method (3). Specifically, the quantization process of RTN to obtain \( X_q \) with \( b \) bits is expressed as

\[
X_q = \mathrm{clamp}([\frac{X}{\mu}] + z, 0, 2^k - 1),
\]
where \( \mu = \frac{\max(X)-\min(X)}{2^{k-1}} \) is the scaling parameter, \( z = -[\frac{\min(X)}{\mu}] \) is the shift parameter. The clamp function denotes restricting the quantization result to the range between \([0, 2^k - 1]\). The notion \([\cdot]\) signifies the nearest rounding operation.

To further binarize the activation, we first quantize the full-precision activation values \( x_i \) into 4 bit quantized values \( x_{q,i} \), which can be further decomposed into four 1 bit Boolean variables \( b_{i,a} \) with \( a = 0, 1, 2, 3 \). In the context of 4-bit quantization, \( \hat{x}_i \) represents the dequantized result

\[
x_i \approx \hat{x}_i = \mu(x_{q,i} + z) = \sum_{a=-1}^{3} \mu_a b_{i,a},
\]
where \( \mu_a = \frac{2^a}{\sum_{a=0}^{3} 2^a} \mu \) for \( a = 0, 1, 2, 3 \) and a special notation of \( \mu_{-1} = \mu z \) and \( b_{i,-1} \equiv 1 \) for the shift constant. Moreover, we can also relax \( \mu_k \) as a free quantization parameter to be tuned manually or learn from data. To further enhance the performance of the quantized model, we adopted a scaling factor balancing strategy to minimize the L1 quantization error of activation values. Detailed implementation and analysis are provided in Appendix A. Note that as the weight is reordered and grouped, the elements of the input activation vector will be permuted accordingly.

4) Binarized FC layer We utilize the commutative property of summation to make sure the inner multiplication summation is binary, so that the proposed method is actually boosting the computational speed.

Substituting (2) and (4)

\[
y_j \approx \sum_{\ell=0}^{[C_{out}/B]-1} \sum_{i=0}^{B-1} \left[ (\alpha_{j,\ell,s} q_{j,(B\ell+i)} + \beta_{j,\ell,s}) \right. \\
\left. \sum_{a=-1}^{3} \mu_{\ell,a} b_{(B\ell+i),a} \right] \\
= \sum_{\ell=0}^{[C_{out}/B]-1} \sum_{a=-1}^{3} \mu_{\ell,a} \sum_{s=0,1} \left[ \alpha_{j,\ell,s} v_{j,\ell,s,a} + \beta_{j,\ell,s} r_{j,\ell,s,a} \right],
\]
with the bit-wise inner product \( v_{j,\ell,s,a} \) and counting number of bits valued 1

\[
v_{j,\ell,s,a} = \sum_{i \in D_{j,\ell,s}} q_{j,(B\ell+i)} b_{(B\ell+i),a}, \\
r_{j,\ell,s,a} = \sum_{i \in D_{j,\ell,s}} b_{(B\ell+i),a}.
\]
Since the above two summation only involves bit variable \( q_{j,(B\ell+i)}, b_{(B\ell+i),a} \), and the set \( D_{\ell,1} \) and \( D_{\ell,2} \) is represented by a bitmap. Therefore, they can be implemented efficiently by bitwise XOR/AND and popc operation. Specifically, let \( e_{j,\ell,a} = q_{j,\ell} \wedge b_{\ell,a} \) with \( q_{j,\ell} = [q_{j,B\ell}, q_{j,(B\ell+1)}, \ldots, q_{j,(B\ell+B-1)}] \) and \( b_{\ell,a} = [b_{j,B\ell}, b_{j,(B\ell+1)}, \ldots, b_{j,(B\ell+B-1)}] \). Then, \( v_{j,\ell,s,a}, r_{j,\ell,s,a} \) in (6) can be computed by

\[
\begin{align*}
v_{j,\ell,s=0,a} &= \mathrm{Popc}\ (e_{j,\ell,a} \wedge m_{j,\ell}), \\
v_{j,\ell,s=1,a} &= \mathrm{Popc}\ (e_{j,\ell,a} \wedge (-m_{j,\ell})), \\
r_{j,\ell,s=0,a} &= \mathrm{Popc}\ (b_{\ell,a} \wedge m_{j,\ell}), \\
r_{j,\ell,s=1,a} &= \mathrm{Popc}\ (b_{\ell,a} \wedge (-m_{j,\ell}),
\end{align*}
\]
where \( m_{j,\ell} \in \{0,1\}^B \) is a bitmap indicating the fine-grain group affiliation with \([m_{j,\ell}]_i = 0\) if \( i \in D_{j,\ell,0} \), otherwise \( m_i = 1 \).
The inner loop, which originally be a multiplication-summation operation of 64 numbers, reduces to four bit-wise operations on 128 bits length variables, which can be efficiently implemented on both GPU and CPU.

5) Outlier activation  In this paper, we trick the last channel-wise group as outlier, and use 8 bit to quantize these channels. It is hardware-friendly and can efficiency implemented via reordering the channels (Yuan et al., 2023; Zhao et al., 2024). Different from other methods, which usually use more channels for outliers, we only use minimal 1 group, which only contain 128 channels for the outliers (both in activation and weight). This minimizes the outlier overhead to approximately 3% of the total channels in 8 bit and representing all other normal channels in 1 bit. Experiments for set more outlier groups are done in Appendix C, which can further improve the performance at the expense of costing extra bits.

3.2 Weight binarization and parameterization by Fine-Grained Group Hessian-Aware Quantization

In this subsection, we propose an EM-based method to determine the binarized values of the weight matrices and the associated parameters for dequantization.

The weight matrix is quantized according to (2), where \( q_{j,(B+i)} \) represents the binarized weight, and \( \alpha_{j,\ell,s} \), \( \beta_{j,\ell,s} \) denote the scaling and shifting parameters, respectively, used for dequantization. Furthermore, we also require a bit map matrix of identical dimension to the weight matrix, which serves to determine the fine-group affiliation of each corresponding element.

We start from minimizing the L2 norm of the weight matrix \( W \), utilizing an approximate Hessian weight as proposed in (Hassibi and Stork, 1992; Frantar and Alistarh, 2022)

\[
\mathcal{L}_W = \left\| \frac{1}{\operatorname{diag}(H^{-1})} (W - \hat{W}) \right\|_2^2,
\]

where the Hessian matrix \( H = X X^T \) is determined using a validation dataset, encapsulates information about the activation values during the computation of linear layers and gradient information during back propagation.

In our binarized parameterization (2), each channel (indexed by \( j \)) and each channel-wise group (indexed by \( \ell \)) are parameterized independently. Without loss of generality, we only focus on a single channel-wise group, which has \( B \) weight elements \( w_i, i \in 0, 1, \ldots, B-1 \). The binary representation of \( W \) with a binary element-wise group can at most have four different float-point values. Therefore, the quantization problem boils down to a 1-D clustering problem to determine 4 clusters, centered at \( \hat{w}(0,0), \hat{w}(0,1), \hat{w}(1,0), \hat{w}(1,1) \). Formally, for each group, we solve the following minimization problem

\[
\min_{s, q \in \{0,1\}^B, \hat{w} \in \mathbb{R}^4} \sum_{i=0}^{B-1} \left( w_i - \hat{w}(s_i, q_i) \right)^2 / \operatorname{diag}(H^{-1})_i ,
\]

where \( s, q \) represents the fine-group affiliation and the binary weight value respectively. Knowing the four values of \( \hat{w} \), one can recover the scaling and shifting parameters \( \alpha_{j,\ell,s} \) and \( \beta_{j,\ell,s} \) by the method of undetermined coefficients formalized as a set of 4-D linear equations.

We implement a revised EM algorithm to solve the above optimization problem, as shown in Algorithm 1. The detailed steps are as follows:

E-step: Given the centroids \( \hat{w} \), calculate the weighted distances from all weight elements \( w \) in the current layer to each centroid, classify \( w \) into the nearest class \( C \), and determine the grouping accordingly, ensuring that the loss value in Equation (9) is minimized.

M-step: Given the grouping \( D \) and the centroid class \( C \) to which each weight element \( w \) belongs, determine the \( \mu \) centroid value that minimizes the loss in Equation (9) under the current grouping and centroid assignment.

In practical operation, the execution of the EM algorithm resembles a one-dimensional K-means clustering process. Additionally, to reduce the quantization error of the model, we incorporate the block compensation strategy from the GPTQ (Frantar et al., 2022) framework into the quantization process. We conduct the update process in steps according to the block size set in GPTQ, with error compensation inserted between each step. This approach further reduces the overall quantization error of the model.

3.3 Remarks on quantization strategies

We provide additional discussion on the quantization strategies employed in our approach, comparing them to existing methods such as the RTN quantization, vector quantization for weight quantization, and a recent work on binarized residual decomposition for activation quantization.
Algorithm 1 Main Framework of our method

Require:
    \( \mathbf{W} \in \mathbb{R}^{C_{out} \times C_{in}} \), weight matrix
    \( \mathbf{X} \in \mathbb{R}^{T \times C_{in}} \), calibration data
    \( B \), block size
    \( K \), outliers keep in INT8
    \( iters \), EM steps

Ensure:
    \( \mathbf{B} \), weights after dequantization
    \( \mathbf{D} \), fine-grained group information

1: \( \mathbf{W} = \text{reorder}(\mathbf{W}, \text{diag}(\mathbf{XX}^T)) \)
2: \( \mathbf{H} = 2\mathbf{XX}^T \) \( \triangleright \) Hessian matrix
3: \( \mathbf{H}^c = \text{Cholesky}((\mathbf{H} + \lambda I)^{-1}) \)
4: \( \mathbf{B} = 0_{C_{in} \times C_{out}} \)
5: for \( i = 0, B, 2B, ..., C_{out} - K - B \) do
6:     \( \mathbf{W}^p = \mathbf{W}_{:,i:i+B} \)
7:     \( \mathbf{H}^{cp} = \text{diag}(\mathbf{H}^c_{i:i+B,i:i+B}) \)
8:     \( \mathbf{C} = \text{init_centers}(\mathbf{W}^p, \mathbf{H}^{cp}) \)
9:     for \( j = 1 \) to \( iters \) do
10:         \( \mathbf{D} = \text{get_groups}(\mathbf{W}^p, \mathbf{H}^{cp}, \mathbf{C}) \) \( \triangleright \) E-step
11:         \( \mathbf{I} = \text{get_clusters}(\mathbf{W}^p, \mathbf{H}^{cp}, \mathbf{C}, \mathbf{D}) \) \( \triangleright \) M-step
12:         \( \mathbf{C} = \text{update_centers}(\mathbf{W}^p, \mathbf{H}^{cp}, \mathbf{I}, \mathbf{D}) \) \( \triangleright \) M-step
13:     end for
14:     \( \mathbf{B}_{:,i:i+B} = \text{binary}(\mathbf{W}^p, \mathbf{C}, \mathbf{I}, \mathbf{D}) \)
15:     \( \mathbf{E} = \frac{\langle \mathbf{W}^p, \mathbf{B} \rangle}{\|\mathbf{H}^{cp}\|} \)
16:     \( \mathbf{W}_{:,i+B:C_{out}-K} = \mathbf{W}_{:,i+B:C_{out}-K} - \mathbf{E} \cdot \mathbf{H}^c_{i:i+B,i:i+B} \)
17: end for
18: \( \mathbf{B}_{:,C_{out}-K:} = \text{quant\_int8}(\mathbf{W}_{:,C_{out}-K:}) \)
19: Return \( \mathbf{B}, \mathbf{D} \)

RTN quantization v.s. fine-grained group binary weight In our method, we utilize 1 bit to store binary weights and an additional 1 bit to represent element-wise group affiliation, effectively using 2 bit of information. It enables each weight element to take on four distinct values. In contrast to the RTN quantization method widely used in other post-training quantization (PTQ) methods, where dequantized values are equally spaced, our model allows the four values to be chosen arbitrarily, which is optimized by the proposed EM-based algorithm.

Vector quantization v.s. fine-group binary weight The optimization process for dequantization parameters in our approach is similar to that employed in vector-quantization-based methods (Frantar et al., 2022; van Baalen et al., 2024), where \( 2^n \) floating-point values are stored as representative values for \( n \) bit quantization. In these approaches, dequantization must be performed before computing the vector inner product. In contrast, our method further parameterizes the representatives using binary weights and fine-grained group bits, along with floating-point scaling and shift parameters. This enables us to compute the vector inner product using pure Boolean operations as shown in (7), resulting in a significant boost in computational speed. The details of quantization and dequantization after EM Algorithm are shown in Appendix B.

Binarized residual decomposition and 1 × 4 bit representation of activation

The work (Zeng et al., 2024) explored the approach of transforming arbitrary integer weight and activation WxAa into xa × W1A1 to achieve computational acceleration. It make use of bit operation to computation inner-loop vector product, but the original work can not get good below W4A4. On the other hand, the expansion of high bits weight and activation usually result heavy over head, as the number of channel vectors (relates to memory bandwidth) roughly from \( (x + a) \) bits to \( (xa) \) bits. In this work, we manage to reduce W(1+1)A(1×4), together with bitmap operation on the fine-grain group, the over-head cost is marginal.

4 Experiments

Setup. We implemented our method on the PyTorch (Paszke et al., 2019) framework, where all linear layer weights in the original model are quantized to 1+1 bit, and input activations of all linear layers are quantized to 1×4 bits. For weights, we adopt per-channel asymmetric quantization with a clipping ratio set to 1.0 across all experiments, utilizing the GPTQ quantization framework to compensate for quantization errors. For activations, we employ per-token asymmetric quantization with a clipping ratio of 1.0. To optimize performance, we use RTN for dynamic quantization of the activation matrix. For KV caches, we uniformly apply 4 bits quantization to store and load. The quantization group size is 128, and the number of outlier channels is 128 (approximately 3% of all channels). We use 128 random samples from the WikiText2 (Merity et al., 2016) training set as the calibration dataset, with a sequence length of 2048. All experiments were conducted more than three times, and the average values were recorded.
Table 1: Perplexity(\( \downarrow \)) and Zero-shot QA accuracy(\( \uparrow \)) results under the W4A4 and W2A4 settings on LLAMA1-7B and LLAMA2-7B. "FP16" denotes the performance of the original model represented in FLOAT16 format, with the best quantization performance highlighted in bold. The experimental results on the 13B model are presented in Table 7, and Table 8.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Bits</th>
    <th rowspan="2">Method</th>
    <th colspan="2">Wiki.</th>
    <th colspan="2">PTB.</th>
    <th colspan="2">C4.</th>
    <th colspan="2">PIQA.</th>
    <th colspan="2">ARC-E.</th>
    <th colspan="2">ARC-C.</th>
    <th colspan="2">BoolQ.</th>
    <th colspan="2">Hella.</th>
    <th colspan="2">Wino.</th>
    <th rowspan="2">Avg.</th>
  </tr>
  <tr>
    <th>Wiki.\(_\downarrow\)</th>
    <th>PTB.\(_\downarrow\)</th>
    <th>C4.\(_\downarrow\)</th>
    <th>PIQA.\(_\uparrow\)</th>
    <th>ARC-E.\(_\uparrow\)</th>
    <th>ARC-C.\(_\uparrow\)</th>
    <th>BoolQ.\(_\uparrow\)</th>
    <th>Hella.\(_\uparrow\)</th>
    <th>Wino.\(_\uparrow\)</th>
  </tr>
  <tr>
    <td rowspan="5">LLAMA1-7B</td>
    <td>FP16</td>
    <td>-</td>
    <td>5.68</td>
    <td>27.34</td>
    <td>7.08</td>
    <td>77.37</td>
    <td>52.48</td>
    <td>41.38</td>
    <td>73.06</td>
    <td>73.00</td>
    <td>67.01</td>
    <td>64.05</td>
  </tr>
  <tr>
    <td>W4A4</td>
    <td>QuaRot</td>
    <td>6.41</td>
    <td>49.73</td>
    <td>8.43</td>
    <td>74.81</td>
    <td>50.13</td>
    <td>38.74</td>
    <td>70.98</td>
    <td>68.80</td>
    <td>61.56</td>
    <td>61.01</td>
  </tr>
  <tr>
    <td>Atom</td>
    <td>6.30</td>
    <td>30.26</td>
    <td>7.98</td>
    <td>75.35</td>
    <td>51.60</td>
    <td>36.69</td>
    <td>70.86</td>
    <td>67.27</td>
    <td>64.33</td>
    <td>62.21</td>
  </tr>
  <tr>
    <td>W2A4</td>
    <td>QuaRot</td>
    <td>14.39</td>
    <td>222.95</td>
    <td>27.70</td>
    <td>59.52</td>
    <td>37.88</td>
    <td>26.62</td>
    <td>62.20</td>
    <td>41.56</td>
    <td>54.62</td>
    <td>47.07</td>
  </tr>
  <tr>
    <td>Atom</td>
    <td>16.65</td>
    <td>296.42</td>
    <td>33.86</td>
    <td>57.24</td>
    <td>35.23</td>
    <td>26.11</td>
    <td>53.98</td>
    <td>36.71</td>
    <td>59.13</td>
    <td>43.75</td>
    <td>43.57</td>
  </tr>
  <tr>
    <td>W(1+1)A(1x4)</td>
    <td>BiLLM</td>
    <td>18.04</td>
    <td>217.37</td>
    <td>31.39</td>
    <td>61.20</td>
    <td>35.20</td>
    <td>26.70</td>
    <td>58.70</td>
    <td>51.10</td>
    <td>55.58</td>
    <td>45.58</td>
  </tr>
  <tr>
    <td>W(1+1)A(1x4)</td>
    <td>Ours</td>
    <td><b>8.58</b></td>
    <td><b>76.09</b></td>
    <td><b>12.27</b></td>
    <td><b>68.88</b></td>
    <td><b>45.03</b></td>
    <td><b>30.89</b></td>
    <td><b>69.63</b></td>
    <td><b>55.41</b></td>
    <td><b>59.35</b></td>
    <td><b>54.87</b></td>
  </tr>
  <tr>
    <td rowspan="5">LLAMA2-7B</td>
    <td>FP16</td>
    <td>-</td>
    <td>5.47</td>
    <td>22.51</td>
    <td>6.97</td>
    <td>76.93</td>
    <td>53.58</td>
    <td>40.53</td>
    <td>71.07</td>
    <td>72.96</td>
    <td>67.17</td>
    <td>63.71</td>
  </tr>
  <tr>
    <td>W4A4</td>
    <td>QuaRot</td>
    <td>6.32</td>
    <td>71.21</td>
    <td>8.67</td>
    <td>74.32</td>
    <td>51.60</td>
    <td>38.23</td>
    <td>68.41</td>
    <td>69.24</td>
    <td>61.56</td>
    <td>60.89</td>
  </tr>
  <tr>
    <td>Atom</td>
    <td>6.18</td>
    <td>27.94</td>
    <td>8.05</td>
    <td>75.24</td>
    <td>52.74</td>
    <td>37.12</td>
    <td>71.16</td>
    <td>67.89</td>
    <td>63.93</td>
    <td>62.58</td>
  </tr>
  <tr>
    <td>W2A4</td>
    <td>QuaRot</td>
    <td>49.98</td>
    <td>571.22</td>
    <td>80.14</td>
    <td>54.41</td>
    <td>28.45</td>
    <td>23.21</td>
    <td>57.89</td>
    <td>28.57</td>
    <td>48.15</td>
    <td>40.11</td>
  </tr>
  <tr>
    <td>Atom</td>
    <td>19.49</td>
    <td>508.82</td>
    <td>39.46</td>
    <td>56.69</td>
    <td>32.32</td>
    <td>23.21</td>
    <td>58.53</td>
    <td>35.74</td>
    <td>49.49</td>
    <td>42.66</td>
    <td>42.66</td>
  </tr>
  <tr>
    <td>W(1+1)A(1x4)</td>
    <td>BiLLM</td>
    <td>16.23</td>
    <td>381.87</td>
    <td>51.42</td>
    <td>60.59</td>
    <td>26.90</td>
    <td>24.01</td>
    <td>61.80</td>
    <td>52.45</td>
    <td>49.88</td>
    <td>37.61</td>
  </tr>
  <tr>
    <td>W(1+1)A(1x4)</td>
    <td>Ours</td>
    <td><b>8.89</b></td>
    <td><b>69.46</b></td>
    <td><b>12.74</b></td>
    <td><b>68.72</b></td>
    <td><b>46.13</b></td>
    <td><b>30.55</b></td>
    <td><b>66.12</b></td>
    <td><b>55.76</b></td>
    <td><b>58.01</b></td>
    <td><b>54.22</b></td>
  </tr>
</table>

Table 2: Perplexity(\( \downarrow \)) and Zero-shot QA accuracy(\( \uparrow \)) results under the W4A4 and W2A4 settings on Vicuna family. "FP16" denotes the performance of the original model represented in FLOAT16 format, with the best quantization performance highlighted in bold.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Bits</th>
    <th rowspan="2">Method</th>
    <th colspan="2">Wiki.\(_\downarrow\)</th>
    <th colspan="2">PTB.\(_\downarrow\)</th>
    <th colspan="2">C4.\(_\downarrow\)</th>
    <th colspan="2">PIQA.\(_\uparrow\)</th>
    <th colspan="2">ARC-E.\(_\uparrow\)</th>
    <th colspan="2">ARC-C.\(_\uparrow\)</th>
    <th colspan="2">BoolQ.\(_\uparrow\)</th>
    <th colspan="2">Hella.\(_\uparrow\)</th>
    <th colspan="2">Wino.\(_\uparrow\)</th>
    <th rowspan="2">Avg.\(_\uparrow\)</th>
  </tr>
  <tr>
    <th>Wiki.\(_\downarrow\)</th>
    <th>PTB.\(_\downarrow\)</th>
    <th>C4.\(_\downarrow\)</th>
    <th>PIQA.\(_\uparrow\)</th>
    <th>ARC-E.\(_\uparrow\)</th>
    <th>ARC-C.\(_\uparrow\)</th>
    <th>BoolQ.\(_\uparrow\)</th>
    <th>Hella.\(_\uparrow\)</th>
    <th>Wino.\(_\uparrow\)</th>
  </tr>
  <tr>
    <td rowspan="5">Vicuna-v1.5-7B</td>
    <td>FP16</td>
    <td>-</td>
    <td>6.78</td>
    <td>26.78</td>
    <td>8.55</td>
    <td>77.80</td>
    <td>56.06</td>
    <td>39.93</td>
    <td>75.69</td>
    <td>71.06</td>
    <td>67.80</td>
    <td>64.72</td>
  </tr>
  <tr>
    <td>W4A4</td>
    <td>QuaRot</td>
    <td>7.32</td>
    <td>32.44</td>
    <td>10.87</td>
    <td>73.01</td>
    <td>52.20</td>
    <td>37.71</td>
    <td>72.46</td>
    <td>68.07</td>
    <td>62.12</td>
    <td>62.12</td>
  </tr>
  <tr>
    <td>Atom</td>
    <td>7.22</td>
    <td>31.75</td>
    <td>9.36</td>
    <td>75.14</td>
    <td>55.60</td>
    <td>37.63</td>
    <td>72.25</td>
    <td>67.08</td>
    <td>64.40</td>
    <td>64.42</td>
    <td>64.42</td>
  </tr>
  <tr>
    <td>W2A4</td>
    <td>QuaRot</td>
    <td>39.51</td>
    <td>226.50</td>
    <td>65.17</td>
    <td>55.66</td>
    <td>33.38</td>
    <td>22.75</td>
    <td>62.08</td>
    <td>37.11</td>
    <td>50.51</td>
    <td>44.03</td>
    <td>44.03</td>
  </tr>
  <tr>
    <td>Atom</td>
    <td>15.96</td>
    <td>107.68</td>
    <td>25.13</td>
    <td>56.64</td>
    <td>31.90</td>
    <td>29.61</td>
    <td>64.07</td>
    <td>46.30</td>
    <td>55.33</td>
    <td>47.31</td>
    <td>47.31</td>
  </tr>
  <tr>
    <td>W(1+1)A(1x4)</td>
    <td>Ours</td>
    <td><b>9.51</b></td>
    <td><b>45.61</b></td>
    <td><b>13.35</b></td>
    <td><b>69.97</b></td>
    <td><b>50.00</b></td>
    <td><b>33.45</b></td>
    <td><b>71.96</b></td>
    <td><b>57.81</b></td>
    <td><b>59.43</b></td>
    <td><b>57.10</b></td>
    <td><b>57.10</b></td>
  </tr>
  <tr>
    <td rowspan="5">Vicuna-v1.5-13B</td>
    <td>FP16</td>
    <td>-</td>
    <td>5.95</td>
    <td>25.15</td>
    <td>7.78</td>
    <td>78.40</td>
    <td>56.44</td>
    <td>44.80</td>
    <td>76.51</td>
    <td>74.63</td>
    <td>69.06</td>
    <td>66.64</td>
  </tr>
  <tr>
    <td>W4A4</td>
    <td>QuaRot</td>
    <td>6.81</td>
    <td>54.16</td>
    <td>9.64</td>
    <td>74.81</td>
    <td>51.43</td>
    <td>40.53</td>
    <td>70.73</td>
    <td>70.96</td>
    <td>62.12</td>
    <td>62.51</td>
    <td>62.51</td>
  </tr>
  <tr>
    <td>Atom</td>
    <td>6.32</td>
    <td>27.64</td>
    <td>8.25</td>
    <td>76.44</td>
    <td>54.67</td>
    <td>43.34</td>
    <td>74.83</td>
    <td>72.07</td>
    <td>68.46</td>
    <td>65.37</td>
    <td>65.37</td>
    <td>65.37</td>
  </tr>
  <tr>
    <td>W2A4</td>
    <td>QuaRot</td>
    <td>18.32</td>
    <td>273.88</td>
    <td>37.86</td>
    <td>56.69</td>
    <td>36.49</td>
    <td>26.11</td>
    <td>62.42</td>
    <td>38.40</td>
    <td>53.99</td>
    <td>45.54</td>
    <td>45.54</td>
  </tr>
  <tr>
    <td>Atom</td>
    <td>19.84</td>
    <td>174.63</td>
    <td>36.39</td>
    <td>54.95</td>
    <td>34.13</td>
    <td>25.68</td>
    <td>61.74</td>
    <td>37.55</td>
    <td>52.17</td>
    <td>44.37</td>
    <td>44.37</td>
    <td>44.37</td>
  </tr>
  <tr>
    <td>W(1+1)A(1x4)</td>
    <td>Ours</td>
    <td><b>7.01</b></td>
    <td><b>49.71</b></td>
    <td><b>11.45</b></td>
    <td><b>71.44</b></td>
    <td><b>52.46</b></td>
    <td><b>38.05</b></td>
    <td><b>68.93</b></td>
    <td><b>62.34</b></td>
    <td><b>62.34</b></td>
    <td><b>59.35</b></td>
    <td><b>59.35</b></td>
  </tr>
</table>

Table 3: Massive Multitask Language Understanding (MMLU) results (%) under the W2A4 settings on LLAMA1-7B. "FP16" denotes the performance of the original model represented in FLOAT16 format, with the best quantization performance highlighted in bold.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Bits</th>
    <th rowspan="2">Method</th>
    <th colspan="2">MMLU-STEM\(_\uparrow\)</th>
    <th colspan="2">MMLU-humanities\(_\uparrow\)</th>
    <th colspan="2">MMLU-social science\(_\uparrow\)</th>
    <th colspan="2">MMLU-others\(_\uparrow\)</th>
    <th rowspan="2">Avg.\(_\uparrow\)</th>
  </tr>
  <tr>
    <th>MMLU-STEM\(_\uparrow\)</th>
    <th>MMLU-humanities\(_\uparrow\)</th>
    <th>MMLU-social science\(_\uparrow\)</th>
    <th>MMLU-others\(_\uparrow\)</th>
  </tr>
  <tr>
    <td rowspan="3">LLAMA1-7B</td>
    <td>FP16</td>
    <td>-</td>
    <td>30.9</td>
    <td>33.2</td>
    <td>37.9</td>
    <td>37.9</td>
    <td>34.8</td>
  </tr>
  <tr>
    <td>W2A4</td>
    <td>Atom</td>
    <td>25.6</td>
    <td>24.0</td>
    <td>23.6</td>
    <td>27.6</td>
    <td>25.1</td>
  </tr>
  <tr>
    <td>W(1+1)A(1x4)</td>
    <td>Ours</td>
    <td><b>29.4</b></td>
    <td><b>26.0</b></td>
    <td><b>30.5</b></td>
    <td><b>27.2</b></td>
    <td><b>28.0</b></td>
  </tr>
</table>

Models and Datasets. We apply our method to the open-source LLAMA1 (7B, 13B) (Touvron et al., 2023a), LLAMA2 (7B, 13B) (Touvron et al., 2023b), and Vicuna (7B, 13B) (Chiang et al., 2023) models and evaluate their performance on language generation, commonsense QA, and language understanding tasks. The primary metric for language generation tasks is perplexity, assessed on datasets including WikiText2, PTB (Marcus et al., 1994), and C4 (Raffel et al., 2020). For commonsense QA tasks, the main metric is zero-shot accuracy, evaluated on datasets such as PIQA (Bisk et al., 2020), ARC (Clark et al., 2018), BoolQ (Clark et al., 2019), HellaSwag (Zellers et al., 2019), and WinoGrande (Sakaguchi et al., 2021). Except for the C4 dataset, where we randomly select 256 samples of length 2048 from the test set for evaluation, we utilize the entire test set portion of these datasets for our testing. For language understanding tasks, we mainly tested our method on the Massive Multitask Language Understanding (MMLU) (Hendrycks et al., 2020) benchmark.

Baseline. We compare our approach with state-of-the-art (SOTA) PTQ methods for weights and
activations. Since few existing methods explore the W2A4 quantization setting, we implement W2A4 quantization for all compared methods to ensure fairness before evaluation. Our main baselines include Atom (Zhao et al., 2024), QuaRot (Ashkboos et al., 2024), and BiLLM (Huang et al., 2024a). Atom and QuaRot are SOTA methods under the W4A4 quantization setting, while BiLLM is the SOTA for the W(1+1)A16 quantization setting.

4.1 Main Results

Language Generation Tasks. We assess the perplexity of our method on language generation tasks and conduct a fair comparison with existing SOTA methods. As shown in Table 1 and Table 2, Atom and QuaRot, as SOTA methods under the W4A4 setting, experience significant performance drops under the W2A4 setting. In contrast, our method significantly outperforms these methods on all datasets under the W(1+1)A(1×4) setting which is equivalent to W2A4, and our method’s perplexity even approaches that of the FP16 model. It is noteworthy that BiLLM also utilizes an additional 1 bit to store extra fine-grained grouping information, thus we consider it as a W(1+1)A16 approach. As a similar method that employs fine-grained grouping like our method, its performance under the W(1+1)A16 configuration is significantly outperformed by our method with the W(1+1)A(1×4) setting. Furthermore, when its activation values are quantized to 4 bits, the performance of BiLLM rapidly deteriorates.

Zero-Shot Tasks. We also evaluate our method on six important zero-shot tasks. Table 1 and Table 2 presents the comparison results between our method and the baselines. our method significantly outperforms existing methods under the W2A4 quantization setting and demonstrates stable accuracy, approaching the performance of the FP16 model.

Language Understanding Tasks. We simultaneously evaluated the language understanding performance of our quantization method across 47 distinct subdomains of downstream tasks in the MMLU benchmark. Categorized performance reports in Table 3 demonstrate that our approach surpasses existing SOTA quantization solutions while approaching the capabilities of the full-precision LLAMA1-7B model.

![A comparison of the time cost between the W(1+1)A(1×4) kernel and the INT4, INT8 kernels of CUTLASS for matrix multiplication on the A6000. More results are presented in Figure 4.](page_1012_153_377_295.png)

Figure 3: A comparison of the time cost between the W(1+1)A(1×4) kernel and the INT4, INT8 kernels of CUTLASS for matrix multiplication on the A6000. More results are presented in Figure 4.

4.2 Performance Analysis

Speedup. To evaluate the inference acceleration provided by our method, we adopt the kernel implementation from ABQ-LLM (Zeng et al., 2024), which supports decomposing arbitrary-dimensional WxAx operations into multiple W1A1 computations and leverages the acceleration effect of INT1 multiplication for significant speedup in matrix multiplication. We test the speedup of our method under the W(1+1)A(1×4) setting on an A6000 compared to different bit-width settings supported by CUTLASS, such as W8A8 and W4A4. Because the weights actually involved in the computations in our method are 1 bit, with an additional 1 bit solely used for storing fine-grained grouping information, we consider our method as a quantization method that can be viewed as a decomposition from W2A4 downwards in terms of computational acceleration comparison. As shown in Figure 3, in terms of single-layer matrix computations, our method exhibits a more substantial speedup in comparison to other bit-width settings, surpassing the kernel acceleration of CUTLASS by a factor of 3 in matrix computations. This demonstrates that the approach of using INT1 for acceleration in our method can fully leverage the speedup benefits of low-bit computations. Moreover, since both the weights and activations in our method are quantized to very low bit-widths, the additional computational overhead introduced by the decomposition does not significantly impact the gains achieved through INT1 computation.
Table 4: Ablation experiments on the effects of Minimum distance quantization (EM) and fine-grained group, with all experimental results based on LLAMA1-7B and a group size of 128.

<table>
  <tr>
    <th>Minimum distance quantization</th>
    <th>Fine-grained group</th>
    <th>Wiki. PPL↓</th>
    <th>Avg. Accuracy↑</th>
  </tr>
  <tr>
    <td>×</td>
    <td>×</td>
    <td>6348.82</td>
    <td>36.44</td>
  </tr>
  <tr>
    <td>✓</td>
    <td>×</td>
    <td>126.89</td>
    <td>36.00</td>
  </tr>
  <tr>
    <td>×</td>
    <td>✓</td>
    <td>16.65</td>
    <td>43.31</td>
  </tr>
  <tr>
    <td>✓</td>
    <td>✓</td>
    <td>8.58</td>
    <td>54.87</td>
  </tr>
</table>

Table 5: Ablation experiments on the effects of different quantized components used in our method, with all experimental results based on LLAMA1-7B and a group size of 128.

<table>
  <tr>
    <th>Quantization Method</th>
    <th>Wiki. ↓</th>
  </tr>
  <tr>
    <td>LLAMA-7B FP16</td>
    <td>5.68</td>
  </tr>
  <tr>
    <td>W1A4 GPTQ (Group size 128)</td>
    <td>216713</td>
  </tr>
  <tr>
    <td>+ Keep 128 outlier channels in INT8</td>
    <td>6749</td>
  </tr>
  <tr>
    <td>+ Minimum distance quantization</td>
    <td>126.89</td>
  </tr>
  <tr>
    <td>+ Fine-grained group, W(1+1)</td>
    <td>8.69</td>
  </tr>
  <tr>
    <td>+ Hessian-weighted distance metric</td>
    <td>8.65</td>
  </tr>
  <tr>
    <td>+ Binarized Residual Decomposition, A(1×4)</td>
    <td>8.58</td>
  </tr>
</table>

4.3 Ablation Studies

To evaluate the effectiveness of different quantization modules in our method, we compared the accuracy gains or losses among various quantization techniques employed within our method. The results presented in Table 5 demonstrate that outlier handling, Minimum distance quantization, and fine-grained grouping, as the basic processing schemes in our method. Each step significantly enhances the performance of the quantized model, effectively mitigating the performance collapse issue observed in the W1A4 quantized model. The introduction of the Hessian-weighted distance metric and binarized residual decomposition further boosts the quantized model’s performance. Although numerically, the improvements in perplexity brought about by these two methods are not substantial, this is because the previous enhancements have already pushed the performance metrics close to those of the original model, leaving limited room for further improvement. Theoretically, the Hessian-weighted distance metric reveals a measure of weight importance, while the binarized residual decomposition elucidates the direction of performance enhancement after binarization decomposition.

5 Limitations

Although our our method can achieve fully binarized computation to reduce computational overhead during inference, a trade-off must be made regarding the accuracy of the quantized model. Consequently, we need to utilize additional bits to store the information of the quantized matrices, with the actual storage bits for weights and activations equivalent to 2 bits and 4 bits, respectively. This implies that our model has not been compressed to the theoretical extreme of a boolean model, leaving room for further improvement. Meanwhile, although the performance of our our method across various evaluation tasks is close to that of the pre-quantized model, it is not truly lossless quantization. This loss indicates that the quantized model has not fully restored the representational capacity of the original model. In the future, we consider employing methods that integrate quantization-aware training to further enhance the efficiency and performance of our method.

6 Acknowledgement

This work was supported in part by National Science and Technology Major Project under Grant 2022ZD0116500, National Natural Science Foundation of China under Grant 62222609, CAS Project for Young Scientists in Basic Research under Grant YSBR-083, Pioneer Hundred Talents Program of CAS under Grant Y9S9MS08 and Grant E2S40101, and Beijing Municipal Science and Technology Project under Grant No. Z231100010323005.

References

Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. 2024. Quarat: Outlier-free 4-bit inference in rotated llms. arXiv preprint arXiv:2404.00456.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020. Piqqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439.

Adrian Bulat, Yassine Ouali, and Georgios Tzimiropoulos. 2024. Qbb: Quantization with binary bases for llms. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.

Yi Chen, Jian Xu, Xu-Yao Zhang, Wen-Zhuo Liu, Yang-Yang Liu, and Cheng-Lin Liu. 2024. Recoverable compression: A multimodal vision token recovery mechanism guided by text information. arXiv preprint arXiv:2409.01179.

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna.lmsys.org (accessed 14 April 2023), 2(3):6.

Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044.

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457.

M Courbariaux and Y Bengio. 2016. Binarized neural networks: Training neural networks with weights and activations constrained to+ 1 or -1. arxiv. arXiv preprint arXiv:1602.02830.

Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35:30318–30332.

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2024. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36.

Peijie Dong, Lujun Li, Yuedong Zhong, Dayou Du, Ruibo Fan, Yuhai Chen, Zhenheng Tang, Qiang Wang, Wei Xue, Yike Guo, et al. 2024. Stblim: Breaking the 1-bit barrier with structured binary llms. arXiv preprint arXiv:2408.01803.

Elias Frantar and Dan Alistarh. 2022. Optimal brain compression: A framework for accurate post-training quantization and pruning. Advances in Neural Information Processing Systems, 35:4475–4488.

Elias Frantar, Saleh Askhboos, Torsten Hoefler, and Dan Alistarh. 2022. Vptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323.

Babak Hassibi and David Stork. 1992. Second order derivatives for network pruning: Optimal brain surgeon. In Advances in Neural Information Processing Systems, volume 5. Morgan-Kaufmann.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300.

Wei Huang, Yangdong Liu, Haotong Qin, Ying Li, Shiming Zhang, Xianglong Liu, Michele Magno, and Xiaojuan Qi. 2024a. Billm: Pushing the limit of post-training quantization for llms. arXiv preprint arXiv:2402.04291.

Wei Huang, Xudong Ma, Haotong Qin, Xingyu Zheng, Chengtao Lv, Hong Chen, Jie Luo, Xiaojuan Qi, Xianglong Liu, and Michele Magno. 2024b. How good are low-bit quantized llama3 models? an empirical study. arXiv e-prints, pages arXiv–2404.

Haokun Lin, Haobo Xu, Yichen Wu, Jingzhi Cui, Yingtao Zhang, Linzhan Mou, Lingqi Song, Zhenan Sun, and Ying Wei. 2024. Duquant: Distributing outliers via dual transformation makes stronger quantized llms. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.

Yifei Liu, Jicheng Wen, Yang Wang, Shengyu Ye, Li Lynn Zhang, Ting Cao, Cheng Li, and Mao Yang. 2024a. Vptq: Extreme low-bit-vector post-training quantization for large language models. arXiv preprint arXiv:2409.17066.

Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghu Raman Krishnamoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. 2024b. Spinquant–llm quantization with learned rotations. arXiv preprint arXiv:2405.16406.

Liqun Ma, Mingjie Sun, and Zhiqiang Shen. 2024. Fbi-llm: Scaling up fully binarized llms from scratch via autoregressive distillation. arXiv preprint arXiv:2407.07093.

Mitch Marcus, Grace Kim, Mary Ann Marcinkiewicz, Robert MacIntyre, Ann Bies, Mark Ferguson, Karen Katz, and Britta Schasberger. 1994. The penn treebank: Annotating predicate argument structure. In Human Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994.

Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843.

Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bondarenko, Mart Van Baalen, and Tijmen Blankevoort. 2021. A white paper on neural network quantization. arXiv preprint arXiv:2106.08295.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32.

Haotong Qin, Yifu Ding, Mingyuan Zhang, Qinghua Yan, Aishan Liu, Qingqing Dang, Ziwei Liu, and Xianglong Liu. 2022. Bibtet: Accurate fully binarized bert. arXiv preprint arXiv:2203.06390.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67.

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023b. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.

Mart van Baalen, Andrey Kuzmin, Markus Nagel, Peter Couperus, Cedric Bastoul, Eric Mahurin, Tijmen Blankevoort, and Paul Whatmough. 2024. Gptvq: The blessing of dimensionality for llm quantization. arXiv preprint arXiv:2402.15319.

Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang, Yi Wu, and Furu Wei. 2023. Bitnet: Scaling 1-bit transformers for large language models. arXiv preprint arXiv:2310.11453.

Hongyu Wang, Shuming Ma, and Furu Wei. 2024. Bitnet a4. 8: 4-bit activations for 1-bit llms. arXiv preprint arXiv:2411.04965.

Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. 2023. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pages 38087–38099. PMLR.

Yuzhuang Xu, Xu Han, Zonghan Yang, Shuo Wang, Qingfu Zhu, Zhiyuan Liu, Weidong Liu, and Wanxiang Che. 2024. Onebit: Towards extremely low-bit large language models. arXiv preprint arXiv:2402.11295.

Zhihang Yuan, Lin Niu, Jiawei Liu, Wenyu Liu, Xinggang Wang, Yuzhang Shang, Guangyu Sun, Qiang Wu, Jiaxiang Wu, and Bingzhe Wu. 2023. Rptq: Reorder-based post-training quantization for large language models. arXiv preprint arXiv:2304.01089.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830.

Chao Zeng, Songwei Liu, Yusheng Xie, Hong Liu, Xiaojian Wang, Miao Wei, Shu Yang, Fangmin Chen, and Xing Mei. 2024. Abq-llm: Arbitrary-bit quantized inference acceleration for large language models. arXiv preprint arXiv:2408.08554.

Yilong Zhao, Chien-Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, and Baris Kasikci. 2024. Atom: Low-bit quantization for efficient and accurate llm serving. Proceedings of Machine Learning and Systems, 6:196–209.

A Binarized Residual Decomposition

Previous work (Zeng et al., 2024) has explored the approach of transforming arbitrary WxYx into \( xy \times W1A1 \) to achieve computational acceleration. Building upon this foundation, we further investigate the use of residual smoothing scaling factors to enhance the quantization performance of LLMs.

Binarized Decomposition Specifically, we initially demonstrate that any matrix represented in the INT format can be decomposed into a combination of multiple INT1 matrices. The advantage of this approach lies in leveraging the efficient computation of W1A1 to replace the original high-bit computations. Obviously, replacing the original matrix multiplication with INT1 computations will result in a multiple increase in computational load, which is particularly pronounced when the original matrix computations involve high bits. However, our low-bit W1A4 quantization setting only increases the computational load to 4 times the original, which is acceptable given the computational speedup brought by the INT1 operators.

Scaling factor balancing Taking symmetric quantization as an example, the dequantization process of the activation values after binary decomposition is denoted as

\[
\mathbf{X}_{\text{DeQuant}} = \mu \mathbf{X}_{\text{INT4}} = 2^3 \cdot \mu \cdot \mathbf{B}_3 + 2^2 \cdot \mu \cdot \mathbf{B}_2 \\
+ 2^1 \cdot \mu \cdot \mathbf{B}_1 + 2^0 \cdot \mu \cdot \mathbf{B}_0.
\]

(10)
where \( \mathbf{X}_{\text{DeQuant}} \) represents the result of dequantization, and \( \mu \) signifies the scaling factor within the quantization parameters. Unlike the conventional 4 bit dequantization process, where only one scaling factor is adjustable, we have the flexibility to independently alter the scaling factors \( \mu_0, \mu_1, \mu_2, \) and \( \mu_3 \) for each of the four INT1 matrices, under the premise that the quantized INT4 matrix can be equivalently replaced by a \( 4 \times 4 \) INT1 matrix. This adjustment aims to further reduce the quantization error of the activation values. Given that the initial 4 bit quantization demonstrates relatively good performance, we leverage these quantization parameters as prior knowledge to calculate the relative error \( \mathbb{E} \) between the dequantized matrix and the original matrix. This error is then smoothly distributed across \( \mu_0, \mu_1, \mu_2, \) and \( \mu_3 \)

\[
\mu_i = \mu_i + \operatorname{Avg}(\frac{\mu_i \mathbf{B}_i}{\mu \mathbf{X}_{\text{INT4}}} \odot \mathbb{E}),
\]
\[
\mathbb{E} = \mathbf{X}_{\text{FP16}} - \mathbf{X}_{\text{DeQuant}}.
\]

We use \( \odot \) to denote element-wise multiplication of matrices and a fraction to represent element-wise division. \( \mathbf{X}_{\text{FP16}} \) stands for the original value matrix of the activations before quantization. The value of \( i \) ranges from 0 to 3. Based on the Equation 11, we minimize the first-order overall quantization error \( \mathbb{E} \) to zero while preserving the distribution of quantized values in the original low-bit setting, thereby further enhancing the performance of the quantized model.

B Quantization and Dequantization after EM Algorithm

Quantization and Dequantization The quantization results obtained by EM Algorithm are akin to those of vector quantization (Frantar et al., 2022; van Baalen et al., 2024), in that it categorizes the original weight elements into \( 2^k \) values, rather than obtaining a low-bit INT quantized representation and quantization parameters as in RTN. In reality, this approach in vector quantization may be suboptimal because the \( 2^k \) values are not equally spaced and cannot be converted into a pure low-bit INT representation, thus precluding the quantized model from utilizing efficient low-bit matrix multiplication to accelerate inference. However, in our weight binarization process, the clustering centers (totaling \( 2^1 = 2 \)) obtained through the EM algorithm consistently maintain equal spacing and can be transformed into an INT1 representation. The transformation process can be expressed as

\[
\hat{\mathbf{W}}_{i,j} = \alpha \cdot \mathbf{Q}(\mathbf{W}_{i,j}) + \beta,
\]
\[
\mathbf{Q}(\mathbf{W}_{i,j}) = \begin{cases}
1, & \mathbf{W}_{i,j} \in C_1 \\
-1, & \mathbf{W}_{i,j} \in C_2
\end{cases}.
\] (12)

Here, The quantization results obtained by EM Algorithm are \( C = \{C_1, C_2\}, \mu = \{\mu_1, \mu_2\} \), from which the dequantization parameters can be calculated as \( \alpha = \frac{\mu_1 - \mu_2}{2}, \beta = \frac{\mu_1 + \mu_2}{2} \). Furthermore, the results of EM Algorithm are transformed into a form that enables efficient INT1 computation.

C Additional Experimental Results

Table 6: Model size comparison of LLAMA family.

<table>
  <tr>
    <th>Models</th>
    <th>FP16</th>
    <th>Ours</th>
  </tr>
  <tr>
    <td>LLAMA-7B</td>
    <td>13.5GB</td>
    <td>2.69GB</td>
  </tr>
  <tr>
    <td>LLAMA-13B</td>
    <td>24.2GB</td>
    <td>4.82GB</td>
  </tr>
  <tr>
    <td>LLAMA-30B</td>
    <td>60.5GB</td>
    <td>12.05GB</td>
  </tr>
  <tr>
    <td>LLAMA-65B</td>
    <td>121.0GB</td>
    <td>24.11GB</td>
  </tr>
</table>

Time required for quantization. Our method quantizes the weight matrices within all linear layers of the full-precision model. The quantization process for the 7B model can be completed in approximately 20 minutes, while the 13B model requires only about 30 minutes.

Results on 13B models. As shown in Table 7 and Table 8, we evaluated the performance of our method and other quantization methods on language generation tasks and zero-shot QA task accuracy using LLAMA1-13B and LLAMA2-13B models. Our findings indicate that, in general, the model performance adheres to the principle that increasing the number of model parameters leads to improved model performance. Furthermore, our method achieved state-of-the-art results across all evaluated metrics.

Results of different outlier channel number settings. In Table 9, we compare the relationship between different numbers of outlier channels and the quantization performance of our method. Since the group size is set to 128, we also use 128 as the unit here. The results demonstrate that preserving a small number of outliers with high precision can ensure overall quantization performance. Furthermore, when the number of outlier channels is
Table 7: Perplexity(\( \downarrow \)) results under the W4A4 and W2A4 settings on LLAMA1-13B and LLAMA2-13B. "FP16" denotes the performance of the original model represented in FLOAT16 format, with the best quantization performance highlighted in bold.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Bits</th>
    <th rowspan="2">Method</th>
    <th colspan="3">Perplexity\(\downarrow\)</th>
    <th rowspan="2">Model</th>
    <th rowspan="2">Bits</th>
    <th rowspan="2">Method</th>
    <th colspan="3">Perplexity\(\downarrow\)</th>
  </tr>
  <tr>
    <th>Wiki.</th>
    <th>PTB</th>
    <th>C4</th>
    <th>Wiki.</th>
    <th>PTB</th>
    <th>C4</th>
  </tr>
  <tr>
    <td rowspan="5">LLAMA1-13B</td>
    <td>FP16</td>
    <td>-</td>
    <td>5.09</td>
    <td>19.23</td>
    <td>6.61</td>
    <td>FP16</td>
    <td>-</td>
    <td>4.88</td>
    <td>28.87</td>
    <td>6.47</td>
  </tr>
  <tr>
    <td>W4A4</td>
    <td>QuaRot</td>
    <td>5.71</td>
    <td>36.10</td>
    <td>7.57</td>
    <td>W4A4</td>
    <td>QuaRot</td>
    <td>5.59</td>
    <td>64.27</td>
    <td>7.84</td>
  </tr>
  <tr>
    <td></td>
    <td>Atom</td>
    <td>5.47</td>
    <td>22.16</td>
    <td>7.04</td>
    <td>LLAMA2-13B</td>
    <td>Atom</td>
    <td>5.26</td>
    <td>32.46</td>
    <td>6.95</td>
  </tr>
  <tr>
    <td>W2A4</td>
    <td>QuaRot</td>
    <td>11.14</td>
    <td>156.30</td>
    <td>20.80</td>
    <td>W2A4</td>
    <td>QuaRot</td>
    <td>17.49</td>
    <td>386.40</td>
    <td>38.88</td>
  </tr>
  <tr>
    <td></td>
    <td>Atom</td>
    <td>11.69</td>
    <td>115.62</td>
    <td>19.55</td>
    <td></td>
    <td>Atom</td>
    <td>11.24</td>
    <td>152.68</td>
    <td>18.15</td>
  </tr>
  <tr>
    <td></td>
    <td>W(1+1)A(1×4)</td>
    <td><b>Ours</b></td>
    <td><b>7.19</b></td>
    <td><b>37.20</b></td>
    <td><b>10.18</b></td>
    <td></td>
    <td>W(1+1)A(1×4)</td>
    <td><b>Ours</b></td>
    <td><b>7.17</b></td>
    <td><b>56.91</b></td>
    <td><b>10.44</b></td>
  </tr>
</table>

Table 8: Zero-shot QA accuracy(\( \uparrow \)) results under the W4A4 and W2A4 settings on LLAMA1-13B and LLAMA2-13B. "FP16" denotes the performance of the original model represented in FLOAT16 format, with the best quantization performance highlighted in bold.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Bits</th>
    <th rowspan="2">Method</th>
    <th colspan="6">Zero-shot Accuracy\(\uparrow\)</th>
  </tr>
  <tr>
    <th>PIQA</th>
    <th>ARC-E</th>
    <th>ARC-C</th>
    <th>BoolQ</th>
    <th>HellaSwag</th>
    <th>Winogrande</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td rowspan="5">LLAMA1-13B</td>
    <td>FP16</td>
    <td>-</td>
    <td>79.05</td>
    <td>59.89</td>
    <td>44.71</td>
    <td>68.47</td>
    <td>76.23</td>
    <td>70.24</td>
    <td>66.43</td>
  </tr>
  <tr>
    <td>W4A4</td>
    <td>QuaRot</td>
    <td>76.61</td>
    <td>53.30</td>
    <td>41.64</td>
    <td>67.09</td>
    <td>73.02</td>
    <td>65.59</td>
    <td>64.24</td>
  </tr>
  <tr>
    <td></td>
    <td>Atom</td>
    <td>77.64</td>
    <td>58.38</td>
    <td>41.81</td>
    <td>68.50</td>
    <td>73.75</td>
    <td>65.98</td>
    <td>64.64</td>
  </tr>
  <tr>
    <td>W2A4</td>
    <td>QuaRot</td>
    <td>64.42</td>
    <td>41.30</td>
    <td>28.75</td>
    <td>63.36</td>
    <td>48.49</td>
    <td>56.67</td>
    <td>50.53</td>
  </tr>
  <tr>
    <td></td>
    <td>Atom</td>
    <td>59.68</td>
    <td>36.62</td>
    <td>29.01</td>
    <td>58.36</td>
    <td>44.84</td>
    <td>52.01</td>
    <td>46.79</td>
  </tr>
  <tr>
    <td></td>
    <td>W(1+1)A(1×4)</td>
    <td><b>Ours</b></td>
    <td><b>72.09</b></td>
    <td><b>48.57</b></td>
    <td><b>34.13</b></td>
    <td><b>62.54</b></td>
    <td><b>62.63</b></td>
    <td><b>64.88</b></td>
    <td><b>57.47</b></td>
  </tr>
  <tr>
    <td rowspan="5">LLAMA2-13B</td>
    <td>FP16</td>
    <td>-</td>
    <td>79.00</td>
    <td>57.95</td>
    <td>44.28</td>
    <td>69.02</td>
    <td>76.58</td>
    <td>69.69</td>
    <td>66.09</td>
  </tr>
  <tr>
    <td>W4A4</td>
    <td>QuaRot</td>
    <td>76.93</td>
    <td>52.10</td>
    <td>40.70</td>
    <td>68.10</td>
    <td>72.70</td>
    <td>62.51</td>
    <td>62.99</td>
  </tr>
  <tr>
    <td></td>
    <td>Atom</td>
    <td>77.37</td>
    <td>56.73</td>
    <td>42.32</td>
    <td>67.62</td>
    <td>74.07</td>
    <td>68.27</td>
    <td>65.46</td>
  </tr>
  <tr>
    <td>W2A4</td>
    <td>QuaRot</td>
    <td>59.09</td>
    <td>34.60</td>
    <td>24.23</td>
    <td>62.11</td>
    <td>35.03</td>
    <td>51.38</td>
    <td>44.41</td>
  </tr>
  <tr>
    <td></td>
    <td>Atom</td>
    <td>61.15</td>
    <td>40.36</td>
    <td>29.52</td>
    <td>61.56</td>
    <td>45.62</td>
    <td>51.22</td>
    <td>48.24</td>
  </tr>
  <tr>
    <td></td>
    <td>W(1+1)A(1×4)</td>
    <td><b>Ours</b></td>
    <td><b>71.98</b></td>
    <td><b>49.92</b></td>
    <td><b>36.26</b></td>
    <td><b>65.90</b></td>
    <td><b>60.52</b></td>
    <td><b>61.80</b></td>
    <td><b>57.73</b></td>
  </tr>
</table>

increased, the model performance exhibits a nearly linear upward trend, with only a modest overall improvement. Therefore, we adopt 128 outlier channels as our baseline setting.

Comparison of different kernels. In Figure 4, we comprehensively evaluate the performance of the W(1+1)A(1×4) kernel and the INT8, INT4 kernels from CUTLASS, based on the matrix multiplication sizes that may occur in the LLAMA model. Since our method incorporates a small amount of INT8 mixed-precision quantization, for the handling of outliers, we separately measure the computational efficiency of outliers and normal values. Subsequently, we derive the overall computational efficiency by considering the proportion of these two components.

Model Size. We present in Table 6 the theoretical compression effectiveness of our method on the LLAMA family. In our calculation of the model size, we have included both the quantization parameters and the additional storage incurred by fine-grained grouping, which results in our findings being slightly larger than those reported in BiLLM (Huang et al., 2024a). The binarization of weights significantly reduces the storage size of quantized LLMs and the GPU memory and bandwidth requirements during inference. Across LLAMA models of different sizes, our method achieves a compression ratio of over 5×.

D AI Assistants in Research or Writing

We use a local LLama3.3 model to polish the draft for checking grammar and improving expression. Research ideas, experiment design and discussion contents are all original by the authors.
Table 9: The impact of different outlier channel number settings of the quantized model on the perplexity (↓) and the zero-shot QA accuracy(↑). "FP16" denotes the performance of the original model represented in FLOAT16 format.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Ch.</th>
    <th colspan="3">Wiki.↓</th>
    <th colspan="3">C4↓</th>
    <th colspan="3">PIQA↑</th>
    <th colspan="3">ARC-E↑</th>
    <th colspan="3">ARC-C↑</th>
    <th colspan="3">BoolQ↑</th>
    <th colspan="3">Hella.↑</th>
    <th colspan="3">Wino.↑</th>
    <th colspan="1">Avg.↑</th>
  </tr>
  <tr>
    <th>PTB↓</th>
    <th>C4↓</th>
    <th>PIQA↑</th>
    <th>ARC-E↑</th>
    <th>ARC-C↑</th>
    <th>BoolQ↑</th>
    <th>Hella.↑</th>
    <th>Wino.↑</th>
    <th>Avg.↑</th>
  </tr>
  <tr>
    <td rowspan="7">LLAMA1-7B</td>
    <td>FP16</td>
    <td>5.68</td>
    <td>27.34</td>
    <td>7.08</td>
    <td>77.37</td>
    <td>52.48</td>
    <td>41.38</td>
    <td>73.06</td>
    <td>72.99</td>
    <td>67.01</td>
    <td>64.05</td>
  </tr>
  <tr>
    <td>0</td>
    <td>471.19</td>
    <td>1025.28</td>
    <td>228.17</td>
    <td>53.59</td>
    <td>28.75</td>
    <td>24.57</td>
    <td>50.73</td>
    <td>28.13</td>
    <td>50.51</td>
    <td>39.38</td>
  </tr>
  <tr>
    <td>128</td>
    <td>8.58</td>
    <td>76.09</td>
    <td>12.27</td>
    <td>68.88</td>
    <td>45.03</td>
    <td>30.89</td>
    <td>69.63</td>
    <td>55.41</td>
    <td>59.35</td>
    <td>54.87</td>
  </tr>
  <tr>
    <td>256</td>
    <td>8.20</td>
    <td>65.97</td>
    <td>11.70</td>
    <td>69.75</td>
    <td>45.33</td>
    <td>32.42</td>
    <td>65.87</td>
    <td>56.31</td>
    <td>57.85</td>
    <td>54.59</td>
  </tr>
  <tr>
    <td>512</td>
    <td>7.80</td>
    <td>57.44</td>
    <td>10.90</td>
    <td>71.27</td>
    <td>47.94</td>
    <td>34.22</td>
    <td>65.57</td>
    <td>58.46</td>
    <td>59.19</td>
    <td>56.11</td>
  </tr>
  <tr>
    <td>768</td>
    <td>7.52</td>
    <td>52.06</td>
    <td>10.44</td>
    <td>71.38</td>
    <td>47.31</td>
    <td>34.04</td>
    <td>66.12</td>
    <td>60.09</td>
    <td>61.17</td>
    <td>56.69</td>
  </tr>
  <tr>
    <td>1024</td>
    <td>7.26</td>
    <td>50.42</td>
    <td>9.95</td>
    <td>72.14</td>
    <td>47.01</td>
    <td>34.39</td>
    <td>69.30</td>
    <td>61.16</td>
    <td>61.01</td>
    <td>57.50</td>
  </tr>
  <tr>
    <td rowspan="4">LLAMA2-7B</td>
    <td>FP16</td>
    <td>5.47</td>
    <td>22.51</td>
    <td>6.97</td>
    <td>76.93</td>
    <td>53.58</td>
    <td>40.53</td>
    <td>71.07</td>
    <td>72.96</td>
    <td>67.17</td>
    <td>63.71</td>
  </tr>
  <tr>
    <td>128</td>
    <td>8.89</td>
    <td>69.46</td>
    <td>12.74</td>
    <td>68.72</td>
    <td>46.13</td>
    <td>30.55</td>
    <td>66.12</td>
    <td>55.76</td>
    <td>58.01</td>
    <td>54.22</td>
  </tr>
  <tr>
    <td>256</td>
    <td>8.52</td>
    <td>61.01</td>
    <td>12.16</td>
    <td>69.97</td>
    <td>47.64</td>
    <td>31.57</td>
    <td>68.50</td>
    <td>56.22</td>
    <td>59.19</td>
    <td>55.51</td>
  </tr>
  <tr>
    <td>512</td>
    <td>8.00</td>
    <td>56.77</td>
    <td>11.43</td>
    <td>69.80</td>
    <td>46.97</td>
    <td>31.66</td>
    <td>67.83</td>
    <td>57.55</td>
    <td>60.77</td>
    <td>55.76</td>
  </tr>
  <tr>
    <td rowspan="3">Vicuna-v1.5-7B</td>
    <td>FP16</td>
    <td>6.78</td>
    <td>26.78</td>
    <td>8.55</td>
    <td>77.80</td>
    <td>56.06</td>
    <td>39.93</td>
    <td>75.69</td>
    <td>71.06</td>
    <td>67.80</td>
    <td>64.72</td>
  </tr>
  <tr>
    <td>128</td>
    <td>9.51</td>
    <td>45.61</td>
    <td>13.35</td>
    <td>69.97</td>
    <td>50.00</td>
    <td>33.45</td>
    <td>71.96</td>
    <td>57.81</td>
    <td>59.43</td>
    <td>57.10</td>
  </tr>
  <tr>
    <td>256</td>
    <td>9.28</td>
    <td>44.01</td>
    <td>12.94</td>
    <td>70.35</td>
    <td>51.05</td>
    <td>34.04</td>
    <td>72.97</td>
    <td>58.42</td>
    <td>61.40</td>
    <td>58.04</td>
  </tr>
  <tr>
    <td></td>
    <td>512</td>
    <td>8.88</td>
    <td>41.49</td>
    <td>12.42</td>
    <td>71.87</td>
    <td>50.76</td>
    <td>34.22</td>
    <td>73.79</td>
    <td>58.80</td>
    <td>64.09</td>
    <td>58.92</td>
  </tr>
</table>

![Three bar and line charts comparing computational efficiency between W(1+1)A(1×4) kernel and INT4, INT8 kernels of CUTLASS for matrix multiplication with varying input lengths](page_1012_1342_1022_246.png)

Figure 4: A comparison of the computational efficiency between the W(1+1)A(1×4) kernel and the INT4, INT8 kernels of CUTLASS for matrix multiplication with varying input lengths is conducted on the A6000.

Reviewer paper summaries: This paper proposes a post-training quantization framework for LLMs, achieving 1-bit weight and 1×4-bit activation quantization (W(1+1)A(1×4)). The method introduces Hessian-aware fine-grained grouping for weight quantization and decomposes 4-bit activations into four 1-bit Boolean variables, enabling efficient binary operations during inference. Through EM-based quantization parameter optimization and channel grouping to mitigate outliers, the authors maintain model performance while significantly reducing model size and computational costs. 

Extensive experiments on LLaMA, LLaMA2, and Vicuna models demonstrate state-of-the-art perplexity and zero-shot accuracy in low-bit settings, outperforming existing methods like QuaRot and Atom.
Summary of strengths: 1. **Innovative Quantization Design**: The W(1+1)A(1×4) configuration is interesting, particularly the decomposition of 4-bit activations into Boolean variables and fine-grained weight grouping. This effectively balances quantization error reduction and computational efficiency.

2. **Practical Efficiency**: The framework leverages bitwise operations (e.g., XOR/AND and popc) to accelerate inner-loop computations, achieving model compression and hardware-friendly implementation.

3. **Comprehensive Evaluation**: Extensive experiments across models (7B–65B), tasks (language generation, zero-shot QA), and datasets (WikiText2, PIQA) validate the method’s superiority in low-bit settings, even approaching FP16 performance in some cases.
Summary of weaknesses: 1. **Complexity and Overhead**: EM-based optimization and Hessian-aware grouping introduce algorithmic complexity. While claimed to be lightweight, detailed runtime/memory cost analysis during calibration/deployment is missing, raising practicality concerns.

2. **Limited Component Focus**: Targeting fully connected layers (90% of computation) while ignoring critical components (e.g., attention mechanisms, normalization layers) may impact end-to-end performance in full-model deployment.
Comments, suggestions and typos: 1. **Training Overhead**: Hessian matrix computation requires validation data and gradients. How does this scale with model size (e.g., 65B parameters)? How does calibration time compare to RTN-based methods?

2. **Outlier Handling**: The paper uses 128 INT8 channels for outliers. How sensitive is performance to this hyperparameter? Could adaptive outlier detection improve results?

3. **Limited Comparison**: Experiments focus on W4A4 and W2A4 baselines but ignore recent advances (e.g., mixed-precision quantization, dynamic bit allocation), limiting competitiveness assessment in broader contexts.
Overall assessment scores: 3.5
Metareview score: 3
Metareview: The paper proposes a PTQ framework for LLMs targeting fully binarized weight and activation representations. It introduces a W(1+1)A(1x4) config, where weights are binarized with an extra bit for fine-grained grouping, and activations are decomposed from INT4 to 4xINT1 for efficient binary ops. The method leverages a Hessian-aware EM-based optimization for weight quant and smooths activation scales to reduce error.
Summary of reasons to publish: The proposed method achieves strong empirical results across multiple tasks and models, surpassing SOTA under W2A4. The method is simple, hardware-friendly, and well-supported by ablations, making it a practical advancement toward extremely low-bit LLMs.