STBLLM: Breaking the 1-Bit Barrier with Structured Binary LLMs

## Abstract

Abstract In this paper, we present the first structural binarization method for LLM compression to less than 1-bit precision. Although LLMs have achieved remarkable performance, their memory-bound nature during the inference stage hinders the adoption of resource-constrained devices. Reducing weights to 1-bit precision through binarization substantially enhances computational efficiency. We observe that some weights in binarized LLMs can be randomly flipped without significant performance degradation, suggesting the potential for further compression. To exploit this, our STBLLM employs an N:M sparsity technique to achieve structural binarization of the weights. Specifically, we introduce a novel Standardized Importance (SI) metric, which considers weight magnitude and input feature norm to more accurately assess weight significance. Then, we propose a layer-wise approach, allowing different layers of the LLM to be sparsified with varying N:M ratios, thereby balancing compression and accuracy. Furthermore, we implement a fine-grained grouping strategy for less important weights, applying distinct quantization schemes to sparse, intermediate, and dense regions. Finally, we design a specialized CUDA kernel to support structural binarization. We conduct extensive experiments on LLaMA-1/2/3, OPT family, and Mistral to evaluate the effectiveness of STBLLM. The results demonstrate that our approach performs better than other compressed binarization LLM methods while significantly reducing memory requirements.

## 1 Introduction

The advent of large language models (LLMs), such as (Zhang et al., ; Touvron et al., ; Brown et al., ), has revolutionized the field of natural language processing (NLP) (Wei et al., ). These powerful models exhibit remarkable performance, surpassing human capabilities in certain domains (Wei et al., ; Bubeck et al., ). However, the immense scale and complexity of LLMs present significant challenges in terms of memory requirements, hindering their widespread deployment, especially in resource-constrained environments.
To address this issue, model compression techniques, such as quantization (Frantar et al., ; Lin et al., ; Dong et al., ), pruning (Meng et al., ), distillation (Gu et al., ), and low-rank decomposition (Ashkboos et al., ), have gained increasing attention in reducing the computational footprint while preserving their performance. One promising approach is network binarization, the most aggressive quantization method. Binarization quantizes original floating-point weights with binary values ($-1$ or $+1$), significantly reduces memory storage.

Figure: Figure 1: The impact of random flipping non-salient binarized weights on accuracy in a 1-Bit LLaMA-2-7B. The x-axis represents the percentage of binarized weights flipped from -1 to 1 or vice versa. As the ratio increases, the accuracy does not decline significantly, indicating redundancy in the 1-bit representation.
Refer to caption: extracted/5906344/pic/figure1.png

Pioneering binarization methods (Rastegari et al., ; Liu et al., ) present customized binary structures and training paradigms for binarized neural networks (BNNs) in vision tasks. Building upon these foundational approaches, subsequent methods (Wang et al., ; ; ; Liu et al., ; Li & Ren, ; Munagala et al., ) have advanced the field by integrating sparse kernel techniques (Wang et al., ; ) and pruning methodologies (Wang et al., ; Li & Ren, ; Munagala et al., ). For LLMs, inspired by the success of 4-bit and 8-bit quantization methods, some studies (Huang et al., ; Xu et al., ; Shang et al., ) continue to explore ultra-low-bit or even 1-bit precision. For example, the post-training method PB-LLM (Shang et al., ) partially binarizes LLMs with an optimal scaling factor strategy, preserving a small subset of the higher bit-precision weights. BiLLM (Huang et al., ) proposes a residual approximation strategy to improve 1-bit LLMs. While these methods represent the most aggressive quantization approaches, it is crucial to consider that popular floating-point LLMs already contain model sizes ranging from 7 billion to 140 billion parameters. As a result, 1-bit LLMs still need to be further accelerated and optimized for many resource-constrained devices and real-time scenarios. This naturally raises a key question: *Is there any compression method with less than 1-bit weight representation that can further push the quantization of LLMs?*

For this question, there are two key observations: ① Not all weights contribute equally to the performance of 1-bit LLMs. As shown in Figure 2, performing random weight flipping for non-salient weights results in only a minimal performance drop (For more details, refer to Appendix B). This finding indicates that even in highly quantized 1-bit LLMs, a subset of redundant weights exists that can be compressed without impacting the performance. It suggests the potential for further compression by selectively encoding the most significant weights while discarding or compressing the less important ones. ② Structured sparsity techniques, such as N:M sparsity methods (Hubara et al., ; Zhang et al., ; Zhou et al., ), leverage the inherent structure and patterns in the weight distribution, allowing for more efficient compression. These N:M sparsity methods have good hardware-accelerated support in recent LLM pruning models (Frantar & Alistarh, ; Sun et al., ; Dong et al., ), enabling efficient deployment on NVIDIA Ampere architecture (Nvidia, ). However, traditional binarization techniques (Rastegari et al., ) often treat weights as independent entities, failing to exploit the inherent structure and patterns in the weight matrices. These observations encourage us to explore N:M sparsity tailored specifically for 1-bit LLMs to achieve further speedups and compression gains.

Based on these observations, we develop our STBLLM approach, STructured Binarization for LLMs to achieve extreme compression while mitigating performance degradation. Our workflow applies the metric-based sparsity and performs the adaptive N:M binarization. Specifically, to measure the importance of weights, we introduce a Standardized Importance (SI) metric that addresses the issues of extreme weight values and computationally expensive Hessian-based methods used in prior work. We then propose an adaptive layer-wise structured binarization approach, where different layers of the LLM can be sparsified with varying N:M ratios to balance compression and accuracy. We employ a residual approximation technique (Huang et al., ) for the salient parameters to preserve the critical information. For the non-salient parameters, we utilize a fine-grained grouping strategy based on a trisection search algorithm to find optimal splitting points $p^{*}$ and apply different quantization schemes to the sparse, intermediate, and dense regions as presented in Figure 3(c). By tailoring these structured representations specifically for 1-bit LLMs, we unlock a new avenue for model compression and optimization, enabling more widespread deployment of these powerful LLMs in resource-constrained environments.

To validate the effectiveness of STBLLM, we conduct extensive experiments on various LLMs, including the LLaMA-1/2/3 (Touvron et al., ; ), OPT (Zhang et al., ) and Mistral (Jiang et al., ). As presented in Figure 2, our STBLLM achieves a better trade-off between performance and bit-width. STBLLM with 0.8 bit can achieve lower perplexity than BiLLM with 1.1 bit. STBLLM achieves a perplexity of 31.72 at just 0.55 bits per weight, compared to 688.73 for BiLLM - an over 20$\times$ gain. Even at 65B parameters, our 0.55-bit STBLLM outperforms BiLLM’s 0.7-bit and PB-LLM’s 1.7-bit versions. STBLLM retains significantly higher accuracy for zero-shot benchmarks than BiLLM under 4:8 and 6:8 structured binarization settings across 13B and 30B LLaMA. For example, on LLaMA-1-30B, our 0.55-bit STBLLM achieves 51.78% average accuracy versus just 43.72% for BiLLM. The contribution of our work is as follows:

- •
We introduce STBLLM, a novel structural binarization framework that compresses Large Language Models (LLMs) to less than 1-bit precision, enabling significant memory and computational savings while preserving model performance.
- •
STBLLM employs an N:M binary weight kernel approach, where we perform structural binarization of the weights using efficient gradient-free metrics to determine weight importance, channel rearrangement to preserve salient weights, and adaptive layer mixed-structure binarization for better accuracy-efficiency trade-offs.
- •
We implement a specialized CUDA kernel for structural binarization, leveraging NVIDIA’s Ampere GPU sparse tensor cores, achieving a 17.85x speedup over ABQ-LLM’s 2-bit implementation.
- •
Extensive experiments on various LLMs, including the LLaMA-1/2/3 and OPT, demonstrate STBLLM’s superior performance compared to other compressed binarization methods.

## 2 Related Work

Figure: Figure 3: (a) PTQ framework in Structured Binarized LLM (STBLLM). We apply structured binarization to all of the weights. (b) Structured Binarized Weight Computation Procedure. We first perform N:M structure pruning to pre-trained weight (here N=2, M=4), then perform binarization by assign weight to salient and non-salient one. (c) Trisection partition for Symmetric Gaussian Distribution of Non-salient Weight. (d) Illustration of Weight Standardization on LLaMA-2-7B.
Refer to caption: extracted/5906344/pic/basic_block.png

Quantization and Binarization. Quantization reduces full-precision parameters to lower bits, thereby decreasing storage and computation requirements. Recent research has effectively applied Quantization-Aware Training (QAT) and Post-Training Quantization (PTQ) to LLMs. QAT (Liu et al., ; Du et al., ; Chen et al., ) incorporates quantization during training, allowing to learn better representations for low-bit weights. However, due to the massive parameters, retraining is too costly and inefficient for LLMs. In contrast, PTQ (Frantar et al., ; Chee et al., ; Lin et al., ; Lee et al., ; Dettmers et al., ) can directly quantize pre-trained models without additional training.
GPTQ (Frantar et al., ) and QuIP (Chee et al., ) minimized LLM block quantization errors through second-order error compensation. Other approaches (Lin et al., ; Lee et al., ; Dettmers et al., ) focus on prioritizing salient weights to maintain their information representation capacity. Binarization, which constrains quantized parameters to a 1-bit representation, is the most extreme quantization method and has proven effective for vision tasks, such as XNOR-Net (Rastegari et al., ) and Bi-Real Net (Liu et al., ). To further compress binary neural networks, sparse kernel techniques (Wang et al., ; ; ; Liu et al., ) are introduced to reduce the redundancy in binary neural networks. For LLM binarization, BitNet (Wang et al., ) trained 1-bit LLM from scratch. OneBit (Xu et al., ) employ the QAT paradigm for 1-bit LLM while BiLLM (Huang et al., ) employ the PTQ paradigm with residual approximation technique. In this paper, we further reduce weights below 1-bit by identifying and removing the redundant parameters.

Sparsity Methods for LLM. Pruning removes less important parameters from a neural network to reduce its size and improve efficiency.
For LLMs, Pruning can be divided to structured pruning (Ma et al., ; Ashkboos et al., ; Xia et al., ; An et al., ), semi-structured pruning (Frantar & Alistarh, ; Sun et al., ; Zhang et al., ) and unstructured pruning (Frantar & Alistarh, ; Sun et al., ; Dong et al., ). Structured pruning methods, including LLM-Pruner (Ma et al., ) and Sheared LLaMA (Xia et al., ), aim to simplify LLM by removing specific components such as heads, layers, and dimensions. Although these techniques enhance model efficiency, they often result in significant performance degradation and require extensive retraining to recover lost capabilities. In contrast, unstructured pruning methods (Frantar & Alistarh, ; Sun et al., ) remove individual weights based on their significance within the model. However, this approach leads to irregular sparsity patterns that do not effectively leverage hardware acceleration. Semi-structured pruning offers a more balanced approach to model optimization. Methods such as SparseGPT (Frantar & Alistarh, ) and Wanda (Sun et al., ) exemplify this strategy by maintaining regular, hardware-friendly sparsity patterns, such as N:M sparsity. This approach combines the fine-grained control characteristic of unstructured pruning with the operational efficiency associated with structured pruning.

Synergy of Pruning and Quantization. The complementary nature of pruning and quantization has been extensively explored in the literature, where pruning reduces the number of parameters in a neural network, and quantization focuses on the precision of those parameters. Deep Compression (Han et al., ) integrates pruning, trained quantization, and Huffman coding into a unified compression pipeline to significantly reduce the storage requirements of deep neural networks (DNNs). Subsequent works have developed in-parallel pruning-quantization methods (Tung & Mori, ; Yang et al., ; Hu et al., ) to optimize compression allocation, such as unstructured pruning sparsity and quantization strategies. For extreme cases like binarization, several approaches (Munagala et al., ; Li & Ren, ; Wang et al., ) combine pruning and compression to achieve high levels of compression and speedup. Specifically, STQ-Nets (Munagala et al., ) extend convolutional neural network (CNN) binarization by incorporating structured pruning, BNN Pruning (Li & Ren, ) utilizes weight flipping frequency for further pruning of binary neural networks (BNNs), and BAP (Wang et al., ) introduces binary augmented sparse convolution to attain 98% sparsity. However, these methods often necessitate a fine-tuning process, which is impractical for LLMs.

Figure: Algorithm 1 Framework of STBLLM: Details of each function are shown in Algorithm 2

## 3 Methodology

In this section, we introduce our STBLLM framework, as depicted in Figure 3. We employ structured binarization for all weights within the Feed-forward Network (FFN) and Multi-head Self-attention (MHSA) modules. Specifically, we introduce the concept of Standardized Importance (SI) to evaluate the saliency of each weight under N:M sparsity constraints (refer to the left part of Figure 3(b)). We leverage the Hessian matrix to distinguish between salient and non-salient weights for the binarization process. Salient weights are handled using residual approximation, following the methodology outlined in BiLLM (Huang et al., ). For non-salient weights, we propose a Non-salient Aware Quantization technique, which further divides these weights into Dense, Intermediate, and Sparse regions (as shown in the right part of Figure 3(c)). To optimally partition the non-salient weights into three distinct regions, we utilize a trisection search strategy to determine the appropriate $p^{*}_{1}$ and $p^{*}_{2}$ values. In the subsequent update step, we apply block-wise error compensation (Frantar & Alistarh, ; Frantar et al., ) to preserve performance following post-training quantization (PTQ). Algorithm 1 provides a comprehensive overview of the STBLLM process, with detailed implementation steps in Appendix A.

### 3.1 Preliminaries

Binarization. Binarized compression seeks to quantize floating-point (FP) weights, represented as $\mathbf{W}_{FP}$, into 1-bit values (i.e., $\pm$1). During forward propagation, the sign function is used to binarize the original parameter tensor:

$$ $\displaystyle\mathbf{B}$ $\displaystyle:=\alpha\cdot\text{sign}(\mathbf{W}_{FP}),$ (1) $\displaystyle\operatorname{sign}(w)$ $\displaystyle:=\begin{cases}1&\text{if $x\geq 0$},\\ -1&\text{others},\end{cases}$ (2) $$

where $\mathbf{W}_{FP}\in\mathbb{R}^{n\times m}$ is the 32-bit floating-point weight, and $\mathbf{B}\in\mathbb{R}^{n\times m}$ is the binarized output, and $\alpha:=\frac{||\mathbf{W}||_{l_{1}}}{m}$. The parameter $n$ and $m$ represent the size of the weight matrix. The scaling factor $\alpha\in\mathbb{R}^{n}$ is applied in a channel-wise manner (Rastegari et al., ).

N:M Sparsity. Inspired by the experiments shown in Figure 2, we observe the binarized the redundancy in LLMs. By applying the N:M binarization for LLMs, we can achieve an extreme compression ratio of less than 1 bit. Specifically, we introduce an innovative N:M sparsity technique that encodes N consecutive non-zero elements in the weight matrix using a single M-bit representation. Although this approach can accelerate computations, it may lead to performance degradation. To alleviate this problem, we propose several techniques from different perspectives: ① Importance Measurement. Previous methods (Frantar et al., ; Chen et al., ; Huang et al., ) utilize Hessian-based methods to measure the importance, but these methods can be computationally expensive and may not capture the true importance of parameters in LLMs. ② Layer-wise Assignment. Previous PTP methods (Frantar & Alistarh, ; Zhang et al., ) utilize the uniform sparsity ratio among different layers. However, recently, evidence (Yin et al., ) shows that not all layers have the same redundant level thus non-uniform sampling can help retain the performance. ③ Hierarchical Quantization. Previous PTQ methods for LLM like AWQ (Lin et al., ), OWQ (Lee et al., ) and BiLLM (Huang et al., ) split the weights into salient and non-salient parameters using the magnitude of activation or Hessian matrix. They mainly focus on salient weights, as most researchers believe they contribute to the final performance. However, the non-salient parameters also play an essential role in quantization.

### 3.2 Standardized Importance Metric

Many previous works, such as DB-LLM (Chen et al., ), SparseGPT (Frantar & Alistarh, ), GPTQ (Frantar et al., ), and BiLLM (Huang et al., ), utilize the Hessian metric to measure the importance of weights. However, we observe that extreme values in the weights significantly impact Hessian computation (See Appendix D). To address this issue, we present a Standardized Importance (SI) metric. The computation of SI does not involve the second-order information of the weights, which can be computationally expensive for LLMs. Specifically, we employ standardization to mitigate the issue of extreme values in weights by transforming the weights to have a mean of zero and a standard deviation of one. This process ensures that all weights are on a similar scale, reducing the disproportionate influence of extreme values on the Hessian matrix. For a linear layer with weight $\mathbf{W}\in\mathbb{R}^{n\times m}$, which takes in input activation $\mathbf{X}\in\mathbb{R}^{r\times d}$, where $r$ is the batch size and $d=m$ is the input dimension. We propose to evaluate the importance of each weight by the product of its magnitude and the corresponding input feature norm. The score for the current weight $\mathbf{W}_{i,j}$ is defined as:

$$ $\displaystyle\mathbf{S}_{i,j}=\sigma(\mu(|\mathbf{W}_{i,j}|))\cdot||\mathbf{X} _{:,j}||_{2},\quad\sigma(\hat{w})=\frac{w-\mu_{\mathbf{W}}}{\sigma_{\mathbf{W} }},\quad\mu(|\mathbf{W}_{i,j}|)=\frac{|\mathbf{W}_{i,j}|}{\sum_{j}|\mathbf{W}_ {i,j}|}+\frac{|\mathbf{W}_{i,j}|}{\sum_{i}|\mathbf{W}_{i,j}|},$ (3) $$

where $\sigma(\cdot)$ is a normalization function that standardizes the weight magnitude $\mu(|\mathbf{W}_{i,j}|)$ using the mean $\mu_{\mathbf{W}}$ and standard deviation $\sigma_{\mathbf{W}}$ of all weights in the layer. The weight magnitude $\mu(|\mathbf{W}_{i,j}|)$ is computed as the sum of the L1-normalized magnitude across the input dimension $j$ and the output dimension $i$. The input feature norm $||\mathbf{X}_{:,j}||_{2}$ is calculated as the L2 norm of the $j$-th column input activation $\mathbf{X}$. By multiplying the standardized weight magnitude $\sigma(\mu(|\mathbf{W}_{i,j}|))$ with the input feature norm $||\mathbf{X}_{:,j}||_{2}$, the importance score $\mathbf{S}_{i,j}$ takes into account both the significance of the weight itself and the activation level of the associated input feature. To prune the linear layer, we rank all the weights based on their importance scores $\mathbf{S}_{i,j}$ and remove a specified percentage of the weights with the lowest scores. This pruning strategy aims to preserve the most significant weights contributing to the layer’s output while eliminating less important weights to reduce the model’s size and computational complexity.

### 3.3 Adaptive Layer-wise Binarization

N:M Binary Weight Vector. To achieve compression beyond standard binarization, we propose an $N:M$ sparsity approach, where M binary values are represented by N values (N $<$ M). This allows for further compression while preserving the salient information in the weight tensors. Specifically, we employ the mixed N:8 sparsity configuration following DominoSearch (Sun et al., ).

Layer-wise N:M Assignment. To achieve better accuracy-efficiency trade-offs, we introduce adaptive layer-wise structured binarization, where different layers of the LLM can be sparsified with different N:M ratios. (For example, with a target ratio of 4:8, layers can have ratios like 3:8, 4:8, and 5:8 while maintaining the overall 4:8 ratio.) This flexibility allows for more aggressive compression in less important layers while preserving higher precision in crucial layers.
The layer-wise N:M ratios are assigned based on the relative importance of each layer, measured by the L2 norm of its weight parameters. Let $\omega_{i}$ and $\omega_{\text{total}}$ be the L2 norm of layer $i$ and the sum across all layers, respectively. The relative importance $\alpha_{i}$ of layer $i$ is $\alpha_{i}=\frac{\omega_{i}}{\omega_{\text{total}}}$. The N:M ratio for layer $i$ is $\frac{N_{i}}{M_{i}}=\alpha_{i}+(1-\alpha_{i})\cdot R_{\text{target}}$, where $R_{\text{target}}$ is the target overall compression ratio. More important layers have higher N:M ratios (less sparsification), approaching 1:1 for the most important ones. Less important layers have lower N:M ratios, approaching $R_{\text{target}}$ for the least important ones. This ensures the overall compression ratio meets $R_{\text{target}}$.

### 3.4 Non-salient Aware Quantization

Based on the observations that a small fraction of salient weights is critical to the LLM quantization (Lin et al., ; Shao et al., ), we split the weights into the salient and non-salient parts and then apply a higher bit for salient one and lower-bit for non-salient one, as:

Salient Part: In our cases, for salient weight, we apply residual approximation (Huang et al., ), which is composed of residual approximation weight, as follows:

$$ $\begin{cases}\alpha_{o}^{*},\mathbf{B}_{o}^{*}=\arg\min_{\alpha_{o},\mathbf{B} _{o}}\|\mathbf{W}-\alpha_{o}\mathbf{B}_{o}\|^{2},\\ \alpha_{r}^{*},\mathbf{B}_{r}^{*}=\arg\min_{\alpha_{r},\mathbf{B}_{r}}\|( \mathbf{W}-\alpha_{o}^{*}\mathbf{B}_{o}^{*})-\alpha_{r}\mathbf{B}_{r}\|^{2}, \end{cases}$ (4) $$

where $\mathbf{B}_{o}$ denotes the original binary tensor, and $\mathbf{B}_{r}$ represent the residual binarized matrix as the compensation. The final approximation of $\mathbf{W}$ is $\mathbf{W}\approx\alpha_{o}^{*}\mathbf{B}_{o}^{*}+\alpha_{r}^{*}\mathbf{B}_{r}
^{*}.$

Non-Salient Part:
For the non-salient part (which is also symmetric Gaussian distribution), we find that significant information is retained in the non-salient part. To make the trade-off with bit and performance, we utilize a fine-grained grouping strategy called the Trisection search algorithm (See Algorithm 2), whose aim is to find the optimal two break-point $p^{*}_{1},p^{*}_{2}$. With these two break-points, we can segment the symmetric Gaussian distribution into three groups, which is sparse $R_{s}[-m,-p^{*}_{2}]\cup[p^{*}_{2},m]$, intermediate $R_{i}[-p^{*}_{2},-p^{*}_{1}]\cup[p^{*}_{1},p^{*}_{2}]$, and dense region $R_{d}[-p^{*}_{1},p^{*}_{1}]$. Then, we derive the quantization error:

$$ $\displaystyle\theta^{2}_{p^{*}_{1},p^{*}_{2}}=||\mathbf{W}_{s}-\alpha_{s} \mathbf{B}_{s}||^{2}+||\mathbf{W}_{i}-\alpha_{i}\mathbf{B}_{i}||^{2}+||\mathbf {W}_{d}-\alpha_{d}\mathbf{B}_{d}||^{2},$ (5) $\displaystyle\alpha_{s}=\frac{1}{n_{s}}||\mathbf{W}_{s}||_{l1},\ \ \alpha_{i}= \frac{1}{n_{i}}||\mathbf{W}_{i}||_{l1},\ \ \alpha_{d}=\frac{1}{n_{d}}||\mathbf {W}_{d}||_{l1}$ (6) $$

where $\mathbf{W}_{s}$, $\mathbf{W}_{i}$, $\mathbf{W}_{d}$ are the sums of absolute weight values in the sparse, intermediate, and dense regions. $\mathbf{B}_{s}$, $\mathbf{B}_{i}$, $\mathbf{B}_{d}$ are the binarized weights for those regions.
These three regions are binarized separately. This method introduces an additional 2 bits for group identification, which constitutes a minor portion of the overall bit count, while the majority of computing parameters remain at 1 bit.

Average Bits. In STBLLM, we introduce extra bits while pruning the redundant or less important weights. The overhead of weight parameters is $N_{\text{param}}=2\times r_{\text{salient}}+1\times(1-r_{\text{salient}})$. The additional hardware overhead is $N_{\text{storing}}=2+\frac{1}{b_{\text{size}}}$, where $r_{salient}$ denotes the proportion of salient weights and $b_{\text{size}}$ denotes the block size in OBC compensation, with 2 bits allocated for marking the division of non-salient weights. Under N:M binarization settings, where N and M are positive integers with N ¡ M, we prune the model weights by retaining only a fraction (N/M) of the original weights. Consequently, the number of parameters in the pruned STBLLM model is $N_{\text{stbllm}}=N_{\text{param}}\times\frac{N}{M}$. This N:M binarization method allows for a significant reduction in model size.

**Table 1: Average bit results from structural searching and residual binarization of OPT, LLaMA-1, and LLaMA-2 families. *OPT-66B, LLaMA-1-65B and LLaMA-2-70B.**
| Model | BiLLM | BiLLM-4:8 | BiLLM-5:8 | BiLLM-6:8 |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | 7B | 13B | 30B | 65-70B* | 7B | 13B | 30B | 65-70B* | 7B | 13B | 30B | 65-70B* | 7B | 13B | 30B | 65-70B* |
| OPT | 1.10 | 1.12 | 1.12 | 1.13 | 0.55 | 0.56 | 0.56 | 0.56 | 0.69 | 0.70 | 0.70 | 0.71 | 0.83 | 0.84 | 0.84 | 0.85 |
| LLaMA-1 | 1.09 | 1.09 | 1.10 | 1.10 | 0.54 | 0.54 | 0.55 | 0.55 | 0.68 | 0.68 | 0.69 | 0.69 | 0.82 | 0.82 | 0.83 | 0.83 |
| LLaMA-2 | 1.07 | 1.08 | N/A | 1.09 | 0.53 | 0.54 | N/A | 0.54 | 0.67 | 0.67 | N/A | 0.68 | 0.80 | 0.81 | N/A | 0.82 |

**Table 2: Perplexity comparison of PB-LLM and BiLLM on the LLaMA model family. The columns represent the perplexity results on the Wikitext2 for different model sizes. The average bit-width for each model is provided in the table. For more precise bit-width results, please refer to Table 1.**
| Settings | LLaMA-1 | LLaMA-2 | LLaMA-3 |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | Block Size | W-Bits | 7B | 13B | 30B | 65B | 7B | 13B | 8B |
| FullPrecision | - | 16 | 5.68 | 5.09 | 4.1 | 3.53 | 5.47 | 4.88 | 6.10 |
| RTN | - | 1 | 1.7e5 | 1.4e6 | 1.5e4 | 6.5e4 | 1.6e5 | 4.8e4 | 2.7e6 |
| GPTQ | 128 | 1 | 2.7e5 | 1.1e5 | 6.7e4 | 2.5e4 | 1.2e5 | 9.4e3 | 5.7e4 |
| PB-LLM | 128 | 1.7 | 102.36 | 36.6 | 33.67 | 12.53 | 69.2 | 151.09 | 41.80 |
| BiLLM | 128 | 1.09 | 35.04 | 15.14 | 10.52 | 8.49 | 32.48 | 16.77 | 28.30 |
| BiLLM | 128 | 0.80 (6:8) | 80.36 | 22.55 | 13.22 | 9.09 | 50.25 | 27.28 | 94.15 |
| BiLLM | 128 | 0.70 (5:8) | 126.99 | 39.61 | 18.69 | 11.57 | 87.84 | 58.14 | 161.48 |
| BiLLM | 128 | 0.55 (4:8) | 688.73 | 124.72 | 37.96 | 29.22 | 263.61 | 124.78 | 663.91 |
| STBLLM | 128 | 0.80 (6:8) | 15.03 | 9.66 | 7.56 | 6.43 | 13.06 | 11.67 | 33.44 |
| STBLLM | 128 | 0.70 (5:8) | 19.48 | 11.33 | 9.19 | 7.91 | 18.74 | 13.26 | 49.12 |
| STBLLM | 128 | 0.55 (4:8) | 31.72 | 17.22 | 13.43 | 11.07 | 27.93 | 20.57 | 253.76 |

**Table 3: Perplexity results on Wikitext2 datasets of OPT and Mistral models with BiLLM and STBLLM. For more precise bit-width results, please refer to Table 1.**
| Settings | OPT | Mistral |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Method | W-Bits | 1.3B | 2.7B | 6.7B | 30B | 7B |
| BiLLM | 0.80 (6:8) | 51.62 | 23.03 | 15.82 | 15.82 | 72.29 |
| BiLLM | 0.70 (5:8) | 69.15 | 30.62 | 20.58 | 20.58 | 82.84 |
| BiLLM | 0.55 (4:8) | 106.99 | 55.28 | 79.68 | 79.68 | 189.73 |
| STBLLM | 0.80 (6:8) | 29.84 | 17.02 | 12.79 | 12.80 | 27.31 |
| STBLLM | 0.70 (5:8) | 33.01 | 20.82 | 14.38 | 14.38 | 25.64 |
| STBLLM | 0.55 (4:8) | 45.11 | 30.34 | 18.80 | 18.80 | 70.14 |

## 4 Experiments

### 4.1 Implementation Details

Experimental Setup. Our STBLLM utilizes PyTorch (Paszke et al., ) and Huggingface (Wolf et al., ) libraries. Most LLMs except 65B can be evaluated on a single NVIDIA A800 GPU. For the LLaMA-1-65B model, we employ four NVIDIA A800 GPUs for evaluation.
It takes 1.8 hours for the post-training process of 7B models on an RTX 4090 GPU and 2.8 hours for 13B models on an A6000 GPU. Following BiLLM (Huang et al., ), our proposed STBLLM also eliminates the need for fine-tuning, offering an efficient post-training quantization framework.

Datasets and Models. We measure the perplexity for language generation tasks on Wikitext2 (Merity et al., ), C4 (Raffel et al., ) and PTB (Marcus et al., ), and accuracy for the zero-shot tasks including Winogrande (Sakaguchi et al., ), OBQA (Mihaylov et al., ), Hellaswag (Zellers et al., ), BoolQ (Clark et al., ), ARC (Clark et al., ) and RTE (Chakrabarty et al., ). We conduct experiments on LLaMA-1/2/3 (Touvron et al., ; ), OPT (Zhang et al., ), and Mistral (Jiang et al., ). For perplexity evaluation in Table 2 and  3, we employ the C4 dataset as the calibration dataset and report the perplexity on Wikitext2.

**Table 4: Accuracies (%) for 7 zero-shot tasks from structured binarized LLaMA-1-13B, LLaMA-2-13B, and LLaMA-1-30B with BiLLM and STBLLM. We compare the performance under the same N:M setting to achieve sub-1-bit quantization.**
| Models | Method | Winogrande | OBQA | Hellaswag | Boolq | ARC-e | ARC-c | RTE | Mean |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| LLaMA-1-13B | FullPrecision | 72.77 | 33.20 | 59.94 | 77.89 | 77.40 | 46.50 | 70.40 | 62.59 |
| BiLLM(6:8) | 58.80 | 30.60 | 46.25 | 62.96 | 49.96 | 23.97 | 53.42 | 46.57 |  |
| BiLLM(4:8) | 52.09 | 28.00 | 30.82 | 61.25 | 32.66 | 21.25 | 53.07 | 39.88 |  |
| STBLLM(6:8) | 65.98 | 36.20 | 63.67 | 65.38 | 68.86 | 34.04 | 56.68 | 55.83 |  |
| STBLLM(4:8) | 63.06 | 34.80 | 52.65 | 62.48 | 56.90 | 28.33 | 52.71 | 50.13 |  |
| LLaMA-2-13B | FullPrecision | 72.22 | 35.20 | 60.06 | 80.52 | 79.42 | 48.46 | 65.34 | 63.03 |
| BiLLM(6:8) | 56.43 | 30.60 | 35.53 | 62.48 | 41.29 | 24.74 | 53.43 | 43.50 |  |
| BiLLM(4:8) | 50.59 | 24.00 | 28.96 | 62.08 | 30.51 | 22.35 | 53.07 | 38.79 |  |
| STBLLM(6:8) | 63.93 | 37.00 | 57.76 | 71.53 | 60.56 | 31.99 | 54.15 | 53.85 |  |
| STBLLM(4:8) | 55.88 | 29.40 | 44.03 | 64.31 | 48.86 | 26.54 | 52.71 | 45.96 |  |
| LLaMA-1-30B | FullPrecision | 75.69 | 36.00 | 63.35 | 82.69 | 80.30 | 52.82 | 66.79 | 65.38 |
| BiLLM(6:8) | 66.54 | 36.40 | 58.18 | 66.15 | 62.37 | 31.91 | 46.93 | 50.32 |  |
| BiLLM(4:8) | 54.93 | 29.40 | 38.85 | 62.17 | 43.6 | 24.74 | 52.35 | 43.72 |  |
| STBLLM(6:8) | 71.59 | 41.00 | 69.85 | 77.37 | 71.55 | 41.3 | 48.01 | 60.10 |  |
| STBLLM(4:8) | 64.01 | 34.60 | 56.46 | 63.06 | 60.86 | 31.48 | 51.99 | 51.78 |  |

Baseline. Our primary baseline is BiLLM (Huang et al., ), which is a 1-bit PTQ framework for LLMs. We perform an N:M sparse pattern on pre-trained LLMs and then conduct the same procedure as BiLLM to report the results that are less than 1 bit (e.g. 0.8, 0.7, 0.55 bits). We conduct the N:M sparsity using Wanda (Sun et al., ) as the baseline, a gradient-free post-training pruning method. We compare the results of STBLLM with BiLLM under the same N:M settings. For more information on average bits under N:M settings, please refer to Table 1. Previous low-bit methods like PB-LLM (Shang et al., ), GTPQ (Frantar et al., ) and vanilla RTN are also selected for comparison.

### 4.2 Main Results

Comparison with PTQ methods.
We comprehensively compare the performance of different LLaMA families across various model sizes (7B-65B). For a fair comparison, we set the same block size to 128. As presented in Table 2, the model under RTN and GPTQ fails to retain the performance at 1-bit. PB-LLM has shown a satisfactory perplexity under 1.7 bit but deteriorates performance compared with BiLLM under 1.09 bit. To further compare the performance at sub-1-bit, we apply the same N:M setting to BiLLM and our proposed STBLLM. As shown in Figure 2, our proposed STBLLM achieves a better trade-off between bit-widths and perplexity across model sizes from 7B to 65B.
STBLLM surpasses BiLLM by a large margin ($688.73\rightarrow 31.72$) on LLaMA-1-7B, especially on the most extreme compression case, 4:8 structured binarization, which means setting half of the parameter to zero.
It is also noteworthy that when the parameter size reaches 65B, our STBLLM, at 0.55 bit, achieves a perplexity of 11.07, surpassing that of PB-LLM (12.53) at 1.7 bit and that of BiLLM (11.57) at 0.7 bit.
To our knowledge, our STBLLM is the first work that breaks the 1-bit barriers by further reducing the redundant weights in an N:M pattern.
Moreover, we conduct further experiments on the OPT family from 1.3B to 30B and Mistral-7B at sub-1-bit PTQ settings. From Table 3, we observe the same trend as in LLaMA. Our proposed STBLLM performs significantly better than BiLLM across all models and all N:M structured binarization settings.

Table: Table 5: Ablation for pruning metric.

| Model | Magnitude | Wanda | SparseGPT | Ours (SI) |
| --- | --- | --- | --- | --- |
| LLaMA-1-7B | 4797.41 | 207.32 | 32.82 | 31.72 |
| LLaMA-2-7B | 2287.24 | 97.54 | 31.55 | 27.93 |

Table: Table 6: Ablation study for allocation strategy.

| Models | Uniform | Sin-shape | Ours |
| --- | --- | --- | --- |
| LLaMA-1-7B | 80.36 | 67.78 | 15.03 |
| LLaMA-2-7B | 50.25 | 33.61 | 13.06 |

Zero-Shot Performance.
To conduct a more comprehensive evaluation of binary LLMs, we extend our experiments to 7 zero-shot datasets on LLaMA-1-13B, LLaMA-2-13B, and LLaMA-1-30B, each tested with FullPrecision, BiLLM(6:8), BiLLM(4:8), STBLLM(6:8), and STBLLM(4:8) methods. We mainly focus on the performance of these models under the sub-1-bit setting. Specifically, we compare the BiLLM and our STBLLM under 4:8 and 6:8 structured binarization settings. As illustrated in Table 4, we find that the performance drop in reduced precision methods is more pronounced in BiLLM methods compared to STBLLM methods, indicating that STBLLM methods are more robust alternatives when memory resources are constrained.

### 4.3 Enhancing Inference Efficiency on Hardware.

We present specialized CUDA kernels designed to support 1-bit 2:4 sparsification. As illustrated in Figure 4(a), we utilize a 2-bit implementation on recent RTX4090 GPU from ABQ-LLM (Zeng et al., ) as the baseline (W2A16 and W2A8) and compare it with our highly-optimized 2:4 1-bit implementation. We provide a comparative analysis of runtime and throughput across various sequence lengths, demonstrating the significant gains in computational efficiency and reduced memory footprint.
Specifically, for typical sequence lengths of 4096 and 8192, our implementation achieves up to 17.85 times speedup compared to ABQ-LLM’s 2-bit implementation. At a sequence length of 8192, our kernel reaches 263.45 TFLOPS, which is 79.74% of the RTX4090’s 2:4 sparse tensor core peak performance. Notably, the speedup becomes more pronounced as sequence length increases. Furthermore, as illustrated in Figure 4 (b), our method yields lower perplexity for LLaMA-1/2 models. This indicates superior model performance and accuracy compared to 2-bit round-to-nearest (RTN), GPTQ, and AWQ. Refer to Appendix C for the memory comparison and implementation details.

### 4.4 Ablation Studies

Ablation for Metric. Table 4.2 shows the impact of post-training pruning metrics (Magnitude, Wanda (Sun et al., ), SparseGPT (Frantar & Alistarh, ) and our SI) on STBLLM regarding LLaMA-1-7B and LLaMA-2-7B. During PTP, we employ the C4 dataset as the calibration dataset and report the perplexity on the Wikitext2 dataset. SparseGPT requires second-order information, which involves a massive computation burden. Similar to Wanda, our SI does not require gradient or second-order information. Our method achieves better performance among these metrics.

Ablation for Quantization Strategy. We conduct an ablation study on different quantization strategies. Comparing the perplexity of our Non-salient Aware Quantization (dubbed as Non-salient) and Bell-shaped Distribution Splitting (dubbed as Bell-shaped) in BiLLM (Huang et al., ) on both LLaMA-1-7B and LLaMA-2-7B, as shown in Table 4.4. The perplexity of Non-salient changes a lot when moving from LLaMA-1-7B to LLaMA-2-7B, while our Non-salient exhibits nearly identical perplexity in both models, significantly lower than that of Bell-shaped.

Ablation for Allocation Strategy. Table 4.2 presents an ablation study on different allocation strategies. We compare our method with Uniform and Sin-shaped allocation strategies. The Sin-shaped strategy assigns layer-wise sparsity following a sine wave pattern, where the initial layers have lower sparsity and the latter have higher sparsity. The performance of Uniform and Sin-shaped strategies varies significantly across different models. In contrast, our strategy consistently achieves nearly identical performance across both models, outperforming the other two allocation strategies.

Ablation for Group Size. Table 9 presents the results of our ablation study on the group size configuration. We evaluate the perplexity of LLaMA-1-7B and LLaMA-2-7B with group sizes of 64, 128, 256, and 512. Generally, as the group size increases, performance improves. However, this also results in higher computational and storage demands. We choose a group size of 128 to balance performance and resource consumption.

**Table 7: Comparison of Magnitude, Wanda, SparseGPT, and SI across different datasets.**
| Models | LLaMA-1-7B |  | Models | LLaMA-2-7B |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Dataset | Magnitude | Wanda | SparseGPT | Ours(SI) | Dataset | Magnitude | Wanda | SparseGPT | Ours(SI) |
| PTB | 11608.88 | 306.57 | 61.53 | 68.48 | PTB | 45564.36 | 2027.33 | 236.03 | 690.76 |
| C4 | 1545.34 | 153.29 | 33.06 | 36.04 | C4 | 1034.84 | 86.45 | 30.53 | 30.81 |
| Wikitext2 | 4797.42 | 207.32 | 32.82 | 31.72 | Wikitext2 | 2287.25 | 97.54 | 31.56 | 27.93 |

Table: Table 8: Ablation for quantization strategy.

| Models | Bell-shaped | Non-salient |
| --- | --- | --- |
| LLaMA-1-7B | 80.35 | 15.03 |
| LLaMA-2-7B | 50.25 | 13.06 |

Table: Table 9: Ablation for group size.

| Model | 64 | 128 | 256 | 512 | 1024 |
| --- | --- | --- | --- | --- | --- |
| LLaMA-1-7B | 29.58 | 31.72 | 33.97 | 41.29 | 146.46 |
| LLaMA-2-7B | 27.12 | 27.93 | 50.62 | 54.68 | 507.44 |

Figure: (a) Runtime and throughput comparison.
Refer to caption: extracted/5906344/pic/hardware_p1.png

## 5 Conclusion

In this paper, we introduce STBLLM, a structured Binary LLM PTQ framework designed for sub-1-bit quantization. We address redundancy in binarized LLMs, highlighting the potential for further compression. Specifically, we present a Standardized Importance (SI) metric for N:M structured pruning. Then, we use the Hessian matrix to partition weights into salient and non-salient categories. We propose Non-salient Aware Quantization for non-salient weights, identifying optimal splitting points to create sparse, intermediate, and dense regions, each with tailored binarization. Finally, we design a specialized CUDA kernel with a sparse tensor core to achieve significant speedup. We validate the performance of STBLLM across LLaMA-1/2/3, OPT, and Mistral, demonstrating that STBLLM achieves a superior trade-off at sub-1-bit settings. By achieving LLM performance under 1 bit, STBLLM highlights the potential of extreme LLM compression. Limitation: STBLLM does not support Mixture of Experts (MoE) or Mamba-based language models.

## Appendix Overview

- •
Section A: STBLLM Implementation.
- •
Section B: Details of Motivation Experiments.
- •
Section C: Details of Hardware Accelerations.
- •
Section E: More Experimental Results.
- •
Section D: Impact of Extreme Weight on the Hessian Matrix.

## Appendix A STBLLM Implementation

Following BiLLM (Huang et al., ), STBLLM does not change the operations on salient weights. Instead, STBLLM mainly focuses on the non-salient weight. We present NonSalientAwareQuant and Trisection function in Algorithm 2.

For NonSalientAwareQuant function, it aims to find two optimal break-points to partition the symmetric Gaussian distribution of non-salient weight. A naive approach for searching the break-point is using two nested loops, whose complexity is $O(N^{2})$, where $N$ denotes the length of the search space. To reduce the complexity to $O(N)$, we propose to utilize $p_{2}=\sigma\times p_{1}$ to locate the $p_{2}$. It is natural to assume that $p_{2}>p_{1}$ and we have $\sigma>1$. In practice, we set the $\sigma=2$ and it works well.

For Trisection function, it aims to partition the symmetric Gaussian distribution presented in Figure 3(c) into three parts, which are Sparse, Intermediate, and Dense region. These three parts have no intersection and by uniting them together, we have all of the non-salient structured binarized weight.

Figure: Algorithm 2 STBLLM

## Appendix B Details of Motivation Experiment

In this section, we delineate the specifics of the motivation experiments as illustrated in Figure 2. Initially, we elucidate the procedure for inverting the signs of elements within a matrix, as detailed in Algorithm 3, to examine its effects on various computational tasks. This algorithm is designed to efficiently invert the signs of a specified proportion of elements in a given matrix $\mathbf{W}$. Subsequently, we employ this function on $RES_APPROX$ and invert the signs of each binary matrix, including $B_{1}$ and $B_{2}$.

Figure: Algorithm 3 Algorithm for Efficiently Flipping Signs of Matrix Elements

## Appendix C Details of Hardware Acceleration

### C.1 Implementation Details

Recent advancements in low-precision computing have significantly enhanced the practical implementation of efficient neural network techniques. A prime example is the introduction of Ladder (Wang et al., ), released as BitBLAS, a software library that seamlessly integrates into existing Deep Neural Network (DNN) and Large Language Model (LLM) frameworks. This integration enables highly efficient low-precision computations across various hardware platforms. The impact of these developments is evident in popular frameworks like llama.cpp (Gerganov, ), which now supports 1.5-bit quantization through BitNet (Wang et al., ). This advancement has resulted in impressive performance gains, achieving 198 tokens per second on a single CPU core. Moreover, for large-scale models such as LLaMA-2-70B, the implementation of Ladder (Wang et al., ) to accelerate BitNet 1.58 (Wang et al., ) has yielded remarkable results, demonstrating a 4.6$\times$ speedup compared to FP16 precision.

The emergence of Sparse Tensor Cores (SPTCs) since NVIDIA’s Ampere architecture has revolutionized the processing of sparse matrices, offering an efficient mechanism for handling 50% sparsity. Theoretically, by eliminating half of the computations, SPTCs can potentially double the computational power compared to Dense Tensor Cores. There are already several research over accelerating dense tensor core, including ULPPack (Won et al., ), NGEMM (Bao et al., ) and QQQ (Zhang et al., ). However, efficiently representing 1-bit values ($+1$, $-1$, and 0 0 for sparsity) and achieving sufficient real-world acceleration pose significant challenges. To address these, we propose a novel 2-bit integer representation method, particularly useful for the General Matrix Multiply (GEMM) operation, formulated as $D=A(E)\times B+C$. Here, $A$ represents the 2:4 1-bit sparse matrix, $B$, $C$, and $D$ are dense tensors, and $E$ employs uint16 to denote the valid indices of $A$.

Our approach introduces a 6-bit encoding scheme for each group of 2:4 sparse 1-bit values. This scheme comprises four bits for indexing and two bits for physical value representation, where $1\rightarrow+1$, $0\rightarrow-1$, and positions unmarked by $E$ indicate sparsity (0 0). This method significantly improves memory efficiency compared to a baseline approach using 2-bit integers to represent $-1$, 0 0, and $+1$, which would require 8 bits for an equivalent group size. Consequently, our encoding method reduces memory footprint by approximately 25%, leading to decreased global memory access requirements. In memory-bound scenarios typical of large-scale model inference, this approach theoretically offers up to a 1.333-fold increase in processing speed compared to the 2-bit variant.

Figure: Figure 5: Structured Pruned Matrix 4-Group Index Representation for 2:4 Structured Sparsity Acceleration.
Refer to caption: extracted/5906344/pic/hardware-figure1.png

Figure: Figure 6: The overview of sparsity pattern of 1-bit kernel that convert weight matrix to structured pruned matrix.
Refer to caption: extracted/5906344/pic/hardware-figure2.png

To fully leverage these optimizations, we employ semi-structured pruning techniques specifically tailored for NVIDIA’s GPU architecture. These techniques enable the use of Sparse Tensor Cores optimized for processing sparse matrices. By structuring the sparsity (e.g., $N$:$M$ sparsity where $N$ out of $M$ weights are non-zero), we can effectively utilize the Sparse Tensor Cores, leading to substantial improvements in processing speed and efficiency. Specifically, the process of implementing these optimizations involves several key steps in matrix compression and manipulation:

- 1.
Matrix Compression: The input matrix is partitioned into 4-element groups as shown in Figure 5. Zero elements are identified and non-zero elements are extracted. The positions of non-zero elements are recorded in a Non-Zeros Meta Index, which is then encoded into a compact Uint16 Meta Index. This encoding facilitates efficient localization of non-zero elements during matrix operations, enhancing computational speed by enabling the omission of zero elements.
- 2.
Value Compression: Similar to matrix compression, the matrix is divided into 4-element groups as shown in Figure 7. The sign of each non-zero element is extracted to form the Non-Zeros Pruned-Binary Real Value. These values are then converted to a binary format, creating the Non-Zeros Real Value. Finally, the each of eight binary values are concatenated into a compact Uint8 Real Value, optimizing storage and computation by focusing on the non-zero elements and their signs.
- 3.
Matrix Multiplication with Structured Sparsity: The input matrices undergo pruning to retain only non-zero elements and their corresponding indices as shown in Figure 7. The pruned matrices are then quantized, extracting non-zero values and their positions. Both processed matrices are subsequently input into the Sparse Tensor Cores, which executes efficient multiplication by focusing on the non-zero elements, resulting in a compressed and accelerated computation.

Figure: Figure 8: Roofline for Sparse GEMM Quantization.
Refer to caption: extracted/5906344/pic/roofline_analysis.png

### C.2 Theory Analysis

To evaluate the performance of various matrix multiplication algorithms across different problem sizes, we present a comprehensive roofline model analysis in Figure 8. Each subplot depicts the relationship between arithmetic intensity (FLOPS/Byte) and performance (TFLOPS).
During prefilling stage, $N$ denotes the product of sequence length and batch size. For decoding stage, $N$ denotes the batch size. $M$ and $K$ correspond to the dimension of weight matrix.
To compare different implementations, we include standard FP16 GEMM, 2-bit quantized GEMM, and 1-bit 2vs4 quantized GEMM, alongside theoretical performance limits represented by roofline models for Tensor Core and Tensor Core Sparse operations. From our analysis, we observe that as N increases, all algorithms exhibit improved performance, with quantized versions consistently outperforming standard GEMM. We find that our 1-bit 2vs4 quantized GEMM demonstrates superior performance, particularly at larger N values, often approaching the Tensor Core Sparse roofline.

The advantages of our 1-bit 2:4 quantized GEMM kernel arise from reduced memory access overhead and the higher compute upper bound of Sparse Tensor Cores (SPTCs). When $N$ is small (particularly during the decoding phase), all GEMM kernels are memory-bound, but our 1-bit 2:4 quantized GEMM kernel achieves relatively better performance due to its higher compression rate. As $N$ increases (especially during the prefilling stage), the quantized GEMM kernels tend to become compute-bound. In this case, our specialized GEMM kernel can theoretically achieve a 2$\times$ speedup compared to other GEMM kernels.

This extreme quantization approach significantly reduces both computational overhead and memory footprint by limiting the precision of weights to just two possible states. Such a method is particularly advantageous in resource-constrained environments, improving the deployability of large models on devices with limited hardware capabilities.

### C.3 Memory Comparison

As illustrated in Figure 9, we present the memory consumption of FP16, CUTLASS, ABQ-LLM, and our implementation for LLaMA-7B, 13B, and 30B models. Our proposed methodology demonstrates a substantial memory compression gain, exceeding 3.1 times that of SmoothQuant. This performance significantly surpasses current mainstream inference techniques. Furthermore, our approach achieves an approximate 15% reduction in memory usage compared to ABQ-LLM. These notable improvements have important implications for the field of large language models (LLMs). By reducing the memory footprint, our method decreases the operational costs associated with LLM services and facilitates their practical deployment in real-world applications.

Figure: Figure 9: Memory Usage Comparison of Various Quantization Methods for LLaMA Models
Refer to caption: extracted/5906344/pic/memory_hardware.png

## Appendix D Impact of Extreme Weight on the Hessian Matrix

The Hessian matrix $H$ is defined as: $H_{ij}=\frac{\partial^{2}L}{\partial w_{i}\partial w_{j}}$, where $L$ is the loss function, and $w_{i}$ and $w_{j}$ are weights. If a weight $w_{k}$ has extreme values, the corresponding elements in the Hessian matrix, particularly $H_{kk}$, will be significantly larger than others.

For instance, if $w_{1}$ is an extreme value, the Hessian matrix might look like:

$$ $H=\begin{pmatrix}h_{11}&h_{12}&\cdots&h_{1n}\\ h_{21}&h_{22}&\cdots&h_{2n}\\ \vdots&\vdots&\ddots&\vdots\\ h_{n1}&h_{n2}&\cdots&h_{nn}\end{pmatrix}$ $$

Here, $h_{11}$ is much larger than other elements. This disproportionate value significantly influences the Hessian’s eigenvalues, with at least one eigenvalue becoming very large.
During optimization, methods like Newton’s method update weights using the inverse of the Hessian matrix:

$$ $\mathbf{w}_{\text{new}}=\mathbf{w}-\eta H^{-1}\nabla L(\mathbf{w}),$ $$

where $\eta$ is the learning rate, and $\nabla L(\mathbf{w})$ is the gradient. The presence of an extreme value in $h_{11}$ causes the corresponding element in $H^{-1}$ to be very small, affecting the step size in weight updates:

$$ $\Delta w_{1}\approx-\eta\frac{\partial L}{\partial w_{1}}/h_{11},$ $$

$$ $\Delta w_{2}\approx-\eta\frac{\partial L}{\partial w_{2}}/h_{22}.$ $$

Since $h_{11}$ is large, $\Delta w_{1}$ becomes small, indicating minimal adjustments for the extreme value weight, while $\Delta w_{2}$ remains relatively larger for the normal weights.

## Appendix E More Experimental Results

Figure: Figure 10: Ablation study on post-training pruning metrics on STBLLM on LLaMA-1-7B and LLaMA-2-7B. Our method achieves the best performance among these metrics.
Refer to caption: x2.png

Figure: Figure 12: Comparison across different sizes for LLaMA-1-7B and LLaMA-2-7B.
Refer to caption: x4.png

### E.1 Module Ablation Study

To evaluate the interdependent interaction between quantization and pruning within our STBLLM framework, we conduct a module ablation study. This study isolates the effects of quantization-only, pruning-only, and our combined method on the performance of the LLaMA-1-7B and LLaMA-2-7B models across the PTB, C4, and Wikitext2 datasets. The results are presented in Table 10.

**Table 10: Comparison of Quant-Only, Structure-Only, and ours across different datasets.**
|  | LLaMA-1-7B | LLaMA-2-7B |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Dataset | Quant-Only | Structure-Only | Ours | Quant-Only | Structure-Only | Ours |
| PTB | 23.52 | 14.24 | 68.48 | 2071.44 | 69.25 | 690.76 |
| C4 | 15.75 | 10.52 | 36.04 | 14.62 | 10.29 | 30.81 |
| Wikitext2 | 12.29 | 8.13 | 31.72 | 11.17 | 7.85 | 27.93 |

The ablation results highlight the synergistic effect of combining quantization and pruning in our approach, significantly outperforming each method applied in isolation.

LLaMA-1-7B Analysis

- PTB Dataset: Our combined method achieves a score of 68.48, markedly higher than quantization-only (23.52) and pruning-only (14.24). This demonstrates the substantial performance gains achieved by leveraging the complementary strengths of both techniques.

- C4 Dataset: Our method scores 36.04, compared to 15.75 for quantization-only and 10.52 for pruning-only. The combined approach effectively mitigates the limitations of individual methods, resulting in superior performance.

- Wikitext2 Dataset: The score of 31.72 for our method far exceeds the results of quantization-only (12.29) and pruning-only (8.13), underscoring the enhanced model efficiency and accuracy through our integrated approach.

LLaMA-2-7B Analysis

- PTB Dataset: Although quantization-only achieves an unusually high score of 2071.44, our combined method still significantly outperforms pruning-only (690.76 vs. 69.25). This suggests that while quantization might retain certain advantageous structures, the integration with pruning leads to a more balanced and robust model.

- C4 Dataset: The combined method’s score of 30.81 surpasses quantization-only (14.62) and pruning-only (10.29), highlighting the effectiveness of our method in maintaining high performance across varying model versions.

- Wikitext2 Dataset: Our method’s score of 27.93 is higher than both quantization-only (11.17) and pruning-only (7.85), further confirming the synergistic benefits of combining these techniques.

### E.2 Ablation Study of Calibration Dataset

Table 11 presents an ablation study comparing the performance of LLaMA-1-7B and LLaMA-2-7B models when trained on different calibration datasets: C4, PTB, and Wikitext2. The purpose of this experiment is to investigate how the choice of calibration dataset affects the models’ performance on various evaluation datasets.

In this study, both LLaMA-1-7B and LLaMA-2-7B models are trained on each of the three calibration datasets separately. The trained models are then evaluated on all three datasets, resulting in a 3x3 matrix of performance scores for each model.

The performance scores in the table likely represent some evaluation metric, such as perplexity or loss, where lower values indicate better performance. The diagonal values (e.g., C4 evaluated on C4) represent in-domain performance, while off-diagonal values represent out-of-domain performance.

**Table 11: Comparison of C4, PTB, and Wikitext2 across LLaMA-1-7B and LLaMA-2-7B**
|  | LLaMA-1-7B | LLaMA-2-7B |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Dataset | C4 | PTB | Wikitext2 | C4 | PTB | Wikitext2 |
| C4 | 36.04 | 68.48 | 31.72 | 30.81 | 690.76 | 27.93 |
| PTB | 54.57 | 35.13 | 49.27 | 43.04 | 4569.03 | 40.94 |
| Wikitext2 | 40.76 | 71.81 | 20.48 | 37.01 | 1970.76 | 20.60 |

### E.3 Ablation Study of Group Size

Table 12 and Figure 12 presents an ablation study that compares the performance of LLaMA-1-7B and LLaMA-2-7B models across different group sizes. The purpose of this experiment is to investigate how the choice of group size affects the models’ performance on various evaluation datasets.

In this study, both LLaMA-1-7B and LLaMA-2-7B models are trained with different group sizes: 64, 128, 256, 512, and 1024. The trained models are then evaluated on three datasets: C4, PTB, and Wikitext2. The performance scores in the table likely represent some evaluation metric, such as perplexity or loss, where lower values indicate better performance. By comparing the performance scores across different group sizes and evaluation datasets, researchers can gain insights into the impact of group size on the models’ performance and generalization capabilities.

The results show that the performance of both models varies with the choice of group size. For LLaMA-1-7B, the best performance on C4 and Wikitext2 is achieved with a group size of 64, while for PTB, the best performance is obtained with a group size of 128. For LLaMA-2-7B, the best performance on C4 and Wikitext2 is also achieved with a group size of 64, while for PTB, the best performance is obtained with a group size of 256. Interestingly, the performance of both models deteriorates significantly when the group size is increased to 1024, suggesting that excessively large group sizes may lead to overfitting or other training issues.

Table: Table 12: Comparison across different sizes for LLaMA-1-7B and LLaMA-2-7B

**Table 13: Motivation vs Top Percentage**
| Top Percentage | Perplexity | Top Percentage | Perplexity | Top Percentage | Motivation |
| --- | --- | --- | --- | --- | --- |
| 0.01 | 27.770422 | 0.02 | 30.168285 | 0.03 | 34.049734 |
| 0.04 | 36.191769 | 0.05 | 33.821476 | 0.06 | 36.452296 |
| 0.07 | 38.702617 | 0.08 | 39.169894 | 0.09 | 44.818825 |
| 0.10 | 54.451229 | 0.11 | 49.835159 | 0.12 | 71.762848 |
| 0.13 | 52.129317 | 0.14 | 52.568348 | 0.15 | 65.945448 |
| 0.16 | 62.712751 | 0.17 | 117.990227 | 0.18 | 138.912356 |

The provided Figure 2 and Table 13 present an experiment that investigates the relationship between the top Percentage of data and the corresponding perplexity scores in a LM. The purpose of this experiment is to understand how the choice of top Percentage affects the model’s performance and to determine an optimal threshold for data selection.
In this experiment, we randomly flip 1%-16% weights from binarized LM and evaluate their downstream tasks’ performance including ARC (Clark et al., ), BoolQ (Mihaylov et al., ), Hellaswag (Zellers et al., ) and RTE (Chakrabarty et al., ). Table 13 shows the perplexity scores for each top Percentage. Lower perplexity scores indicate better language model performance, as the model is better able to predict the next word in a sequence.

Figure 2 provides a visual representation of the relationship between the top Percentage and perplexity scores. It shows that the perplexity scores initially improve as the top Percentage increases, indicating that including more high-quality data points benefits the model’s performance. However, beyond a certain threshold (around 0.05 to 0.10), the perplexity scores start to deteriorate, suggesting that including lower-quality data points negatively impacts the model’s performance.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: See weakness
soundness: 2
strengths: * The analysis on flipping non-salient binarized weights is intriguing. I am wondering what would happen if we increase the ratio from 0.15 to 0.5?
* The proposed method achieves the lowest perplexity among all compared methods in the sub-1-bit regime.
* A specialized CUDA kernel for structural binarization, leveraging NVIDIA's Ampere GPU sparse tensor cores, achieves a 17.85x speedup over ABQ-LLM's 2-bit implementation.
summary: This work presents a structural binarization method for LLMs by combining N:M sparsity, residual approximation, and block-wise error compensation. Extensive experiments on LLaMA-1/2/3, OPT, and Mistral are conducted to evaluate the effectiveness of STBLLM. In addition, a specialized CUDA kernel is designed to support structural binarization.
weaknesses: * The proposed method is a combination of several existing techniques including N:M sparsity, residual approximation, block-wise error compensation, and Trisection search (for the non-salient part). This raises some novelty concerns. I suggest the authors to 1) highlight the main novelty and contribution of the current submission; 2) provide ablation studies on a. how important the residual approximation is, b. the impact of Trisection search for grouping and why there are two groups.
In addition, which techniques contribute the most to efficiency and which method contributes the most to the accuracy?
* The benchmark results are based on various N:M configurations. However, NVIDIA GPUs mainly support 2:4. The authors may discuss how practical the proposed method is on NVIDIA GPUs.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Please see the weaknesses.
soundness: 2
strengths: + 1-bit weight quantization is important for accelerating LLM inference.
+ Dedicated CUDA implementations for the proposed method.
summary: This paper proposes a structured binary quantization method to accelerate LLM inference. It combines n:m pruning and binary quantization, compressing the model weights to an average of less than 1 bit. In n:m pruning, the authors introduce an SI method for indentifying significant weights, and a layer-wise dynamic n:m allocation method. In binary quantization, the authors partition the weights into salient and non-salient parts for separate processing and further apply a group-wise quantization method to the non-salient part. Experimental results demonstrate that STBLLM outperforms BiLLM under the same bit budget. In addition, significant performance improvement (17x) is achieved with customized CUDA kernels.
weaknesses: + incremental novelty

While the proposed method is interesting and performs better than BiLLM, its novelty is limited: 1) The proposed SI method is very similar to Wanda, with the main difference being the introduction of additional data normalization. 2) The binary quantization method is quite similar to BiLLM, where the hessian matrix is used to divide weights into salient and non-salient parts, and residual approximation is employed to handle the salient part. The only difference is that STBLLM processes the non-salient weights into three parts instead of two as in BiLLM.

+ mismatch between motivation and methodology

The motivation of this paper lies in the observation that some weights in binary LLMs do not significantly affect accuracy and can be further compressed (Section 3.1 and fig 1). Under this narrative, a reasonable approach would be to perform pruning on the binarized model to achieve further compression. In contrast, the method proposed in this paper adopts a ‘’prune-then-quantize’’ approach, which does not align with the motivation. The paper does not explain why pruning should be performed first and does not discuss how changing the order of pruning and binarization might affect the results. 

The motivation behind using a trisection-based partition for non-salient weights is confusing. It seems the authors aim to balance bits and performance (Section 3.4). However, the evaluation results show that the improved compression ratio and performance are due to n:m pruning, rather than the processing of non-salient weights. So, why should we partition the non-salient weights into three parts? Why not four or five? What do the terms dense, intermediate, and sparse mean?

+  confusing evaluations

While the experimental results of STBLLM are promising, the source of the accuracy improvements remains unclear. The experimental settings in the ablation study are somewhat confusing. For instance, Table 5 examines the effectiveness of the SI method in n:m pruning, but the results seem to represent 4:8 pruning plus binarization. What binarization method is used in the baselines? Table 8 directly compares STBLLM with BiLLM to illustrate the effectiveness of trisection partitioning, yet the pruning methods used in STBLLM and BiLLM are not the same (SI vs. Wanda). A detailed, step-by-step breakdown analysis of each technique's effectiveness would be helpful. Moreover, where does the 17x performance improvement come from when reducing 2-bit weights to 1 bit?

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: + Regarding Figure 3, part (b), after structured pruning, the empty parts should have no values. Why are zeros assigned to these parts? Additionally, structured pruning usually doesn't achieve weight-wise pruning, so what does "structured" mean in this context?
soundness: 3
strengths: + The paper is well-organized and easy to follow, with a clearly stated problem.
+ It introduces a new metric to assess weight importance, avoiding expensive second-order gradient computations and mitigating the impact of extreme values.
+ It is interesting that separate binarization for non-salient weights retains crucial information in this segment, enhancing model performance.
+ The approach is logical and rigorous, discussing the method from various perspectives and fully validating its effectiveness through comprehensive experiments.
summary: This paper proposes an efficient framework for LLMs, combining pruning and binarization to compress large, post-trained models. By applying N:M sparsity, it achieves precision below 1-bit and identifies salient weights through a newly introduced Standardized Importance (SI) metric. This metric considers both weight and activation values, avoiding the costly second-order computations typically required. Additionally, during pruning and binarization, the method separates non-salient weights into three groups to preserve as much information as possible in these parts. Extensive experiments demonstrate that the proposed method significantly reduces computational costs, accelerates inference, and maintains strong performance.
weaknesses: - In the zero-shot experiment, the paper mentions seven zero-shot tasks. It would be helpful to include a brief description of each task to provide readers with a clearer understanding of the evaluation scope.

---

confidence: 2
contribution: 2
initial_rating: 5
presentation: 3
questions: - Regarding the calculation of average bit count:
   - Could you clarify whether the overhead from indices associated with sparsity has been factored into the calculation?
   - It would be helpful if you could provide a concrete example illustrating the calculation methodology, including both the weight bits and any additional storage requirements.

- In Algorithm 1, there appears to be some ambiguity regarding the Semi-Structured function:
   - Is this function performing sparsification based on SI?
   - Neither the main text nor the appendix provides details about this function's implementation. Could you please elaborate on its mechanism?

- The term "OBC" in Algorithm 1 requires clarification:
   - While BiLLM mentions this as an abbreviation from another work, it would be beneficial to provide the full reference and explanation for completeness.

- Regarding computational requirements:
   - Could you provide an estimate for the computational time required for the 65B model, perhaps through theoretical scaling analysis?

- In Figure 3, there appears to be an overlap between Salient Weight and Non-salient Weight distributions:
   - Could you explain the underlying reasons for this overlap?
   - How does this overlap affect the overall performance of the method?

- Concerning Tables 5 and 7:
   - There seems to be redundancy as Table 5 appears to be a subset of Table 7's Wikitext2 results. Could you justify the inclusion of both tables?
   - The manuscript lacks discussion of Table 7's results, particularly regarding:
     * Why does SI perform worse than SparseGPT on PTB and C4 datasets?
     * What factors contribute to the different performance patterns across datasets?
     * Could you provide insights into these performance variations?
soundness: 3
strengths: - This approach integrates sparsity with quantization, achieving a significantly higher compression ratio by reducing both the number of active weights and the bit precision required to represent them. The sparsity aspect not only reduces storage needs but also opens up additional acceleration opportunities, as sparse models can skip unnecessary computations, leading to more efficient inference.
- A dedicated CUDA kernel was developed to optimize the performance of the sparse and quantized model on GPU hardware. This kernel was specifically tailored to exploit the structure of the sparse and binarized weights, enabling efficient memory access patterns and computation. The actual runtime performance of the model was measured using this custom kernel, providing a practical assessment of speedup gains achieved through the combined compression and acceleration strategy.
summary: This paper presents a sparse and binarized compression method for large language models (LLMs), achieving an average bit count of less than one bit. Specifically, in terms of sparsity, a new metric matrix is proposed to represent the importance of different weights, along with a method for calculating the sparsity level for each layer based on this metric. This allows for effective sparsification of the model weights. For quantization, weights are grouped and binarized within each group, thereby reducing quantization error. Experiments on models such as LLaMA-1/2/3 demonstrate that this method achieves superior performance at higher compression ratios.
weaknesses: - While the proposed quantization methodology shows promise, the performance improvements over the baseline BiLLM implementation appear to be incremental. I would encourage the authors to further highlight the distinctive advantages of their approach and potentially explore additional optimization strategies to achieve more substantial gains.
- The manuscript would benefit from enhanced clarity in several sections. Of particular importance is the need for a more comprehensive explanation of the average bit count calculation methodology. I suggest:
  - Including a detailed step-by-step breakdown of the calculation process
  - Providing specific examples to illustrate the computational procedure at inference time
  - Clarifying how this calculation relates to the overall system performance on speedup or memory reduction

Meta-review: The paper introduces STBLLM, a new method for compressing large language models (LLMs) to less than 1-bit precision, i.e. the paper's main claim. The main strength of the paper lies in its structured binarisation method, which employs N:M sparsity and fine-grained weight quantisation to achieve sub-1-bit compression. This method allows for more efficient storage and computation compared to previous techniques like BiLLM and PB-LLM. One of the main contributions is the Standardised Importance (SI) metric, which estimates weight importance without the need for computationally expensive Hessian-based methods. By considering weight magnitude and input feature norm, the SI metric allows for more effective weight pruning and sparsification.

Another strength is the layer-wise adaptive binarisation, which enables different layers of the LLM to have varying N:M sparsity ratios. This approach achieves a better balance between compression and model accuracy. Weights are further divided into sparse, intermediate, and dense regions, with each region undergoing a unique quantisation scheme. This fine-grained approach ensures that critical weights are preserved while less important weights are aggressively compressed.

The paper also demonstrates practical efficiency improvements through the use of a specialised CUDA kernel for structured binarisation, which is optimised for NVIDIA's Ampere sparse tensor cores. This optimisation results in a 17.85x speedup compared to existing 2-bit implementations. Some empirical validation on LLaMA (1, 2, and 3), OPT, and Mistral models confirms the method's claims. Across zero-shot, perplexity, and efficiency metrics, STBLLM consistently outperforms other methods like BiLLM and PB-LLM, achieving superior compression with minimal impact on model performance.

Despite its strengths, the paper has several limitations, as highlighted by the reviewers. One of them is that STBLLM does not support Mixture of Experts (MoE) models or Mamba-based LLMs, which limits its scope; however, it is to be expected in my opinion. Additionally, the approach introduces considerable complexity due to its reliance on multiple stages of partitioning, trisection, and residual binarisation. 

Another potential limitation is that while the method demonstrates improved performance on perplexity and zero-shot evaluations, the impact on real-world downstream tasks (like question answering or multi-step reasoning) is not discussed in detail. This raises questions about the generalisation of STBLLM in practical applications.

Finally, while the authors highlight the robustness of the approach to random weight flipping, the paper shows that performance fluctuations occur as the flipping ratio increases. While the approach appears stable in controlled experiments, it is unclear how robust it would be in real-world scenarios where noise or hardware faults might cause weight flips. Furthermore, while STBLLM claims to “break the 1-bit barrier,” the paper does not adequately stress the theoretical limits of this approach.

Having said all the above, I do believe the paper has been improved since its original submission. I would encourage the authors to add any remaining revisions to the paper based on the rebuttal for the camera ready version.

Decision: Accept (Poster)