STBLLM: Breaking the 1-Bit Barrier with Structured Binary LLMs

Authors: Peijie Dong, Lujun Li, Yuedong Zhong, DaYou Du, Ruibo FAN, Yuhan Chen, Zhenheng Tang, Qiang Wang, Wei Xue, Yike Guo, Xiaowen Chu
Primary area: foundation or frontier models, including LLMs
Keywords: structured sparsification, language model, model compression, binary neural networks, computational efficiency

Abstract:
In this paper, we present the first structural binarization method for LLM compression to less than 1-bit precision. Although LLMs have achieved remarkable performance, their memory-bound nature during the inference stage hinders the adoption of resource-constrained devices. Reducing weights to 1-bit precision through binarization substantially enhances computational efficiency. We observe that randomly flipping some weights in binarized LLMs does not significantly degrade the model's performance, suggesting the potential for further compression. To exploit this, our STBLLM employs an N:M sparsity technique to achieve structural binarization of the weights. Specifically, we introduce a novel Standardized Importance (SI) metric, which considers weight magnitude and input feature norm to more accurately assess weight significance. Then, we propose a layer-wise approach, allowing different layers of the LLM to be sparsified with varying N:M ratios, thereby balancing compression and accuracy. Furthermore, we implement a fine-grained grouping strategy for less important weights, applying distinct quantization schemes to sparse, intermediate, and dense regions. Finally, we design a specialized CUDA kernel to support structural binarization. We conduct extensive experiments on LLaMA, OPT, and Mistral family. STBLLM achieves a perplexity of 11.07 at 0.55 bits per weight, outperforming the BiLLM by 3×. The results demonstrate that our approach performs better than other compressed binarization LLM methods while significantly reducing memory requirements. Code is released at https://github.com/pprp/STBLLM.

Decision: accept (poster)
Reviewer scores: 6, 5, 8, 5

Reviews and discussion:
<thread>
creation date:
1730034256143

thread_type:
official_review

<thread_object>
creator: reviewer_yy1t

<content>
summary:
This paper presents a sparse and binarized compression method for large language models (LLMs), achieving an average bit count of less than one bit. Specifically, in terms of sparsity, a new metric matrix is proposed to represent the importance of different weights, along with a method for calculating the sparsity level for each layer based on this metric. This allows for effective sparsification of the model weights. For quantization, weights are grouped and binarized within each group, thereby reducing quantization error. Experiments on models such as LLaMA-1/2/3 demonstrate that this method achieves superior performance at higher compression ratios.

strengths:
- This approach integrates sparsity with quantization, achieving a significantly higher compression ratio by reducing both the number of active weights and the bit precision required to represent them. The sparsity aspect not only reduces storage needs but also opens up additional acceleration opportunities, as sparse models can skip unnecessary computations, leading to more efficient inference.
- A dedicated CUDA kernel was developed to optimize the performance of the sparse and quantized model on GPU hardware. This kernel was specifically tailored to exploit the structure of the sparse and binarized weights, enabling efficient memory access patterns and computation. The actual runtime performance of the model was measured using this custom kernel, providing a practical assessment of speedup gains achieved through the combined compression and acceleration strategy.

weaknesses:
- While the proposed quantization methodology shows promise, the performance improvements over the baseline BiLLM implementation appear to be incremental. I would encourage the authors to further highlight the distinctive advantages of their approach and potentially explore additional optimization strategies to achieve more substantial gains.
- The manuscript would benefit from enhanced clarity in several sections. Of particular importance is the need for a more comprehensive explanation of the average bit count calculation methodology. I suggest:
  - Including a detailed step-by-step breakdown of the calculation process
  - Providing specific examples to illustrate the computational procedure at inference time
  - Clarifying how this calculation relates to the overall system performance on speedup or memory reduction

questions:
- Regarding the calculation of average bit count:
   - Could you clarify whether the overhead from indices associated with sparsity has been factored into the calculation?
   - It would be helpful if you could provide a concrete example illustrating the calculation methodology, including both the weight bits and any additional storage requirements.

- In Algorithm 1, there appears to be some ambiguity regarding the Semi-Structured function:
   - Is this function performing sparsification based on SI?
   - Neither the main text nor the appendix provides details about this function's implementation. Could you please elaborate on its mechanism?

- The term "OBC" in Algorithm 1 requires clarification:
   - While BiLLM mentions this as an abbreviation from another work, it would be beneficial to provide the full reference and explanation for completeness.

- Regarding computational requirements:
   - Could you provide an estimate for the computational time required for the 65B model, perhaps through theoretical scaling analysis?

- In Figure 3, there appears to be an overlap between Salient Weight and Non-salient Weight distributions:
   - Could you explain the underlying reasons for this overlap?
   - How does this overlap affect the overall performance of the method?

- Concerning Tables 5 and 7:
   - There seems to be redundancy as Table 5 appears to be a subset of Table 7's Wikitext2 results. Could you justify the inclusion of both tables?
   - The manuscript lacks discussion of Table 7's results, particularly regarding:
     * Why does SI perform worse than SparseGPT on PTB and C4 datasets?
     * What factors contribute to the different performance patterns across datasets?
     * Could you provide insights into these performance variations?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730527443812

thread_type:
official_review

<thread_object>
creator: reviewer_1cy6

<content>
summary:
This paper proposes an efficient framework for LLMs, combining pruning and binarization to compress large, post-trained models. By applying N:M sparsity, it achieves precision below 1-bit and identifies salient weights through a newly introduced Standardized Importance (SI) metric. This metric considers both weight and activation values, avoiding the costly second-order computations typically required. Additionally, during pruning and binarization, the method separates non-salient weights into three groups to preserve as much information as possible in these parts. Extensive experiments demonstrate that the proposed method significantly reduces computational costs, accelerates inference, and maintains strong performance.

strengths:
+ The paper is well-organized and easy to follow, with a clearly stated problem.
+ It introduces a new metric to assess weight importance, avoiding expensive second-order gradient computations and mitigating the impact of extreme values.
+ It is interesting that separate binarization for non-salient weights retains crucial information in this segment, enhancing model performance.
+ The approach is logical and rigorous, discussing the method from various perspectives and fully validating its effectiveness through comprehensive experiments.

weaknesses:
- In the zero-shot experiment, the paper mentions seven zero-shot tasks. It would be helpful to include a brief description of each task to provide readers with a clearer understanding of the evaluation scope.

questions:
+ Regarding Figure 3, part (b), after structured pruning, the empty parts should have no values. Why are zeros assigned to these parts? Additionally, structured pruning usually doesn't achieve weight-wise pruning, so what does "structured" mean in this context?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730533697479

thread_type:
official_review

<thread_object>
creator: reviewer_kgjf

<content>
summary:
This paper proposes a structured binary quantization method to accelerate LLM inference. It combines n:m pruning and binary quantization, compressing the model weights to an average of less than 1 bit. In n:m pruning, the authors introduce an SI method for indentifying significant weights, and a layer-wise dynamic n:m allocation method. In binary quantization, the authors partition the weights into salient and non-salient parts for separate processing and further apply a group-wise quantization method to the non-salient part. Experimental results demonstrate that STBLLM outperforms BiLLM under the same bit budget. In addition, significant performance improvement (17x) is achieved with customized CUDA kernels.

strengths:
+ 1-bit weight quantization is important for accelerating LLM inference.
+ Dedicated CUDA implementations for the proposed method.

weaknesses:
+ incremental novelty

While the proposed method is interesting and performs better than BiLLM, its novelty is limited: 1) The proposed SI method is very similar to Wanda, with the main difference being the introduction of additional data normalization. 2) The binary quantization method is quite similar to BiLLM, where the hessian matrix is used to divide weights into salient and non-salient parts, and residual approximation is employed to handle the salient part. The only difference is that STBLLM processes the non-salient weights into three parts instead of two as in BiLLM.

+ mismatch between motivation and methodology

The motivation of this paper lies in the observation that some weights in binary LLMs do not significantly affect accuracy and can be further compressed (Section 3.1 and fig 1). Under this narrative, a reasonable approach would be to perform pruning on the binarized model to achieve further compression. In contrast, the method proposed in this paper adopts a ‘’prune-then-quantize’’ approach, which does not align with the motivation. The paper does not explain why pruning should be performed first and does not discuss how changing the order of pruning and binarization might affect the results. 

The motivation behind using a trisection-based partition for non-salient weights is confusing. It seems the authors aim to balance bits and performance (Section 3.4). However, the evaluation results show that the improved compression ratio and performance are due to n:m pruning, rather than the processing of non-salient weights. So, why should we partition the non-salient weights into three parts? Why not four or five? What do the terms dense, intermediate, and sparse mean?

+  confusing evaluations

While the experimental results of STBLLM are promising, the source of the accuracy improvements remains unclear. The experimental settings in the ablation study are somewhat confusing. For instance, Table 5 examines the effectiveness of the SI method in n:m pruning, but the results seem to represent 4:8 pruning plus binarization. What binarization method is used in the baselines? Table 8 directly compares STBLLM with BiLLM to illustrate the effectiveness of trisection partitioning, yet the pruning methods used in STBLLM and BiLLM are not the same (SI vs. Wanda). A detailed, step-by-step breakdown analysis of each technique's effectiveness would be helpful. Moreover, where does the 17x performance improvement come from when reducing 2-bit weights to 1 bit?

questions:
Please see the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730924294981

thread_type:
official_review

<thread_object>
creator: reviewer_jqjp

<content>
summary:
This work presents a structural binarization method for LLMs by combining N:M sparsity, residual approximation, and block-wise error compensation. Extensive experiments on LLaMA-1/2/3, OPT, and Mistral are conducted to evaluate the effectiveness of STBLLM. In addition, a specialized CUDA kernel is designed to support structural binarization.

strengths:
* The analysis on flipping non-salient binarized weights is intriguing. I am wondering what would happen if we increase the ratio from 0.15 to 0.5?
* The proposed method achieves the lowest perplexity among all compared methods in the sub-1-bit regime.
* A specialized CUDA kernel for structural binarization, leveraging NVIDIA's Ampere GPU sparse tensor cores, achieves a 17.85x speedup over ABQ-LLM's 2-bit implementation.

weaknesses:
* The proposed method is a combination of several existing techniques including N:M sparsity, residual approximation, block-wise error compensation, and Trisection search (for the non-salient part). This raises some novelty concerns. I suggest the authors to 1) highlight the main novelty and contribution of the current submission; 2) provide ablation studies on a. how important the residual approximation is, b. the impact of Trisection search for grouping and why there are two groups.
In addition, which techniques contribute the most to efficiency and which method contributes the most to the accuracy?
* The benchmark results are based on various N:M configurations. However, NVIDIA GPUs mainly support 2:4. The authors may discuss how practical the proposed method is on NVIDIA GPUs.

questions:
See weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1732025370270

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Thanks to all Reviewers for Recognition and Constructive Comments

comment:
Dear Reviewers, Area Chairs, and Program Chairs,

We sincerely thank all reviewers for their positive feedback and constructive comments. The reviewers have acknowledged the novelty, impact, superior performance, and efficient implementation of our method. Below, we summarize the strengths of our paper as highlighted by the reviewers:

**[Novelty & Methodology]:**

- **Reviewer jQJp**: The work presents a structural binarization method for LLMs by combining N:M sparsity, residual approximation, and block-wise error compensation. The analysis on flipping non-salient binarized weights is intriguing.
- **Reviewer KgJf**: The paper proposes a structured binary quantization method to accelerate LLM inference, combining N:M pruning and binary quantization, and introducing a new SI method for identifying significant weights.
- **Reviewer 1cy6**: The framework combines pruning and binarization to compress large, post-trained models, introducing a new Standardized Importance (SI) metric that avoids costly second-order computations.
- **Reviewer yy1T**: A new metric matrix is proposed to represent the importance of different weights, allowing for effective sparsification and quantization.

**[Impactful & Performance]:**

- **Reviewer jQJp**: The proposed method achieves the lowest perplexity among all compared methods in the sub-1-bit regime.
- **Reviewer KgJf**: Experimental results demonstrate that STBLLM outperforms BiLLM under the same bit budget, achieving significant performance improvement.
- **Reviewer 1cy6**: The method significantly reduces computational costs, accelerates inference, and maintains strong performance.
- **Reviewer yy1T**: The method achieves superior performance at higher compression ratios, integrating sparsity with quantization for efficient inference.

**[Efficient Implementation]:**

- **Reviewer jQJp**: A specialized CUDA kernel is designed to support structural binarization, achieving a 17.85x speedup over ABQ-LLM's 2-bit implementation.
- **Reviewer KgJf**: Dedicated CUDA implementations for the proposed method result in significant performance improvement.
- **Reviewer yy1T**: A dedicated CUDA kernel was developed to optimize the performance of the sparse and quantized model on GPU hardware, enabling efficient memory access patterns and computation.

**[Good Presentation]:**

- **Reviewer 1cy6**: The paper is well-organized and easy to follow, with a clearly stated problem. The approach is logical and rigorous, fully validating its effectiveness through comprehensive experiments.

In the past few days, we have conducted **additional experiments, clarified points, and engaged in discussions** to address the valuable comments provided by the reviewers. Based on the constructive feedback, we have **carefully revised the manuscript of our work using yellow highlights**. We hope our detailed responses can alleviate the concerns.

Best regards and thanks,

Authors of #66
</content>
</thread_object>

</thread>
<thread>
creation date:
1732526492882

thread_type:
official_comment

<thread_object>
creator: area_chair_bhn3

<content>
title:
Please engage in discussions

comment:
Dear all,

Many thanks to the reviewers for their constructive reviews and the authors for their detailed responses.
Please use the next ~2 days to discuss any remaining queries as the discussion period is about to close.

thank you.

Regards,

AC
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces STBLLM, a new method for compressing large language models (LLMs) to less than 1-bit precision, i.e. the paper's main claim. The main strength of the paper lies in its structured binarisation method, which employs N:M sparsity and fine-grained weight quantisation to achieve sub-1-bit compression. This method allows for more efficient storage and computation compared to previous techniques like BiLLM and PB-LLM. One of the main contributions is the Standardised Importance (SI) metric, which estimates weight importance without the need for computationally expensive Hessian-based methods. By considering weight magnitude and input feature norm, the SI metric allows for more effective weight pruning and sparsification.

Another strength is the layer-wise adaptive binarisation, which enables different layers of the LLM to have varying N:M sparsity ratios. This approach achieves a better balance between compression and model accuracy. Weights are further divided into sparse, intermediate, and dense regions, with each region undergoing a unique quantisation scheme. This fine-grained approach ensures that critical weights are preserved while less important weights are aggressively compressed.

The paper also demonstrates practical efficiency improvements through the use of a specialised CUDA kernel for structured binarisation, which is optimised for NVIDIA's Ampere sparse tensor cores. This optimisation results in a 17.85x speedup compared to existing 2-bit implementations. Some empirical validation on LLaMA (1, 2, and 3), OPT, and Mistral models confirms the method's claims. Across zero-shot, perplexity, and efficiency metrics, STBLLM consistently outperforms other methods like BiLLM and PB-LLM, achieving superior compression with minimal impact on model performance.

Despite its strengths, the paper has several limitations, as highlighted by the reviewers. One of them is that STBLLM does not support Mixture of Experts (MoE) models or Mamba-based LLMs, which limits its scope; however, it is to be expected in my opinion. Additionally, the approach introduces considerable complexity due to its reliance on multiple stages of partitioning, trisection, and residual binarisation. 

Another potential limitation is that while the method demonstrates improved performance on perplexity and zero-shot evaluations, the impact on real-world downstream tasks (like question answering or multi-step reasoning) is not discussed in detail. This raises questions about the generalisation of STBLLM in practical applications.

Finally, while the authors highlight the robustness of the approach to random weight flipping, the paper shows that performance fluctuations occur as the flipping ratio increases. While the approach appears stable in controlled experiments, it is unclear how robust it would be in real-world scenarios where noise or hardware faults might cause weight flips. Furthermore, while STBLLM claims to “break the 1-bit barrier,” the paper does not adequately stress the theoretical limits of this approach.

Having said all the above, I do believe the paper has been improved since its original submission. I would encourage the authors to add any remaining revisions to the paper based on the rebuttal for the camera ready version.