STBLLM: Breaking the 1-Bit Barrier with Structured Binary LLMs
Authors: Peijie Dong
Lujun Li
Yuedong Zhong
DaYou Du
Ruibo FAN
Yuhan Chen
Zhenheng Tang
Qiang Wang
Wei Xue
Yike Guo
Xiaowen Chu
Venue: ICLR 2025 Poster (2025)
Keywords: structured sparsification
language model
model compression
binary neural networks
computational efficiency
Primary area: foundation or frontier models, including LLMs
TL;DR: We introduces STBLLM, a novel approach that breaks the 1-bit barrier in language models by leveraging Structured Binary LLMs.

Abstract: In this paper, we present the first structural binarization method for LLM compression to less than 1-bit precision. Although LLMs have achieved remarkable performance, their memory-bound nature during the inference stage hinders the adoption of resource-constrained devices. Reducing weights to 1-bit precision through binarization substantially enhances computational efficiency. We observe that randomly flipping some weights in binarized LLMs does not significantly degrade the model's performance, suggesting the potential for further compression. To exploit this, our STBLLM employs an N:M sparsity technique to achieve structural binarization of the weights. Specifically, we introduce a novel Standardized Importance (SI) metric, which considers weight magnitude and input feature norm to more accurately assess weight significance. Then, we propose a layer-wise approach, allowing different layers of the LLM to be sparsified with varying N:M ratios, thereby balancing compression and accuracy. Furthermore, we implement a fine-grained grouping strategy for less important weights, applying distinct quantization schemes to sparse, intermediate, and dense regions. Finally, we design a specialized CUDA kernel to support structural binarization. We conduct extensive experiments on LLaMA, OPT, and Mistral family. STBLLM achieves a perplexity of 11.07 at 0.55 bits per weight, outperforming the BiLLM by 3×. The results demonstrate that our approach performs better than other compressed binarization LLM methods while significantly reducing memory requirements. Code is released at https://github.com/pprp/STBLLM.

Published as a conference paper at ICLR 2025
STBLLM: BREAKING THE 1-BIT BARRIER WITH STRUCTURED BINARY LLMs
Peijie Dong 1 Lujun Li Yuedong Zhong Dayou Du', Ruibo Fan', Yuhan Chen', Zhenheng Tang?. Qiang Wang" Wei Xue", Yike Guo Xiaowen Chu 1.2 HKUST(GZZ) HKUST SYSU 4 HIT(SZZ)
{pdong212, ddu487, rfan404, ychen906feconneett www.oonnnnc.uust-ggge.e www..z..dd..ooooooo lilujunai@gme aeg.almal..................e.l.e com, synnyaaeeaa122.sss..cdu.... qiang. , wangehit en, (zhtang.ml, weixue, yikeguo)eust..o xwchu@hkust -ga cn en
ABSTRACT
In this paper, present the first structural binarization method for LLM com- pression to less than 1-bit precision. Although LLMs have achieved remarkable performance, their memory-bound nature during the inference stage hinders the adoption of snsource-constrainee devices. Reducing weights to 1-bit precision through binarization substantially enhances computational efficiency. We observe that randomly flipping some weights in binarized LLMs does not significantly degrade the model's performance, suggesting the potential for further compres- sion. To exploit this, our STBLLM employs N:M sparsity technique to achieve structural binarization of the weights. Specifically, we introduce a novel Standard- ized Importance (SI) metric, which considers weight magnitude and input feature norm to more accurately assess weight significance. Then, we propose a layer- wise approach, allowing different layers of the L.M be sparsified with varying N:M ratios, thereby balancing compression and accuracy. Furthermore, we imple- ment a fine-grained grouping strategy for less important weights, applying distinct quantization schemes to sparse, intermediate, and dense regions. Finally, we de- sign a specialized CUDA kernel to support structural binarization. We conduct extensive experiments on LLaMA, OPT, and Mistral family. STBLLM achieves a perplexity of 11.07 0.55 bits per weight, outperforming the BiLLM by 3x. The results demonstrate that our approach performs better than other compressed bina- rization LLM methods while significantly reducing memory requirements. Code is released https/ggihubb.co//pp///SBBLL
INTRODUCTION
The advent of large language models (LLMs), such (Zhang et al. 2022a Touvron al.) 2023a
Brown et al.) 2020), has revolutionized the field of natural language processing (NLP) (Wei et al, 20226). These powerful models exhibit remarkable performance, surpassing human capabilities in various domains (Wei et al.) 2022a Bubeck et al.] 2023). However, the immense scale and com- plexity of LLMs present significant challenges in terms of memory requirements, hindering their widespread deployment, especially in resource-constrained environments. To address this issue, model compression techniques, such as quantization (Frantar et al. 2023 Lin et al. 2024a Dong et al. 2023b), pruning (Meng et al.] 2020), distillation (Li) 2022 Li & Jin 2022, Xiaolong et al.) 2023 and low-rank decomposition (Ashkboos et al. 2024). have gained increasing attention in re- ducing the computational footprint while preserving their performance. One promising approach network binarization, the most aggressive quantization method. Binarization quantizes original floating-polt weights with binary values (-1 +1), significantly reduces memory storage.
Pioneering binarization methods (Rastegari et al.) 2016 Liu et al. 2018) present customized binary structures and training paradigms for binarized neural networks (BNNs) in vision tasks. Building
"Corresponding authors. Equal contribution.

---

Published as a conference paper at ICLR 2025
0.8
124.72
PB-LLM (Int8 10%)
152.31
0.7
10a
RTN
Perplexity in logho scale
GPTQ
0.6
BILLM
Accuracy
39.61
0.5
STBLLM (Ouns)
25.53
22,55
0.4
arc arc_easy
0.3
17.22
boolq
0.2
hellaswag
rte
10
5.68
5.68
6.28
11:110.73
10.73
9.66
11:33
0.1
5.68
5.68
6.19
0.000
0.025
0.050
0.075
0.100
0.125
0.150
Ratio
16
8
2
0.5
Quantization Bit-width
Figure 1: The impact of random flipping non- salient binarized weights on accuracy in a 1-Bit L.LMMA2-7B. The x-axis represents the percentage of binarized weights flipped from -1 to or vice versa. As the ratio increases, the ac- curacy does not decline significantly, indicating redundancy in the 1-bit representation.
Figure 2: The perplexity of LLaMA-1-13B on the Wikitext2 under different bit-widths. RTN and GPTQ (Frantar et al., 2023) show a drastic performance drop ultra-low bit- widths. Our STBLLM maintains low perplexity even at 0.55 bits compared with BiLLM.
upon these foundational approaches, subsequent methods (Wang et al. 2020 Liu al. 2022
Muna-
gala et al. 2020 have advanced the field by integrating sparse kernel techniques (Wang et al. 2023b
202Tb) and pruning methodologies (Wang et al.) 2021a & Ren 2020 Munagala et al.
2020).
For LLMs. inspired by the success of 4-bit and 8-bit quantization methods, some studies Huang
ctt al. 2024; Xu et al.) 2024; Shang et al., 2024) continue to explore ultra-low-bit or even 1-bit precision. For example, the post-traiming method PB-LLM (Shang al. 2024) partially binarizes LLMs with an optimal scaling factor strategy, preserving a small subset of the higher bit-precision weights. BiLLM (Huang al. 2024) proposes residual approximation strategy improve 1-bit LLMs. While these methods represent the most aggressive quantization approaches. it is crucial to consider that popular foating-point LLMs already contain model sizes ranging from billion 140 billion parameters. As a result, 1-bit LLMs still need to be further accelerated and optimized for many eesource-constrail
devices and real-time scenarios. This naturally raises a key question: Is there any compression method with less than 1-bit weight representation that can further push the quantization of LLMs?
For this question, there are two key observations: @ Not all weights contribute equally to the per- formance of 1-bit LLMss. As shown Figure performing random weight flipping for non-salient weights results in only minimal performance drop (For more details, refer to Appendix B). This finding indicates that even in highly quantized 1-bit LLMs, subset of redundant weights exists that can be compressed without impacting the performance. It suggests the potential for further com- pression by selectively encoding the most significant weights while discarding or compressing the less important ones. @ Structured sparsity techniques, such as N:M sparsity methods (Hubara jet al., [2021] ZZhang et al., (002b)) Zhou et al., (001), leverage the inherent structure and patterns the weight distribution, allowing for more efficient compression. These N:M sparsity methods have good hardeare-acceeerrate support in recent LLM pruning models (Frantar & Alistarh [023]]
Sun al. 2024 Dong al., 2024), enabling efficient deployment NVIDIA Ampere architec- ture (Nvidia 2020). However, traditional binarization techniques (Rastegari et al. (016) often treat weights as independent entities, failing exploit the inherent structure and patterns the weight matrices. These observations encourage us to explore N:M sparsity tailored specifically for 1-bit L.LMs achieve further speedups and compression gains.
Based on these observations, we develop our STBLLM approach, STructured Binarization for LLMs to achieve extreme compression while mitigating performance degradation. To our knowledge, STBLLM is the first to push LLM compression below 1-bit precision, opening new possibilities for edge deployment. Our workflow applies the metric-based sparsity and performs the adaptive N:M binarization. Specifically, to measure the importance of weights,
introduce Standard- ized Importance (SI) metric that addresses the issues of extreme weight values and computationally expensive Hessian- based methods used in prior work. We then propose adaptive layer-wise struc-
2

---

Published as a conference paper at ICLR 2025
Activation
Pretruined Weight
Scrvetured
Seruetured
Structured
Bonorised
Binarized Weight
Dinarized Welaht
Standardzzod
Nen-sabem Weight
MasMol
Importance
15
Vidac
Stracaeed Bizarized Weighe PC
(c) Non-salient Weight Partition
N-M Sinuttured
Activation
Prusing
Straskurd Weggit
Structured Binarized Weight FC
Activation
FFN
MHSA
Before Stundardiztionn
Standurdization
(a) Basic block STBLLM
(b) Structured Binarization
(d) Weight Standardization
Figure 3: (a) PTQ framework in Structured Binarized LLM (STBLLM). We apply structured bi- narization to all of the weights. (b) Structured Binarized Weight Computation Procedure. We first perform N:M structure pruning to pre-trained weight (here N=2, M=4), then show how we process the non-salient weight. (c) Trisection partition for Symmetric Gaussian Distribution of Non-salient Weight. (d) Illustration of Weight Standardization on LLaMA-2-7B.
tured binarization approach, where different layers of the LLM can be sparsified with varying N:M ratios balance compression and accuracy. We employ residual approximation technique (Huang al. 2024) for the salient parameters to preserve the critical information. For the non-salient pa- rameters, we utilize
grouping strategy based trisection search algorithm to find optimal splitting points p" and apply different quantization schemes to the sparse, intermediate, and dense regions presented in Figure 3(.). By tailoring these structured representations specifically for 1-bit LLMs, we unlock a new avenue for model compression and optimization, enabling more widespread deployment of these powerful LLMs in
environments.
To validate the effectiveness of STBLLM, we conduct extensive experiments on various LLMs, in- cluding the LLaMA-1/2/3 (Touvron et al. 2023a b), OPT (Zhang et al., 2022a) and Mistral (Jiang eet al. 2023). As presented in Figure 2 our STBLLM achieves a better trade-off between perfor- mance and bit-width. STBLLM with 0.8 bit can achieve lower perplexity than BiLLM with 1.1 bit. STBLLM achieves a perplexity of 31.72 at just 0.55 bits per weight, compared to 688.77 73 for BiLLM an over 20x gain. Even at 65B parameters, our 0.55-bit STBLLM outperforms BiLLM's 0.7-bit and PB-LLM's 1.7-bit versions. STBLLM retains significantly higher accuracy for zero-shot bench- marks than BiLLM under 4:8 and 6:8 structured binarization settings across 13B and 30B LLaMA. For example, on LLaMA-1-30B, our 0.55-bit STBLLM achieves 51.78% average accuracy versus just 43.72% for BiLLM. The contribution of our work is as follows:
We introduce STBLLM, novel structural binarization framework that compresses Large Language Models (LLMs) to less than 1-bit precision, enabling significant memory and computational savings while preserving model performance.
STBLLM employs N:M binary weight kernel approach, where we perform structural binarization of the weights using efficient gradient-fre free metrics to determine weight im- portance, channel rearrangement preserve salient weights, and adaptive layer mixed- structure binarization for better accuracy-efficiency trade-offs.
We implement a specialized CUDA kernel for structural binarization, leveraging NVIDIA's Ampere GPU sparse tensor cores, achieving a 17.85x speedup over ABQ-LLM's 2-bit implementation.
Extensive experiments LLMs like LLaMA and OPT show that STBLLM outperforms other compressed binarization methods, achieving a perplexity 11.07 at 0.55 bits per weight-3x better than BiL.M.
3

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
Quantization and Binarization. Quantization reduces full-precision parameters to lower bits, thereby decreasing storage and computation requirements. Recent research has effectively applied Quantization-Awan Training (QAT) and Post-Training Quantization (PTQ) to LLMs. QAT (Liu ct al. 2024 Du et al. 2024| Chen et al. 2024) incorporates quantization during training, allowing to learn better representations for low- bit weights. However, due to the massive parameters, retraining is too costly and inefficient for LLMs. In contrast, PTQ (Frantar et al., 2023] Chee et al., 2023] Lin
et al. 2024b; Lee et al., 2024: Dong al. 2023b) can directly quantize pre-trane traitrained models with- out additional training. GPTQ (Frantar et al. 2023) and QulP (Chee et al. 2023) minimized LLM block quantization errors through second-order enror compensation. Other approaches (Lin al., 2024b Lee et al. 2024| Dettmers et al. 2023) focus on prioritizing salient weights to maintain their information representation capacity. Binarization, which constrains quantized parameters to a 1-bit representation, is the most extreme quantization method and has proven effective for vision tasks, such XNOR-Net (Rastegari al. 2016) and Bi-Real Net (Liu et al. 2018). To further compress binary neural networks, sparse kernel techniques (Wang et al. 2020: 2021bbg2 a Liu et al.) (2022) are in- troduced to reduce the redundancy in binary neural networks. For I.LM binarization, BitNet (Wang et al. (023a) trained 1 1itt L.M from scratch. OneBit (Xu et al., 2024) employ the QAT paradigm for 1-bit LLM while BiLLM (Huang et al., 2024) employ the PTQ paradigm with residual approxi- mation technique. In this paper, we further reduce weights below 1-bit by identifying and removing the redundant parameters.
Sparsity Methods for LLM. Pruning removes less important parameters from neural network to reduce its size and improve efficiency. For LLMs, Pruning can be divided to structured prun-
ing (Ma et al. 2023] Ashkboos et al. 2024 Xia al. 2024 An et al., 2023), semi-structured pruning (Frantar & Alistarh| [003 Sun et al. 2024 /Zhang et al. 2024b) and unstructured prun-
ing (Li et al. 2024b d Dong et al. 2024). Structured pruning methods, including LLM-Pruner (Mal iet al. 2023) and Sheared LLaMA Xia et al. 2024), aim to simplify LLM by removing specific components such as heads, layers, and dimensions. Although these techniques enhance model effi- ciency, they often result in significant performance degradation and require extensive retraining to recover lost capabilities. In contrast, unstructured pruning methods (Frantar & Alistarh [203]] Sun
et al. 2024) remove individual weights based on their significance within the model. However, this approach leads to irregular sparsity patterns that not effectively leverage hardware acceleration. Semi-structured pruning offers more balanced approach to model optimization. Methods such as GPT (Frantar& & Alistarh| (023) and Wanda (Sun et al. 2024 exemplify this strategy by maintaining regular, hardware- furdware-friendly sparsity patterns, such as N:M sparsity. This approach com- bines the fine-grained control characteristic of unstructured pruning with the operational efficiency associated with structured pruning.
Synergy of Pruning and Quantization. Research highlights the complementary strengths of prun- ing and quantization. Pruning trims neural network parameters, while quantization compresses their precision. Deep Compression (Han et al.) (016) integrates pruning. trained quantization, and Huff- man coding into streamlined framework, significantly lowering the storage footprint of deep neural networks (DNNs). Later works (Tung & Mori 2018 Yang et al. 2019; Hu et al.] (202]) develop in-parallel mruiing-quantizen
methods to optimize compression allocation, such as unstructured pruning sparsity and quantization strategies. For extreme cases like binarization, approaches like STQ-Nets (Munagala et al., 2020) (structured pruning for CNNs), BNN Pruning (Li & Ren 2020) (weight flipping-based pruning), and BAP (Wang et al.] 202 la) (98% sparsity via sparse convolu- tion) achieve significant compression and speedup. However, their reliance on fine-tuning limits applicability to LLMs.
METHODOLOGY
In this section, we introduce our STBLLM framework, as depicted in Figure 3 We employ struc- tured binarization for all weights within the Feed-forward Network (FFN) and Multi-head Self- attention (MHSA) modules. Specifically, we introduce the concept of Standardized Importance (SI) to evaluate the saliency of each weight under N:M sparsity constraints (refer the left part of Fig- 3re33b)), We leverage the Hessian matrix to distinguish between salient and non-salient weights for the binarization process. Salient weights are handled using residual approximation, following the
4

---

Published as a conference paper at ICLR 2025
Algorithm Framework of STBLLM: Details of each function are shown in Algorithm 2
1: function STRUCTURED
X, B. X)
2: Input: W E IR denotes weight matrix; X Rrxd represents calibration data;
3: 3 denotes block size; À represents hessian regularizer
4: Output: B structured binarized weights
5: H. + 2XX
la error hessian matrix
6:
Hº < Cholesky
+II XI)
7:
B < 0nxm
8:
for = 0, 3, 28, .......
........... N do W Standardized Importance W (bb+a)
9:
W Semi-Structured Wet
9.bb++8: W (.bbba)
row, { - Salient(Ve W 1,bbb H
B, + ResrApprox W
AE (row.))
13:
Pi.p2 - AonSalientAAareeuuaall Ws (row
14:
B2, B3, B4 +- Trisection W |w -
15:
B. B.b +- B1 U UB2 B3 B4
16:
E + dbbb++BB B, B..b+++))))) H 8,6986
17:
W,,++: < W.6+8: E H b.b+
b block-wise OBC
18:
end for
19:
return B
20:
end function
methodology outlined in BiLLM (Huang et al., 2024). For non-salient weights, we propose a Non- salient Aware Quantization technique, which further divides these weights into Dense, Intermediate, and Sparse regions (as shown in the right part of Figure Bjc). To optimally partition the non-salient weights into three distinct regions, we utilize a trisection search strategy to determine the appropriate pi and pa values. In the subsequent update step, we apply block-wise error compensation (Frantar & Alistarh 2023 Frantar et al., 2023) to preserve performance following post-training quantization (PTQ), Algorithm I Isummarizes the STBLLM process, with implementation details in Appendix A Building upon BiLLM (Huang et al. 2024), we refine the SI metric, adopt semi-structured adaptive pruning. and implement Non-Salient aware quantization to distinguish our approach.
3.1 PRELIMINARIES
Binarization quantizes floating-poit (FP) weights, represented as WEP. into 1-bit values (i.e., +1). During forward propagation, the sign function is used binarize the original parameter tensor:
B sign W Wrr),
sign (w) = if 2 0,
(2)
others,
where W Fp Raxm is the 32-bit floating-point weight, and B E Raxm the binarized output, and a = |Will, The parameter n and m represent the size of the weight matrix. The scaling factor R" is applied in channel-wise manner (Rastegari al. 2016).
N:M Sparsity, Inspired by the experiments in Figure I we observe the binarized the redundancy LLMs. By applying the N:M binarization for LLMs, we can achieve an extreme compression ratio of less than 1 bit. Specifically, we introduce N:M sparsity that encodes N consecutive non- zero elements the weight matrix using a M-bit representation. While it enhances acceleration. compromises performance. To alleviate this problem, we propose several techniques from different perspectives: S Importance Measurement. Previous methods (Frantar al. 2023 Chen et al.) 2024 Huang al. 2024) utilize Hessian-baa based methods to measure the importance, but they computationally expensive and can not reflect the true importance of parameters in LLMs. 2 Layer- wise Assignment. Previous PTP methods (Frantar& & Alistarh 2023 Zhang et al. 2024b) utilize the uniform sparsity ratio among different layers. However, recently, evidence (Yin et al. (024) shows that not all layers have the redundant level thus non-uniform sampling help retain the
5

---

Published as a conference paper at ICLR 2025
performance. 3 Hierarchical Quantization. Prior PTQ techniques for LLMs (Lin al. 2024b Lee et al., 2024 Huang et al. 2024) separate weights into salient and non-salient categories via activation magnitude or Hessian matrix, focusing salient weights for their role in performance. Here, we reevaluate non-salient parameters, underscoring their vital influence on quantization.
3.2 STANDARDIZED IMPORTANCE METRIC
Many previous works, such DB-LLM CCenn et al. 2024). SparseGPT (Frantar & Alistarh 2023)
GPTQ (Frantar et al. (023). and BiLLM (Huang et 2024), utilize the Hessian metric measure the importance of weights. However, we observe that extreme values in the weights significantly impact Hessian computation (See Appendix D). To address this issue, we present a Standardized Importance (SI) metric. The computation of ST does not involve the second-order information of the weights, which can be computationally expensive for LLMs. Specifically, tion to mitigate the issue of extreme values weights by transforming the weights to have a mean of zero and standard deviation of one. This process ensures that all weights are on a similar scale, reducing the disproportionate influence of extreme values on the Hessian matrix. For a linear layer with weight W E R= which takes in imput activation X E Rxxd where r is the batch size and d m is the input dimension. We evaluate the importance of each weight by the product of its magnitude and the corresponding input feature norm. The score for the W., is defined as:
employ standardiza-
S., = [((((W مll,,
a(w)
w
HW
a (
W + Σ. W Σ3 W.,l
W
33)
ow
where (-) standardizes the weight magnitude y W c,l) using the mean jow and standard devi- ation ow of all weights in the layer. The weight magnitude ((il),) is computed as the sum of the Ll-normalized magnitude across the input dimension and the output dimension The input feature norm ||,, is calculated as the L2 norm of the j-th column input activation X. By mul- tiplying the standardized weight magnitude (H([W.,))) with the input feature norm [X.rll2, the importance score S,, takes into account both the significance of the weight itself and the activation level of the associated imput feature. To prune the linear layer, we rank all the weights based on their importance scores S.j and remove specified percentage of the weights with the lowest scores. This pruning strategy aims to preserve the most significant weights contributing to the layer's output while climinating less important weights to reduce the model's size and computational complexity.
3.3 ADAPTIVE LAYER-WISH BINARIZATION
N:M Binary Weight Vector.
To achieve compression beyond standard binarization, we propose an N .M sparsity approach, where M binary values are represented by N values (N < M). This allows for further compression while preserving the salient information in the weight tensors. Specifically, we employ the mixed N:8 sparsity configuration following DominoSearch (Sun et al. 2021).
Layer-wise N:M Assignment. To achieve hetter acffracy-efficiency trade-offs, we introduce adap- tive layer-wise structured binarization, where different layers of the LLM can sparsified with different N:M ratios. (For example, with a target ratio of 4:8, layers can have ratios like 3:8, 4:8, and 5:8 while maintaining the overall 4:8 ratio.) This flexibility allows for more aggressive com- pression in less important layers while preserving higher precision in crucial layers. The layer-wise N:M ratios are assigned based on the relative importance of each layer, measured by the L2 norm of its weight parameters. Let W, and Whotal be the L2 norm of layer i and the sum across all lay- ers, respectively. The relative importance (; of layer is Or; We The N:M ratio for layer is
M, a, (1 (1-a;) az) Rearget where Runget is the target overall compression ratio. More important layers have higher N:M ratios (less sparsification), approaching 1:1 for the most important ones. Less important layers have lower N:M ratios, approaching Raaget for the least important ones. This ensures the overall compression ratio meets Runget-
3.4 NON-SALIENT AWARE QUANTIZATION
Based on the observations that small fraction of salient weights is critical to the LLM quantiza- tion (Lin et al., 2024b, Shao et al., 2023). we split the weights into the salient and non-salient parts and then apply a higher bit for salient one and lower-bit for non-salient one,

---

Published as a conference paper at ICLR 2025
Table I: Average bit results from structural searching and residual binarization of OPT, LLaMA-1, and LLaMA-2 families. "Results for largest models: OPT-66B, LLaMA-1-65B, LLaMA-2-70B.
Model
BiLLM
BiLLM-4:8
BiLLM-S:8
BiLLM-6:8
138
308
65-708"
7B
138
308
65-7088
78
138
30B
65-708"
7B
308
65-708"
OPT
1.10
1.12
1.12
1.13
0.55
0.56
0.56
0.56
0.69
0.70
0.70
0.71
0.83
0.84
0.84
0.85
LLaMA-1
1.09
1.09
1,10
1100
0.54
0.54
0.55
0.55
0.68
0.68
0.69
0.69
0.82
0.82
0.83
0.83
LLaMA-2
1.08
N/A
0.54
NIA
0.54
N/A
0.68
0.80
N6A
0.82
Table 2: Perplexity comparison of PB-LLM and BiLLM on the LLaMA model family. The columns represent the perplexity results on the Wikitext2 for different model sizes. The average bit-width for each model is provided in the table. For more precise bit-width results, please refer to Table [
Settings
LLaMA-1
LLaMA-2
LLaMA-3
Method
Block Size
W-Bits
7B
L3B
30B
65B
13B
8B
FullPrecision
16
5.68
5.09
4.1
3,53
5.47
4.88
6.10
RTN
1.7e5
1,4e6
1.5e4
6,5e4
1.6e5
4,8e4
2.7e6
GPTQ
128
2.7e5
1.1e5
6.7e4
2,5e4
1.2e5
9,4e3
5.7e4
PB-LIM
128
1.7
102.36
36.6
33.67
12.53
69.2
151.09
41.80
BiLLM
128
1,09
35.04
15.14
10.52
8.49
32.48
16.77
28.30
BiLLM
128
0.80 (6:8)
80.36
22.55
13.22
9.09
50,25
27.28
94.15
BiLLM
128
0.70 (5:8)
126.99
39.61
18.69
11.57
87.84
58.14
161.48
BiLLM
128
0.55 (4:8)
688.73
124.72
37.96
29.22
263.61
124.78
663,91
STBLLM
128
0.80 (6:8)
15.03
9.66
7.56
6.43
13.06
11.67
33.44
STBLLM
128
0.70 (5:8)
19.48
11.33
9.19
7.91
18.74
13.26
49.12
STBLLM
128
0.55 (4:8)
31.72
17.22
13.43
11.07
27.93
20.57
253.76
Salient Part: In our cases, for salient weight, we apply residual approximation (Huang et al. 2024). which is composed of residual approximation weight, as follows:
o B arg millo,,,, B, || IW- a.,.,ll². ar B arg mino,... B, (W a) a, 1,13
(4)
where B, denotes the original binary tensor, and B, represent the residual binarized matrix as the compensation. The final approximation of W is W s a, + B,
Non-Salient Part: For the non-salient part (which is also symmetric Gaussian distribution), we find that significant information is retained in the non-salient part. To make the trade-off with bit and performance, we utilize a fine-grained grouping strategy called the Trisection search algorithm (See Algorithm 2) divide-and-com -nd-cuuuee method find optimal split points pi and p5 for weight regions. With these two break-points, we can segment the symmetric Gaussian distribution into three groups, which sparse Rs [R[-m, - p5] U [pt, m]. intermediate Rs -ril Ral-pi. Pi: pi]. Then, we derive the quantization error:
[pt pt). and dense region
0 P5.P5 = WW 0g C,B,I.
W, B.l. + || 3,13
(5)
1 | wll a. = nd
023 W. Wallu, a;
Walln
(6)
where W,, W. Wa are the sums of absolute weight values in the sparse, intermediate, and dense regions. B,, B; Ba are the binarized weights for those regions. These three regions are binarized separately. This method introduces an additional 2 bits for group identification, which constitutes a minor portion of the overall bit count, while the majority of computing parameters remain at I bit.
Average Bits In STBLLM, we introduce extra bits while pruning the redundant or less important weights. The overhead of weight parameters is Nparam x Tsalient + x (1 - -sllient) The additional hardware overhead is Natoring 2 + where Tsalient denotes the proportion of salient weights and baize denotes the block size in OBC compensation, with 2 bits allocated for marking the division of non-salient weights. Under N:M binarization settings, where N and M are positive integers with N < M, prune the model weights by retaining only a fraction (N/M) of the original weights. Consequently, the number of parameters in the pruned STBLLM model is Nu = N Nparam x N This N:M binarization method allows for significant reduction in model size.
M-
7

---

Published as a conference paper at ICLR 2025
Table 3: Perplexity results on Wikitex12 datasets of OPT and Mistral models with BiLLM and STBLLM. For more precise bit-width results, please refer to Table
Settings
OPT
Mistral
Method
W-Bits
1.3B
2.7B
30B
BiLLM
0.80 (6:8)
51.62
23.03
15.82
15.82
72.29
BiLLM
0.70 (5:8)
69.15
30.62
20.58
20.58
82.84
BiLLM
0.55 (4:8)
106.99
55.28
79.68
79.68
189.73
STBLLM
0,80 (6:8)
29.84
17.02
12.79
12.80
27.31
STBLLM
0.70 (5:8)
33.01
20.82
14.38
14.38
25.64
STBLLM
0.55 (4:8)
45.11
30.34
18.80
18.80
70.14
Table 4: Accuracies (%) for 7 zero-shot tasks from structured binarized LLaMA-1-13B LLaMA-2- 13B, and LLaMA-1-308 with BiLLMM and STBLLM. We compare the performance under the same
N:M setting to achieve sub- 1-bit quantization.
Models
Method
Winogrande
OBQA
Hellaswag
Boolg
ARC-e
ARC-c
RTE
Mean
FullPrecision
72.77
33.20
59.94
77.89
77.40
46.50
70.40
62.59
(6LLM(6:8)
58.80
30.60
46.25
62.96
49.96
23.97
53.42
46.57
LLAMA-I-13R
BiLLM(4:8)
52.09
28.00
30.82
61.25
32.66
21.25
53.07
39.88
8TBLLM(6:8)
65.98
36.20
63.67
65.38
68 86
34.04
56.68
55.83
STBLLM((488)
63.06
34.80
$2.65
62.48
56.90
28.33
52.71
50.13
FullPrecision
72.22
35,20
60.06
80.52
79.42
48.46
65.34
63.03
56.43
30.60
35.53
62.48
41.29
24.74
53.43
43.50
LLaMA-2-13B
BiLLM(4:8)
5U.5y
24.00
28.96
62.08
30.51
22.35
53.07
38.79
STBLLM(6:88
63.93
37.00
$7.76
71.53
60.56
31.99
54.15
53.85
STBLLM(4:8)
55.88
29.40
44.03
64.31
48.86
26.54
52.71
45.96
FullPrecision
75.69
36.00
63.35
82.69
80.30
$2.82
66.79
65.38
(6LLM(668
66.54
36.40
58,18
66.15
62.37
31.91
46.93
50.32
LLaMA-1-30B
BiLLM(4:8)
54.93
29.40
38.85
62.17
43.6
24.74
52.35
43.72
STBLI ((6:8)
71.59
41.00
69.85
77.37
71.55
41.3
48.01
60.10
STBLLM(4:3)
64.01
34.60
56.46
63.06
60.86
31.48
51.99
51.78
4 EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
Experimental Setup. Our STBLLM utilizes PyTorch (Paszke et al. 2019) and Huggingface Wolf et al. 2019) libraries. Most LLMS except 65B can evaluated on a single NVIDIA A800 GPU. For the LLaMA- model, employ four NVIDIA A800 GPUs for evaluation. It takes 1.8 hours for the post-training process of 7B models on an RTX 4090 GPU and 2.8 hours for 13B models on A6000 GPU. Following BiLLMM (Huang et al. 2024), our proposed STBLLM also eliminates the need for fine-tuning, offering an efficient post-training quantization framework.
Datasets and Models. We measure the perplexity for language generation tasks on Wikitext2 (Mer- ity et al. 2016). C4 (Raffel et al. (2020) and PTB (Marcus et al.) 1993), and accuracy for the zero-shot lasks including Winogrande (Sakaguchi et al.]
2021), OBQA (Mihaylov
al., 2018)
Hellaswag (Zellers et al. 2019). BoolQ (Clark et al.
2019). ARC (Clark et al., 2018 and
RTE (Chakrabarty et al. [202I). We conduct experiments on
LLaMA-1/24 (Touvron et al. 2023a b),
OPT (Zhang et al. (2022a), and Mistral (Jiang et al. 2023). For perplexity evaluation in Table 2 and
we employ the C4 dataset as the calibration dataset and report the perplexity on Wikitext2.
Baseline. Our primary baseline is BiLLM (Huang et al. 2024). which is 1 1-bit PTQ framework for LLMs. perform an N:M sparse pattern on pre-trained LLMs and then conduct the same procedure BiLLM to report the results that are less than 1 bit (e.g. 0.8, 0.7, 0.55 bits). We conduct the N:M sparsity using Wanda (Sun et al. (024) as the baseline, a gradient- free post-training pruning method. We compare the results of STBLLM with BiLLM under the same N:M settings. For more information on average bits under N:M settings, please refer to Table [ Previous low-bit methods like PB-LLMM (Shang at. 2024), GTPQ (Frantar et al. 2023 and RTN selected for comparison.
8

---

Published as a conference paper at ICLR 2025
4.2
MAIN RESULTS
Comparison with PTQ methods. We comprehensively compare the performance of different LLaMA families across various model sizes (7B-65B). For fair comparison, we set the same block size to 128. As presented in Table 2 the model under RTN and GPTQ fails retain the performance at 1-bit. PB-LLM has shown satisfactory perplexity under 1.7 bit but deteriorates performance compared with BiLLM under 1.09 bit. To further compare the performance at sub-1 1- bit, apply the same N:M setting to BiLLM and our proposed STBLLM. As shown in Figure 2 our proposed STBLLM achieves better trade-off between bit-widths and perplexity across model sizes from 7B to 65B. STBLLM surpasses BiLLM by large margin (688.73 - 31. LLaMA- 1-7B, especially on the most extreme compression case, 4:8 structured binarization, which means setting half of the parameter to zero. It is also noteworthy that when the parameter size reaches 65B, our STBLLM, at 0.55 bit, achieves a perplexity of 11 11.07, surpassing that of PB-LLM (12.53) at 1.7 bit and that of BiLLM (11.57) at 0.7 bit. To our knowledge, breaks the 1-bit barriers by further reducing the redundant weights in an N:M pattern. Moreover, we conduct further experiments on the OPT family from 1.3B to 30B and Mistral-7B at sub-I-bit PTQ settings. From T we observe the same trend as in LLaMA. Our proposed STBLLM performs significantly better than BiLLM across all models and all N:M structured binarization settings.
STBLLM is the first work that
Table 5: Ablation for pruning metric.
Table 6: Ablation study for allocation strategy.
Model
Magnitude
Wanda
SparseGPT
Ours (SI)
Models
Uniform
Sin-shape
Ours
LLaMA-1-7B
4797.41
207.32
32.82
31.72
LLaMA-I-78
80.36
67.78
15.03
LLaMA 2:7B
2287.24
97.54
31.55
27.93
LLaMA-2-7B
50.25
33.61
13:06
Zero-Shot Performance. To conduct a more comprehensive evaluation of binary LLMs, we ex- tend our experiments to 7 zero-shot datasets on LLaMA-1-13B, LLaMA-2-13B, and LLaMA-1- 30B, each tested with FullPrecision, (6:.L. BiLLM(4:88, STBLLM(6:8), and STBLLM(4:8) methods. We mainly focus the performance of these models under the sub- 1-bit setting. Specifi- cally, we compare the BiLLM and our STBLLM under 4:8 and 6:8 structured binarization settings. As illustrated in Table 4 we find that the performance drop in reduced precision methods is more pronounced in BiLLM methods compared to STBLLM methods, indicating that STBLLM methods are more robust alternatives when memory resources are constrained.
4.3 ENHANCING INFERENCE EFFICIENCY ON HARDWARE.
We present specialized CUDA kernels designed to support 1-bit 2:4 sparsification. As illustrated in Figure 4 a). we utilize 2-bit implementation on recent RTX4090 GPU from ABQ-LLM (Zeng al., 2024) as the baseline (W2A16 and W2A8) and compare with our highly-optimized 2:4 1-bit implementation. We provide comparative analysis of runtime and throughput across vari- ous sequence lengths, demonstrating the significant gains in computational efficiency and reduced memory footprint. Specifically, for typical sequence lengths of 4096 and 8192, our implementation achieves up to 17.85 times speedup compared to ABQ-LLM's 2-bit implementation. This speedup stems from our CUDA kernel's efficient use of Sparse Tensor Cores, detailed in Appendix(C At quence length of 8192, our kerne reaches 263.45 TFLOPS, which is 79.74% of the RTX4090's 2:4 sparse tensor core peak performance. Notably, the speedup becomes more pronounced as sequence length increases. Furthermore, as illustrated in Figure our method yields lower perplexity for LLaMA- 1/2 models. This indicates superior model performance and accuracy compared to btt round-to-neares, (RTN), GPTQ, and AWQ. Refer to Appendix C for the memory comparison and implementation details.
4.4 ABLATION STUDIES
Ablation for Metric. Table 5 shows the impact of post-training pruning metrics (Magnitude, Wanda (Sun et al. 2024). SparseGPT (Frantar & Alistarh 2023) and our SI) on 4:8 binary STBLLM regarding LLaMA 1/2-7-/B--BB. 1/2-7B. During PTP, we employ the C4 dataset as the calibration dataset and re- port the perplexity on the Wikitext2 dataset. SparseGPT requires second-order information, which involves a massive computation burden. Similar to Wanda, our SI does not require gradient or second-order information. Our method achieves better performance among these metrics.

---

Published as a conference paper at ICLR 2025
Table 7: Comparison of Magnitude, Wanda, SparseGPT, and SI across different datasets.
Models
LLaMA-I-7B
Models
LLaMA-2-7B
Dataset
Magnitude
Wanda
SparseGPT
Ours(SI)
Dataset
Magnitude
Wanda
SparseCIP
Ours(SI)
PTB
11608.85 88
306.57
61.53
68.48
PTB
45564.36
2027.33
236.03
690.76
C4
1545.34 4797.42
153.29
33.06
36.04
C4
1034.84
86.45
30.53
30.81
Wikitext2
207.32
32.82
31.72
Wikitext2
2287.25
97.54
31.56
27.93
10
2WSA Okn
ABDO
Thanoghoue (PPOOIS)
250
10a
10*
Runtime
10
Perplexity
10
10a
10a
100
10a
101
50
10a
10
10a
512 1024 2045
4005 8096
512
1024 2043
4006 5006
7D
13b
300
70
130
Sequence Length
Sequence Length
Mocull Size
Model Size
(a) Runtime and throughput comparison.
(b) Perplexity comparison.
Figure 4: (a) Runtime and throughput comparison across sequence lengths for ours and ABQ-LLM. (b) Perplexity comparison across model sizes under 2:4 setting for LLaMA-1/2.
Ablation for Quantization Strategy. We conduct an ablation study on different quantization strate- gies. Comparing the perplexity of our Non-salient Aware Quantization (dubbed as Non-salient) and Bell-shaped Distribution Splitting (dubbed as Bell-shaped) in BiLLMM (Huang et al. 2024) on both LLaMA-1-7B and LLaMA-2-7B, as shown in Table The perplexity of Non-salient changes lot when moving from LLaMA-1-7B to LLaMA-2-7B, while our Non-salient exhibits nearly identical perplexity both models, significantly lower than that of Bell-shaped.
an ablation study on different allocation strate- gies. We compare our method with Uniform and Sin-shaped allocation strategies. The Sin-shaped strategy assigns layer-wise sparsity following a sine wave pattern, where the initial layers have lower sparsity and the latter have higher sparsity. The performance of Uniform and Sin-shaped strategies varies significantly across different models. In contrast, our strategy consistently achieves nearly identical performance across both models, outperforming the other two allocation strategies.
Ablation for Allocation Strategy.
Ablation for Group Size. Table 9 presents the results of our ablation study on the group size configuration. We evaluate the perplexity of LLaMA-1-7B and LLaMA-2-7B with group sizes of 64, 128, 256, and 512. Generally, the group size increases, performance improves. However, this also results in higher computational and storage demands. We choose a group size of 128 to balance performance and resource consumption.
Table 8: Ablation for quantization strategy.
Table 9; Ablation for group size.
Models
Bell-shaped
Non-salient
Model
64
128
256
512
1024
LLaMA-1-7B
80.35 50.25
15.03
LLaMA-1-7B
29.58
31.72
33,97
41.29
146.46
LLaMA-2-7B
13.06
LLaMA-2-7B
27.12
27.93
50.62
54.68
507.44
5 CONCLUSION
In this paper, we introduce STBLLM, a structured Binary LLM PTQ framework designed for sub- 1-bit quantization. We address redundancy in binarized LLMs. highlighting the potential for further compression. Specifically, we present Standardized Importance (SI) metric for N:M structured pruning. Then, we use the Hessian matrix partition weights into salient and non- categories. We propose Non-salient Aware Quantization for non-salient weights, identifying optimal splitting points to create sparse, intermediate, and dense regions, each with tailored binarization. Finally, we design a specialized CUDA kernel with a sparse tensor core to achieve significant speedup. We validate the performance of STBLLM across LLaMA- 1/2/3, OPT, and Mistral, demonstrating that STBLLM achieves superior trade-off sub-1-bin settings. By achieving LLM performance under bit, STBLLM highlights the potential of extreme LLM compression. Limitation: STBLLM does not support Mixture of Experts (MoE) or Mamba-based language models.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
This work was partially supported by National Natural Science Foundation of China under Grant No. 62272122, the Guangzhou Municipal Joint Funding Project with Universities and Enter- prises under Grant No. 2024A03J0616 Guangzhou Municipality Big Data Intelligence Key Lab (2033A0300022), and Theme- basee Research Scheme (T45-205/21-N) from Hong Kong RGC, Gen- erative AI Research and Development Centre from InnoHK, Hong Kong CRF grants under Grant No. C7004-22G and C6015-23G.
REFERENCES
Yongqi An, Xu Zhao, Tao Yu, Ming Tang, and Jinqiao Wang. Fluctuation bassdd adaptive structured pruning for large language models. In AAAI, 2023.
Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. SliceGPT: Compress large language models by deleting rows and columns. In ICLR, 2024.
Wenlei Bao, Li-Wen Chang, Yang Chen, Kefeng Deng, Amit Agarwal, Emad Barsoum, and Abe Taha. Ngemm: Optimizing gemm for deep learning via compiler-based techniques. ArXiv, abs/1910.00178, 2019.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. NeurlPS, 2020.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Ka- mar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4, arXiv preprint arXiv:2303. 12712, 2023.
Tuhin Chakrabarty, Debanjan Ghosh, Adam Poliak, and Smaranda Muresan. Figurative language in recognizing textual entailment, In ACL, 2021.
Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher De Sa. Quip: 2-bit quantization of large language models with guarantees. In NeurIPS, 2023.
Hong Chen, Chengtao Lv, Liang Ding, Haotong Qin, Xiabin Zhou, Yifu Ding, Xuebo Liu, Min Zhang, Jinyang Guo, Xianglong Liu, and Dacheng Tao. Db-llm: Accurate dual-binarization for efficient llms. ACL, 2024.
Christopher Clark, Kenton Lee, Ming-Was Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Exploring the surprising difficulty of natural yes/no questions. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), ACL, pp. 2924-2936, 2019.
Toutanova. BoolQ:
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint, 2018.
Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Salch Ashk- boos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. Spqr: A sparse-quantized repre- sentation for near-lossless Iim weight compression. ICLR, 2023.
Peijie Dong, Lujun Li, and Zimian Wei. Diswot: Student architecture search for distillation without training. In CVPR, 2023a.
Peijie Dong, Lujun Li, Zimian Wei, Xin Niu, Zhiliang Tian, and Hengyue Pan. Emg: Evolving training-free proxies for automated mixed precision quantization. In ICCV, pp. 17076-17086, 2023b.
Peijie Dong, Lujun Li, Zhenheng Tang, Xiang Liu, Xinglin Pan, Qiang Wang, and Xiaowen Chu. Pruner-zero: Evolving symbolic pruning metric from scratch for large language models. In JCML, 2024.
11

---

Published as a conference paper at ICLR 2025
Dayou Du, Yijia Zhang, Shijie Cao, Jiaqi Guo, Ting Cao. Xiaowen Chu, and Ningyi Xu. Bitdistiller: Unleashing the potential of sub-4-bit lims via self-distillation. In ACL, 2024.
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In JCML. pp. 10323-10337, 2023.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. In ICLR, 2023.
Georgi Gerganov. llama.cpp: c/c++ port of facebook ai's Ilama language model, 2023.
Hao Gu, Wei Li, Lujun Li, Zhu Qiyuan, Mark Lee, Shengjie Sun, Wei Xue, and Yike Guo. D2-moe: Delta decompression for moe-based Ilms compression. arXiv preprini arXiv:2502. 172988 2025.
Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding. In ICLR, 2016.
Peng Hu, Xi Peng. Hongyuan Zhu, Mohamed M. Sabry Aly, and Jie Lin. Opq: Compressing deep neural networks with one-shot
AAAI, 2021.
Wei Huang, Yangdong Liu, Haotong Qin, Ying Li, Shiming Zhang, Xianglong Liu, Michele Magno, and Xiaojuan Qi. Billm: Pushing the limit of post-training quantization for llms. ICML, 2024.
Itay Hubara, Brian Chmiel, Moshe Island, Ron Banner, Joseph Naor, and Daniel Soudry. Accel- erated sparse neural training: A provable and efficient method to find n: m transposable masks. NeurlPS,
2021.
Albert Qiaochu Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lu- cile Saulnier, L'elio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, ArXiv, abs/2310.06822 2023.
Changhun Lee, Jungyu Jin, Taesu Kim, Hyungjun Kim, and Eunhyeok Park. Owq: Lessons learned from activation outliers for weight quantization large language models. In AAAI, 2024.
Lujun Li. Self-regulated feature learning via teacher-free feature distillation. ECCV, 2022.
Lujun Li and Zhe Jin. Shadow knowledge distillation: Bridging offline and online knowledge trans- fer. NeulPS, 2022.
Lujun Li, Peijie Dong, Zimian Wei, and Ya Yang. Automated knowledge distillation via monte carlo tree search. In ICCV, 2023.
Lujun Li, Yufan Bao, Peijie Dong, Chuanguang Yang, Anggeng Li, Wenhan Luo, Qifeng Liu, Wei Xue, and Yike Guo. Detkds: Knowledge distillation search for object detectors. In ICML, 2024a.
Lujun Li, Peijic, Zhenheng Tang, Xiang Liu, Qiang Wang, Wenhan Luo, Wei Xue, Qifeng Liu, Xiaowen Chu, and Yike Guo. Discovering sparsity allocation for layer-wise pruning large language models. In NeulPS, 2024b.
Lujun Li, Zimian Wei, Peijie Dong. Wenhan Luo, Wei Xue, Qifeng Liu, and Yike Guo. Attnzero: efficient attention discovery for vision transformers. In ECCV, 2024c.
Wei Li, Lujun Li, Mark Lee, and Shengjic Sun. Als: Adaptive layer sparsity for large language models via activation correlation assessment. In NewlPS, 2024d.
Wei Li, Lujun Li, You-Liang Huang, Mark G. Lee, Shengjie Sun, Wei Xue, and Yike Guo. Struc- tured of LLMs compression via singular value decomposition, 2025. URL
https:://ppenrre
ew net http://frmmmmooooooom
Z8A
Yixing Li and Fengbo Ren. Bnn pruning: Pruning binary neural network guided by weight flipping frequency. 2020 21 International Symposium on Quality Electronic Design (ISQED), pp. 306- 311, 2020.
12

---

Published as a conference paper at ICLR 2025
Haokun Lin, Haobo Xu, Yichen Wu, Jingzhi Cui, Yingtao Zhang, Linzhan Mou, Linqi Song, Zhenan Sun, and Ying Wei. Duquant: Distributing outliers via dual transformation makes stronger quan- tized llms. In NeurIPS, 2024a.
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for Ilm compression and acceleration. In MLSys, 2024b.
Chen Liu, Ziqi Zhao, Sabine Süssttrnnk, and Mathieu Salzmann. Robust binary models by prun- ing randomly-inittalized networks. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), NeurIPS, 2022.
Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, and Kwang-Ting Cheng. Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm. In ECCV, 2018.
Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. LLM-QAT: Data-free quantization aware training for large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), ACL, pp. 467 484, 2024.
Xinyin Ma, Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models. In NeurIPS, 2023.
Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313-330, 1993.
Fanxu Meng. Hao Cheng, Ke Li, Huixiang Luo, Xiaowei Guo, Guangming Lu, and Xing Sun. Pruming filter in filter. In NeurlPS, 2020.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? new dataset for open book question answering. In EMNLP, 2018.
Sri Aurobindo Munagala, Ameya Prabhu, and Anoop M. Namboodiri. Stq-nets: Unifying network binarization and structured pruning. In BMVC, 2020.
Nvidia. Nvidia a100 tensor core gpu architecture, 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. NeurIPS, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 2020.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Joseph. Xnor-net: Imagenet classification using binary convolutional neural networks. In ECCV, 2016.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. ACM, 2021.
Yuzhang Shang, Zhihang Yuan, Qiang Wu, and Zhen Dong. Pb-llm: Partially binarized large lan- guage models. ICLR, 2024.
Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantization for large language models. In ICLR2024 Spotlight, 2023.
Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. A simple and effective pruning approach for large language models. In ICL., 2024.
13

---

Published as a conference paper at ICLR 2025
Wei Sun, Aojun Zhou, Sander Stuijk, Rob G. J. Wijnhoven, Andrew Nelson, Hongsheng Li, and Henk Corporaal. Dominosearch: Find layer-wise fine-graines n.m sparse schemes from dense neural networks. In NewrIPS, 2021.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint, 2023b.
Frederick Tung and Greg Mori. Clip-q: Deep network compression learning by in-parallel pruning- quantization. CVPR, pp. 7873-7882, 2018.
Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijie Wang. Lingxiao Ma, Fan Yang. Ruiping Wang, Yi Wu, and Furu Wei. Bitnet: Scaling 1-bit transformers for large language 2023a.
models. ArXiv, abs/2310.
Lei Wang, Lingxiao Ma, Shijic Cao, Quanlu Zhang, Jilong Xue, Yining Shi, Ningxin Zheng, Zim- ing Miao, Fan Yang. Ting Cao, Yuging Yang, and Mao Yang. Ladder: Enabling efficient low- precision deep learning computing through hardware-award tensor transformation. In USENIX Symposium on Operating Systems Design and Implementation, 2024.
Peisong Wang, Xiangyu He, Gang Li, Tianli Zhao, and Jian Cheng. Sparsity-inducing binarized neural networks. In AAAI, 2020.
Peisong Wang, Fanrong Li, Gang Li, and Jian Cheng. Extremely sparse networks via binary aug- mented pruning for fast image classification. IEEE Transactions on Neural Networks and Learn- ing Systems, 34:4167-4180, 2021a.
Yikai Wang, Yi Yang, Fuchun Sun, and Anbang Yao. Sub-bit bit neural networks: Learning to compress and accelerate binary neural networks. ICCV, pp. $340-5349, 2021b.
Yikai Wang, Wen bing Huang, Yinpeng Dong, Fuchun Sun, and Anbang Yao. Compacting binary neural networks by sparse kernel selection. CVPR, pp. 24374-24383, 2023b.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022a.
Jason Wei, Xuezhi Wang, Dale Schuurmans. Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, et al. Chain-of-thoughh prompting clicits reasoning in large language models. NeurIPS, 35: 24824-24837, 2022b.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's 's transformers: State-of-the-ar natural language processing. arXiv preprint arXiv:1910.03771, 2019.
Jaeyeon Won, Jeyeon Si, Sam Son, Tae Jun Ham, and Jae W. Lee. Ulppack: Fast sub-8 matrix multiply on commodity simd hardware. In Conference Machine Learning and Systems, 2022.
Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared llama: Accelerating language model pre-training via structured pruning. ICLR, 2024.
Liu Xiaolong, Li Lujun, Li Chao, and Anbang Yao. Norm: Knowledge distillation via n-to-one representation matching. In ICLR, 2023.
Yuzhuang Xu, Xu Han, Zonghan Yang, Shuo Wang, Qingfu Zhu, Zhiyuan Liu, Weidong Liu, and Wanxiang Che. Onebit: Towards extremely low-bit large language models. In NeurIPS, 2024.
Haichuan Yang, Shupeng Gui, Yuhao Zhu, and Ji Liu. Automatic neural network compression by 21 2175-2185, 2019.
joint learning: A constrained optimization based approach. CVPR, pp.
14

---

Published as a conference paper at ICLR 2025
Lu Yin, You Wu, Zhenyu Zhang, Cheng- Yu Hsieh, Yaqing Wang, Yiling Jia, Mykola Pechenizkiy, Yi Liang, Zhangyang Wang, and Shiwei Liu. Outlier weighed layerwise sparsity (OWL): A missing secret sauce for pruning LLMs to high sparsity. In ICML, 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? ACL, 2019.
Chao Zeng, Songwei Liu, Yusheng Xie, Hong Liu, Xiaojian Wang, Miao Wei, Shu Yang, Fangmin Chen, and Xing Mei. Abq-llm: Arbitrary-bii quantized inference acceleration for large language models. arXiv preprint arXiv:2408.0883 08554, 2024.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona T. Diab, Xian Li, Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettle- moyer. Opt: Open pre-trained transformer language models. ArXiv, 0ans/205.01066 2022a.
Ying Zhang, Peng Zhang, Mincong Huang, Jingyang Xiang, Yujie Wang, Chao Wang, Yineng Zhang, Lei Yu, Chuan Liu, and Wei Lin. Qqq: Quality quattuor-bit quantization for large language models. ArXiv, 00s/24006009044 2024a.
Yingtao Zhang, Haoli Bai, Haokun Lin, Jialin Zhao, Lu Hou, and Carlo Vittorio Cannistraci. Plug- and-play: An efficient post-training pruning method for large language models. In ICLR, 2024b.
Yuxin Zhang, Mingbao Lin, Zhihang Lin, Yiting Luo, Ke Li, Fei Chao, Yongjian Wu, and Rongrong Ji. Learning best combination for efficient M sparsity. In NeurlPS, 2022b.
Aojun Zhou, Yukun Ma, Junnan Zhu, Jianbo Liu, Zhijie Zhang, Kun Yuan, Wenxiu Sun, and Hong- sheng Li. Learning n: M fine-grained structured sparse neural networks from scratch. In ICLR, 2021.
15

---

Published as a conference paper at ICLR 2025
APPENDIX OVERVIEW
Section
A
STBLLM Implementation.
Section
Details of Motivation Experiments.
Section
Details of Hardware Accelerations.
Section
D
Impact of Extreme Weight on the Hessian Matrix.
Section
More Experimental Results and Ablation Study,
A STBLLM IMPLEMENTATION
Building on BiLLM (Huang et al. 2024). STBLLM preserves operations on salient weights while prioritizing non-salient weights. As detailed in Algorithm 2 the BINARY and RES APPROX functions remain consistent with BiLLM, whereas we introduce the NonSalientAwareQuan and Trisection functions to enhance the framework.
For ConSalientAwareQuuan function, it aims to find two optimal break-points to partition the sym- metric Gaussian distribution of non-salient weight.
The naive implementation requires over 6 hours for LLaMA-2-7B, rendering it infeasible for 70B- scale LLMs. naive approach for searching the break-point is using two nested loops, whose complexity is ON?)), where N denotes the length of the search space. To reduce the complexity to O(N), we propose to utilize P2 we have > 1.
P1 to locate the p2. It is natural to assume that p2 P1 and
To address this, we introduce fixed ratio a mitigate the computational overhead. Specifically, as illustrated in Figure non-salient weights conform to a Gaussian distribution, w ~ NN o with the probability density function given by:
f(w)
exp
V2TO
20
Leveraging the symmetry of the Gaussian distribution, we define the probabilities for three parti- tions:
Sparse partition: Pspurse = 2 (w) dw = 2Q (22)
Intermediate partition: Pintermedaaee = f (w) dw = 2 [Q (0) Q (*))
Dense partition: PDease = 1 f(w) dw 1 - -2Q (
where Q(x) = 10
J 12 denotes the Gaussian tail probability. To ensure equal partition
V2x
areas, we set Psparse Plntermediate PDease 5. yielding:
@ (
= 3
1
Q
D2 = Q ( - Q
1
6'
(8)
Solving these, we obtain
Q-1 (4) and = Q-) Using standard normal inverse Q- function values, we find RJ 2p1, implying that the parameter σ = 2 in Algorithm 2
For Trisection function, it aims to partition the symmetric Gaussian distribution presented in Fig- ure (( into three parts, which are Sparse, Intermediate, and Dense region. These three parts have no interscttoon and by uniting them together, we have all of the non-salient structured binarized weight.
Figure provides a detailed illustration of our weight matrix partitioning strategy, complementing the overview presented in Figure 3(b). The figure demonstrates how we first partition weights into salient and non-salient regions based on Hessian matrix. For the salient weights, which constitute small portion of the total weights, we employ residual approximation following BiLLM (Huang et al. 2024). The non-salient weights undergo our novel trisection partitioning scheme, where they further divided into three distinct regions (dense, intermediate, and sparse) for optimized quanti- zation. This hierarchical partitioning enables fine-grained control over compression while preserv- ing model performance. The visualization shows how each region is processed differently, with
16

---

Published as a conference paper at ICLR 2025
Algorithm 2 STBLLM
fune Salient (W, H))
function NONSSLLEETTAAARRRUUAANYW
1: function SALIENT(W, H
2:
e e 0c
D Searching error
W row:, + topk(sum(abs(SS) dim = 0)
& Salient matrix
3:
pi + 0 b Optimal break-point for trisection
4:
Optimal break-point for trisection
4: e 00
Searching error
5:
for i np.linspace 0.9, 160) do
5: n' <- 0b Optimal number of salient columns
p1 <- i max([[]
6:
for 1 to len(row..) do
7:
P2 f 0 x P1
B1 <- binary (W Wassj J E row, :]]
8:
if p2 > 0.9 max [[] then
8:
B2 <- bimary(We [W.... j. @ row, =])
9:
continue
9:
if IW (B; U B2)l
then
10:
end if
10:
IW (B, U B2]ll
11:
B1 < BINARY (ma))>>))))
n
12:
B2 BINARY W
pr Pi<lus, 5/S
12:
13:
B₃ < BINARY W |ma.ySpp
13:
end for
14:
if W (B: + B2 + Ba)]l < then
14: return row, [ n
15:
W (B, B + Bs]]l
15: end function
16:
P1
17:
pi <- P2
I: function BINARY(W)
18:
end if
<X 6- n
19:
end for
3: B E- a sign W)
20:
return pi.pE
4: return B
21:
end function
5: end function
1: function TRISECTION pi, P5)
1: function RES APPROX(W) W)
B2 BINARY (eqg)>gg))
2: B; + BINARY(W)
3: B3 < BINARY W Pj ((
3: R W B,
4: Ba BINARY W [er
B2 < (R)
B B1 + B2
5: return B B,, B,
6: return B
end function
7: end function
Non-Salient Group Partition
Non-umpty wrigh
Emply wiiht
Monked weight
Non-Sulient Column Bitnap
Pruning
Group Bitmap
Salient Group Partition
Salient Blimap
Figure 5: Column bitmap under 2:4 semi-structure pruned binarized matrix partition illustration.
the trisection boundaries clearly delineating the transitions between dense, intermediate, and sparse non-salient weight regions.
17

---

Published as a conference paper at ICLR 2025
1.0
RTE
0.9
HELLASWAG
BOOLQ
0.8
ARC. EASY
Ascurucy
0.1
0.25
0.35
0.55
Ratio
Figure 6: The impact of random flipping non-salient binarized weights on accuracy 1-Bit LLaMA- 2-7B with group size of 512.
B DETAILS OF MOTIVATION EXPERIMENT
In this section, we delineate the specifics of the motivation experiments as illustrated in Figure | Initially, we elucidate the procedure for inverting the signs of elements within matrix, as detailed in Algorithm 3 to examine its effects on various computational tasks. This algorithm is designed to efficiently invert the signs of specified proportion of elements given matrix W. Subsequently, we employ this function on RES APPROX and invert the signs of each binary matrix, including Bj and B2.
Algorithm Algorithm for Efficiently Flipping Signs of Matrix Elements.
1: function FLIPSIGNSEFFRCIENTWWW ratio, C + None)
2: n <- numel(W)
Total number of elements W D Number of elements to flip
3:
k <- int(n x ratio)
4:
if + None then
5:
assert shape( C) shape W)
Ensure C matches W Flatten and get sorted indices Select least significant elements to flip
6:
.,, .ixxx 5- sort(C.view--I)
7:
idx to flip idx[: k]
8:
else
9:
idx_to
idiptoffiip handom_indicess k)
Random select elements to flip
10:
end if
W Ali W.clone()
Create copy of W View the copy as ID tensor Flip the signs of selected
12: W Wilip-flat t- W "'lip- (iip-view(11)
Wnip flt Wiip--nttiid lidx flip] W Aipp Aat [idx flip] -1
elements
return WRip
15: end function
We further visualize the impact of random flipping non-salient binarized weights on accuracy at higher ratios (from 0.2 to 0.5) in Figure 6 As shown in the table below, we find that as the ratio increases, the performance fluctuates but does not deteriorate drastically across multiple tasks like RTE (accuracy ranging from 0.3-0.9), HellaSwag (stable around 0.3), BoolQ (0.2-0.7), and ARC Easy (0.3-0.5). This suggests a degree of robustness in our approach to varying ratios of flipped non-salient weights.
18

---

Published as a conference paper at ICLR 2025
Structured Pruned Matrix 4- 4-Group Index representation
2
0
3
3
2
0
0
0
3
2
0
2
Non-Zeros Meta Index
zero items
non--2ero items
Q1
00
11
01
10
00
10
meta
6 Meta Index
Figure 7: Structured Pruned Matrix 4-Group Index Representation for 2:4 Structured Sparsity Ac- celeration.
C DETAILS OF HARDWARE ACCELERATION
C.I IMPLEMENTATION DETAILS
Recent advancements low-precision computing have significantly enhanced the practical imple- mentation of efficient neural network techniques. A prime example is the introduction of Ladder (Wang et al. 2024). released as BitBLAS, a software library that seamlessly integrates into existing Deep Neural Network (DNN) and Large Language Model (LLM) frameworks. This integration ables highly efficient low precision computations across various hardware platforms. The impact of these developments is evident in popular frameworks like llama.cpp (Gerganov (023), which supports 1.5-bit quantization through BitNet (Wang et al.) (2023)). This advancement has resulted in impressive performance gains, achieving 198 tokens per second on a single CPU core. Moreover, for large-scale models such as LLLMA-2-7000 the implementation of Ladder (Wang et al. (024) to accelerate BitNet 1.58 (Wang al. 2023a) has yielded remarkable results, demonstrating a 4.6x speedup compared to FPI6 precision.
The emergence of Sparse Tensor Cores (SPTCs) since NVIDIA's Ampere architecture has revo- lutionized the processing of sparse matrices, offering efficient mechanism for handling 50% sparsity. Theoretically, by eliminating half of the computations, SPTCs can potentially double the computational power compared to Dense Tensor Cores. There are already several research over accelerating dense tensor core, including ULPPack (Won et al. 2022), NGEMM (Bao et al. 2019) and QQQ (Zhang et al.) 2024a). However, efficiently representing 1-bit values (+1, -1 and 0 for sparsity) and achieving sufficient real-world acceleration pose significant challenges. To address these, we propose novel 2-bit integer representation method, particularly useful for the General Matrix Multiply (GEMM) operation, formulated as = A(E) x B + C. Here, A represents the 2:4 1-bit sparse matrix, B, C, and D are dense tensors, and E employs u uintle 6 to denote the valid indices of A.
Our approach introduces 6-bit encoding scheme for each group 2:4 sparse 1-bit values. This scheme comprises four bits for indexing and two bits for physical value representation, where + +1,0 -1, and positions unmarked by E indicate sparsity (0). This method significantly improves memory efficiency compared to baseline approach using 2-bit integers to represent - 0, and +1, which would require 8 bits for an equivalent group size. Consequently, our encoding method reduces memory footprint by approximately 25%, leading to decreased global memory access requirements. In memory-bound scenarios typical of large-scale model inference, this approach theoretically offers up to 1.333-fold increase in processing speed compared to the 2-bit variant.
To fully leverage these optimizations, we employ semi-structured pruning techniques specifically tailored for NVIDIA's GPU architecture. These techniques enable the use of Sparse Tensor Cores optimized for processing sparse matrices. By structuring the sparsity (e.g., N:M sparsity where N out of M weights non-zero), stantial improvements in processing speed and efficiency. Specifically, the process of implementing these optimizations involves several key steps in matrix compression and manipulation:
effectively utilize the Sparse Tensor Cores, leading to sub-
19

---

Published as a conference paper at ICLR 2025
2:4 Struetured Matrix
Activation
3
-4
23
0
15
-9
0
2
16
Notr-emoo tndex
1
0
-1
-
Selector
-1
Non-Zeros Pruncd-Binary Real Value
n
O
I
10
00
01
Non-Zeros Real Value
ftems
ilems
1011 0101
value
Tensor Core
Uint& Value
Figure 8: The overview of sparsity pattern of 1-bit kernel Figure 9: 2:4 Structured Sparsity Matrix that convert weight matrix to structured pruned matrix.
Multiplication Using Tensor Cores.
The input matrix is partitioned into 4-element groups as shown in Figure 7 Zero elements identified and non-zero elements extracted. The positions non-zero zero elements are recorded in a Non-Zeros Meta Index, which then encoded into compact Uint16 Meta Index. This encoding facilitates efficient localization of zero elements during matrix operations, enhancing computational speed by enabling the omission of zero elements.
Matrix Compression:
2. Value Compression: Similar matrix compression, the matrix is divided into 4-element groups as shown in Figure g The sign of each non-zero element is extracted to form the Non-Zeros Pruned-Binar Real Value. These values are then converted to binary format, creating the Non-Zeros Real Value. Finally, the each of eight binary values are concatenated into compact Uint Real Value, optimizing storage and computation by focusing on the non-zero elements and their signs.
3. Matrix Multiplication with Structured Sparsity: The input matrices undergo pruning to retain only non-zero elements and their corresponding indices as shown in Figure 9 The pruned matrices are then quantized, extracting non-zero values and their positions. Both processed matrices are subsequently input into the Sparse Tensor Cores, which executes efficient multiplication by focusing on the non-zero elements, resulting in compressed and accelerated computation.
C.2 THEORY ANALYSIS
To evaluate the performance of various matrix multiplication algorithms across different problem sizes, we present a comprehensive roofline model analysis in Figure TO Each subplot depicts the relationship between arithmetic intensity (FLOPS/Byte) and performance (TFLOPS). During pre- filling stage, N denotes the product of sequence length and batch size. For decoding stage, N denotes the batch size. M and K correspond the dimension of weight matrix. To compare dif- ferent implementations, we include standard FP16 GEMM, 2-bit quantized GEMM, and 1-bit 2vs4 quantized GEMM. alongside theoretical performance limits represented by roofline models for Ten- sor Core and Tensor Core Sparse operations. From our analysis, we observe that as N increases, all algorithms exhibit improved performance, with quantized versions consistently outperforming standard GEMM. We find that our 1-bit 2vs4 quantized GEMM demonstrates superior performance, particularly at larger values, often approaching the Tensor Core Sparse roofline.
The advantages of our 1-bit 2:4 quantized GEMM kernel arise from reduced memory access over- head and the higher compute upper bound of Sparse Tensor Cores (SPTCs). When is small (particularly during the decoding phase), all GEMM kernels are memory-bounde but our 1-bit 2:4 quantized GEMM kernel achieves relatively better performance due its higher compression rate. As N increases (especially during the prefilling stage), the quantized GEMM kernels tend to become
20

---

Published as a conference paper at ICLR 2025
M = K = 4096, N = 8
M = K = 4096, N = 32
Performance (TFLOPS)
Performance (TFLOPS)
10a
10
Tarisc Core FP16
Torarr C Core FP16.
Tanso FP16 Spacie
Jasu EPts Spaze:
Memory Uandwidth
Memory Bandwidth
10
RoollinciTaeee Corc)
10
Roofline (Tonsor Corz)
Rootline Spurse)
Roofline (Tensor Spans)
GEMM
GEMM
2-hit Quant GEMM
24ht Quant GEMM
Zvoil
2vpl Quam
10"
10
10
10'
10²
10
10
10'
10
10"
Arithmetic Intensity (FL.OPS/Byte)
Arithmetic
Intensity (FI (FI.OPS/Btte)
M K = 4096, N 64
M
K 4096, N 128
Performance (TFLOPS)
Performance (TFLOPS)
10a
10
Lenbee Core FP16
lersor Core FP16
Tattooe Core Sparie
Tensur EP16 Spars:
Memory Bandwidth
Memary Bendwidth
10'
Roofline (Tmnor Corc)
10
Roofline (Tenser Coro)
Rootlire (Tznsor
Reafline (Tensor Spans]
GEMM
GEMM
Quant
Quant
Ivs4 GEMM
2us4 GEMM
10º
10
10
10'
10²
10
10
10'
10
10
Arithmetic Intensity (FLOPS/Byte)
Arithmetic Intensity (FLOPS/Byte)
Figure 10: Roofline for Sparse GEMM Quantization.
compute-bound. In this case, our specialized GEMM kernel can theoretically achieve 2x speedup compared other GEMM kernels.
This extreme quantization approach significantly reduces both computational overhead and memory footprint by limiting the precision of weights to just possible states. Such method is par- ticularly advantageous in resource-constraing environments, improving the deployability of large models on devices with limited hardware capabilities.
Given the success of the 1-bit 2:4 quantized GEMM kernel in improving the efficiency of large language models by reducing memory and computational demands, we now plan to extend the STBLLM framework to compress Mixture of Experts (MoE) (Gu et al. 2025 Li et al.) (2025). By leveraging AutoML (Li et al. 2024a] Dong al. 2023a; Li et al.) 2023 2024c), we aim to automate and optimize the compression process for these complex models, ensuring they can also be efficiently deployed on resource-constrained devices.
C.3 MEMORY COMPARISON
As illustrated in Figure 11 we present the memory consumption of FP16, CUTLASS, ABQ-LLM, and our implementation for LLaMA-7B, 13B, and 30B models. Our proposed methodology demon- strates a substantial memory compression gain, exceeding 3.1 times that of SmoothQuant. This performance significantly surpasses current mainstream inference techniques. Furthermore, proach achieves an approximate 15% reduction in memory usage compared to ABQ-LLM. These notable improvements have important implications for the field of large language models (LLMs). By reducing the memory footprint, our method decreases the operational costs associated with LLM services and facilitates their practical deployment in real-world applications.
ap-
21

---

Published as a conference paper at ICLR 2025
FP16
WEAL&CUTLASS)
W4AIRCUTLLASS
W2ARABQ4IMMG
Oun
1LMM/-TB
ILMMA-13B
ILaMA-30
Manory (CB))
10
128
156
512
1024
128
256
512
1024
128
256
512
1024
Seguence Length
Sequence Length
Length
Figure 11: Memory Usage Comparison of Various Quantization Methods for LLaMA Models.
Table 10: Comparison of Quant-Only, Structure-Only. and ours across different datasets.
LJaMA-1-7B
LLaMA-2-7B
Dataset
Quant-Only
Structure-Oni Only
Ours
Quant-Onij
Structure-Only
Ours
PTB 23.52
14,24
68.48
2071.44
69.25
690.76
15.75
10.52
36.04
14.62
10.29
Wikitext2
12.29
31.72
7.85
27.93
D IMPACT OF EXTREME WEIGHT ON THE HESSIAN MATRIX
The Hessian matrix is defined as: Hij Sw, where is the loss function, and wy and wy are weights. a weight wi has extreme values, the corresponding elements in the Hessian matrix, particularly Hgg. will be significantly larger than others.
AL
For instance, if w, is an extreme value, the Hessian matrix might look like:
hn
h12
hin
h2
hza
h2a
H
hnl
hn2
Here, hn is much larger than other elements. This disproportionate value significantly influences the Hessian's eigenvalues, with least eigenvalue becoming very large. During optimization, methods like Newton's 's method update weights using the inverse of the Hessian matrix:
W w nH LV((),
where 7) is the learning rate, and V[(w) is the gradient. The presence of an extreme value hn causes the corresponding element in H-1 to be very small, affecting the step size in weight updates:
0L Awj -17 Ow
hh11.
0L /
Auta -17 Ouzz
Since hi is large, Aw; becomes small, indicating minimal adjustments for the extreme value weight, while Awz remains relatively larger for the normal weights.
E MORE EXPERIMENTAL RESULTS
E.1 MODULE ABLATION STUDY
To evaluate the interdependent interaction between quantization and pruning within our STBLLM framework, conduct a module ablation study. This study isolates the effects of quantization-only, pruning-only, and our combined method on the performance the LLaMA-1-1 1- and LLaMA-2-7B models across the PTB, C4, and Wikitext2 datasets. The results presented in Table 10
22

---

Published as a conference paper at ICLR 2025
Magnimude Wanda
SparseGPT
80
Unifomn
10
4797.42
70
67.78
Sin-shape
2287,25
Perplexity
60
Perplexity
50.25
10
50
33.61
30
10"
31.72
31.56 27.93
20
15.05
13.06
10
10
LLaMA-1-7B
LLaMA-2-7B
LLaMA-1-7B
LLaMA-2-7B
Model
Model
Ablation study on post-training pruning metries on STBLLM on L-aMA-1- 7B and LLaMA-2-7B. Our method achieves the best performance among these metrics.
Figure 12:
Figure 13: strategies on STBLLM on LLaMA-1-7B and I.L.MA--7BB Our method achieves nearly identical perplexity.
Ablation study allocation
18
1099.1
907. 44
348
LLaMA-I-7B
LLaMA-1-7B
LLaMA-I-7B
LLaMA-2-7B
1000
LLaMA-2-7B
LLaMA-2-7B
Perplexity
690.76
400
543.48
200
129
500
24
252
56
233
200
146
36
100
30.91
25.73
36,64
30
39
45
40.44
44.31
90
27.93
33
30.62
31
64
128
256
512
1024
64
128
256
512
1024
64
128
256
$12
1024
C4
PTB
Wikitext2
Group Size
Figure 14: Comparison across different sizes for LLaMA-1-7B and LLaMA-2-7B.
The ablation results highlight the synergistic effect of combining quantization and pruming in our approach, significantly outperforming each method applied in isolation.
LLAMA-1-7B Analysis
PTB Dataset: Our combined method achieves score of 68 68.48, markedly higher than quantization- only (23.52) and pruning-only (14.24). This demonstrates the substantial performance gains achieved by leveraging the complementary strengths of both techniques.
C4 Dataset: Our method scores 36.04, compared to 15.75 quantization-onie and 10.52 for pruning-only. The combined approach effectively mitigates the limitations of individual methods, resulting in superior performance.
The score of 31.72 for our method far exceeds the results of quantization-only (12.29) and pruning-only (8.13), underscoring the enhanced model efficiency and accuracy through integrated approach.
Wikitex12 Dataset:
LLaMA-2-7B Analysis
PTB Dataset: Although quantization-only achieves unusually high score of 207 1.7,,44, our com- bined method still significantly outperforms pruning-only (690.76 76 vs. 69.25). This suggests that while quantization might retain certain advantageous structures, the integration with pruning leads to more balanced and robust model.
C4 Dataset: The combined method's score of 30.81 surpasses quantization- (14.62) and pruning-only (10.29), highlighting the effectiveness of our method maintaining high performance across varying model versions.
23

---

Published as a conference paper at ICLR 2025
Table 11: Comparison of C4, PTB, and Wikitext2 across LLaMA-1-7B and LLaMA-2-7B
LLaMA-1-7B
LLaMA-2-7B
Dataset
C4
PTB
Wikitex12
C4
PTB
Wikitext2
36.04
68, 48
31.72
30.81
690.76
27.93
PTB
54.57
35.13
49.27
43.04
4569.03
40.94
Wikitext2
40.76
71.81
20.48
37.01
1970.76
20.60
Table 12: Comparison across different sizes for LLaMA-1-7B and LLaMA-2-7B
LLaMA-1-7B
LLaMA-2-7B
Group Size
C4
PTB
Wikitext2
C4
PTB
Wikitext2
64
30.91
63.24
29.58
28.73
543.48
27.12
128
36.04
68.48
31.72
30.81
690.76
27.93
256
39.45
68.15
33.97
40.64
244.22
50.62
512
44.31
90.19
41.29
40.52
282.58
54.68
1024
129.62
283.25
146.46
348.18
1099.61
507.44
Wikitext2 Dataset: Our method's score of 27.93 is higher than both quantization-only (11.17) and pruning-only (7.85), further confirming the synergistic benefits of combining these techniques.
E.2 ABLATION STUDY OF CALIBRATION DATASET
Table II presents ablation study comparing the performance of LLaMA-1-7B and LLaMA-2-7B models when trained on different calibration datasets: C4, PTB, and Wikitex12. The purpose of this experiment is investigate how the choice of calibration dataset affects the models' performance on various evaluation datasets.
In this study, both LLaMA-1-7B and LLaMA-2-7B models are trained on each of the three calibra- tion datasets separately. The trained models are then evaluated on all three datasets, resulting in 3x3 matrix of performance scores for each model.
The performance scores in the table likely represent some evaluation metric, such as perplexity or loss, where lower values indicate better performance. The diagonal values (c.g.. C4 evaluated on C4) represent in-domain performance, while off-diagonal values represent out-of-domain performance.
E.3 ABLATION STUDY OF GROUP SIZE
Table 12 and Figure 14 presents an ablation study that compares the performance of LLaMA-1- 7B and LLaMA-2-7B models across different group sizes. The purpose of this experiment is to investigate how the choice of group size affects the models' performance on various evaluation datasets.
In this study, both LLaMA-1-7B and LLaMA-2-7B models are trained with different group sizes: 64, 128, 256, 512, and 1024. The trained models are then evaluated on three datasets: C4, PTB, and Wikitext2. The performance scores in the table likely represent some evaluation metric, such as perplexity loss, where lower values indicate better performance. By comparing the performance scores across different group sizes and evaluation datasets, researchers can gain insights into the impact of group size on the models' performance and generalization capabilities.
The results show that the performance of both models varies with the choice of group size. For LLaMA-1-7B, the best performance on C4 and Wikitext2 is achieved with group size of 64, while for PTB, the best performance is obtained with group size of 128. For LLaMA-2-7B, the best performance on C4 and Wikitext2 also achieved with a group size of 64, while for PTB, the best performance is obtained with group size of 256. Interestingly, the performance of both models deteriorates significantly when the group size is increased to 1024, suggesting that excessively large group sizes may lead overfitting other training issues.
24

---

Published as a conference paper at ICLR 2025
Table 13: Motivation: The perplexity on different top percentages.
Top Percentage
Perplexity
Top Percentage
Perplexity
Top Percentage
Perplexity
0.01
27.770422
0.02
30.168285
0.03
34.049734
0.04
36.191769
0.05
33.821476
0.06
36,452296
0.07
38.702617
0.08
39.169894
0.09
44.818825
0.10
54.451229
0.11
49.835159
0.12
71.762848
0.13
52.129317
0.14
52.568348
0.15
65,945448
0.16
62.712751
0.17
117.990227
0.18
138.912356
The provided Figure and Table 13 present experiment that investigates the relationship be- tween the top Percentage of data and the corresponding perplexity scores in LM. The purpose of this experiment is to understand how the choice of top Percentage affects the model's perfor- mance and to determine an optimal threshold for data selection. In this experiment, we randomly flip 1%-6666 16% weights from binarized LM and evaluate their downstream tasks' performance includ- ing ARC (Clark et al., 2018). BoolQ (Mihaylov et al. 2018), Hellaswag (Zellers et al. 2019) and RTE (Chakrabarty et al. 2021). p. [[show the perplexity scores for each top Percentage. Lower perplexity scores indicate better language model performance, as the model is better able to predict the next word in a sequence.
Figure ! provides visual representation of the relationship between the top Percentage and per- plexity scores. shows that the perplexity scores initially improve as the top Percentage increases, indicating that including more high-quality data points benefits the model's performance. However, beyond certain threshold (around 0.05 to O. 10), the perplexity scores start deteriorate, suggesting that including lower-quality data points negatively impacts the model's performance.
E.4 ABLATION STUDY: PRUNE- THEN- QUANTIZE VS. QUANTIZE-THEN-I PRUNE
To validate our choice of mrune-then-quantee strategy, we conduct an ablation study comparing it with the alternative quantize-thie -eent -her-rrtte---eeeeeeeeee approach. As shown in Table the prune- nume-then--uaan then-quantize approach consistently achieves better perplexity scores across both LLaMA-1-77 and LLaMA-2-7B models.
Table 14: Comparison of Pruning and Quantization Order (6:8 ratio).
Approach
Averaged Bits
Model
Perplexity
Prune + Quantize
0.82
LLaMA-1-1 1-7B
15.03
Prune Quantize
0,80
LLaMA-2-7B
13.06
Quantize >> Prune
0.82
LLaMA-1-7B
34.02
Quantize Prune
0.80
LLaMA-2-7B
31.98
These results empirically support our design choice, showing that prune-then-quantzz achieves sig- nificantly lower perplexity compared to quantize-then-prume Our analysis suggests this is because quantization typically causes less performance degradation compared pruning. When applying the more damaging operation (pruning) after quantization, it becomes more challenging to recover performance through block-wise OBC. Conversely, applying quantization after pruning allows for better performance recovery.
E.5 ABLATION STUDY: IMPACT OF WEIGHT PARTITIONING STRATEGIES
To investigate the effectiveness of different weight partitioning approaches, we conducted an abla- tion study comparing various partitioning strategies under identical experimental conditions using the LLaMA-2-7B model with 6:8 sparsity ratio:
The bell-shaped distribution approach, originally proposed in BiLLM, and non-salient parti- tioning strategy in STBLLM demonstrate comparable computational efficiency with search times of approximately 0.5 hours. While the naive implementation method achieves slightly lower perplex-
25

---

Published as a conference paper at ICLR 2025
Table 15: Comparison of Different Weight Partitioning Strategies and their Search Time.
# Partitions
Perplexity
Search Time
(Bell-shaped)
50.25
00.5
2 (Non-salient)
13.06
0.5h
2 (Naive implementation)
12.78
~6h
ity score of 12.78 compared search time of approximately hours a 12-fold increase in computational cost.
non-salient approach (13.06), requires significantly longer
Based on these empirical results, we adopt T-2 (three partitions) for our non-salient partitioning strategy as it provides an optimal balance between granular weight importance differentiation and computational efficiency.
E.6 ABLATION STUDY: IMPACT DIFFERENT PRUNING METHODS
To comprehensively evaluate the effectiveness of different pruning methods, we conducted exper- iments using both Wanda and SI pruning on different weight distributions in LLaMA-1-7B and LLaMA-2-78 models. [obeent the percentage of weights affected by each pruning method across different weight distributions.
Table 16: Impact of Different Pruning Methods on Weight Distributions.
Model
Distribution
Method
Percentage (%)
LLaMA-1-7B
Bell-shaped
Wanda SI
80.35
Non-salient
15.03
Bell-shaped
SI
40.25
Non-salient
Wanda
31.72
LLaMA-2-7B
Bell-shaped
Wanda SI
50.25
Non-salient
13.06
Bell-shaped
SI
24.54
Non-salient
Wanda 27.93
The results demonstrate that different pruning methods exhibit varying effectiveness depending on the weight distribution. For bell-shaped distributions, Wanda pruning affects a larger percentage of weights (80.35% and 50.25% for LLaMA-1-7B and LLaMA-2-7B respectively), while SI pruning shows better efficiency non-salient weights (15.03% and 13.06%). This analysis supports strategy of applying different pruning methods based on the weight distribution characteristics.
26


Reviews:
[{"review_id": "Gp0bSq0XZ6", "reviewer": "Reviewer_jQJp", "summary": "This work presents a structural binarization method for LLMs by combining N:M sparsity, residual approximation, and block-wise error compensation. Extensive experiments on LLaMA-1/2/3, OPT, and Mistral are conducted to evaluate the effectiveness of STBLLM. In addition, a specialized CUDA kernel is designed to support structural binarization.", "questions": "See weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* The analysis on flipping non-salient binarized weights is intriguing. I am wondering what would happen if we increase the ratio from 0.15 to 0.5?\n* The proposed method achieves the lowest perplexity among all compared methods in the sub-1-bit regime.\n* A specialized CUDA kernel for structural binarization, leveraging NVIDIA's Ampere GPU sparse tensor cores, achieves a 17.85x speedup over ABQ-LLM's 2-bit implementation.", "weaknesses": "* The proposed method is a combination of several existing techniques including N:M sparsity, residual approximation, block-wise error compensation, and Trisection search (for the non-salient part). This raises some novelty concerns. I suggest the authors to 1) highlight the main novelty and contribution of the current submission; 2) provide ablation studies on a. how important the residual approximation is, b. the impact of Trisection search for grouping and why there are two groups.\nIn addition, which techniques contribute the most to efficiency and which method contributes the most to the accuracy?\n* The benchmark results are based on various N:M configurations. However, NVIDIA GPUs mainly support 2:4. The authors may discuss how practical the proposed method is on NVIDIA GPUs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Jj4Bo95DBZ", "reviewer": "Reviewer_KgJf", "summary": "This paper proposes a structured binary quantization method to accelerate LLM inference. It combines n:m pruning and binary quantization, compressing the model weights to an average of less than 1 bit. In n:m pruning, the authors introduce an SI method for indentifying significant weights, and a layer-wise dynamic n:m allocation method. In binary quantization, the authors partition the weights into salient and non-salient parts for separate processing and further apply a group-wise quantization method to the non-salient part. Experimental results demonstrate that STBLLM outperforms BiLLM under the same bit budget. In addition, significant performance improvement (17x) is achieved with customized CUDA kernels.", "questions": "Please see the weaknesses.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "+ 1-bit weight quantization is important for accelerating LLM inference.\n+ Dedicated CUDA implementations for the proposed method.", "weaknesses": "+ incremental novelty\n\nWhile the proposed method is interesting and performs better than BiLLM, its novelty is limited: 1) The proposed SI method is very similar to Wanda, with the main difference being the introduction of additional data normalization. 2) The binary quantization method is quite similar to BiLLM, where the hessian matrix is used to divide weights into salient and non-salient parts, and residual approximation is employed to handle the salient part. The only difference is that STBLLM processes the non-salient weights into three parts instead of two as in BiLLM.\n\n+ mismatch between motivation and methodology\n\nThe motivation of this paper lies in the observation that some weights in binary LLMs do not significantly affect accuracy and can be further compressed (Section 3.1 and fig 1). Under this narrative, a reasonable approach would be to perform pruning on the binarized model to achieve further compression. In contrast, the method proposed in this paper adopts a ‘’prune-then-quantize’’ approach, which does not align with the motivation. The paper does not explain why pruning should be performed first and does not discuss how changing the order of pruning and binarization might affect the results. \n\nThe motivation behind using a trisection-based partition for non-salient weights is confusing. It seems the authors aim to balance bits and performance (Section 3.4). However, the evaluation results show that the improved compression ratio and performance are due to n:m pruning, rather than the processing of non-salient weights. So, why should we partition the non-salient weights into three parts? Why not four or five? What do the terms dense, intermediate, and sparse mean?\n\n+  confusing evaluations\n\nWhile the experimental results of STBLLM are promising, the source of the accuracy improvements remains unclear. The experimental settings in the ablation study are somewhat confusing. For instance, Table 5 examines the effectiveness of the SI method in n:m pruning, but the results seem to represent 4:8 pruning plus binarization. What binarization method is used in the baselines? Table 8 directly compares STBLLM with BiLLM to illustrate the effectiveness of trisection partitioning, yet the pruning methods used in STBLLM and BiLLM are not the same (SI vs. Wanda). A detailed, step-by-step breakdown analysis of each technique's effectiveness would be helpful. Moreover, where does the 17x performance improvement come from when reducing 2-bit weights to 1 bit?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1zdIFBsqPB", "reviewer": "Reviewer_1cy6", "summary": "This paper proposes an efficient framework for LLMs, combining pruning and binarization to compress large, post-trained models. By applying N:M sparsity, it achieves precision below 1-bit and identifies salient weights through a newly introduced Standardized Importance (SI) metric. This metric considers both weight and activation values, avoiding the costly second-order computations typically required. Additionally, during pruning and binarization, the method separates non-salient weights into three groups to preserve as much information as possible in these parts. Extensive experiments demonstrate that the proposed method significantly reduces computational costs, accelerates inference, and maintains strong performance.", "questions": "+ Regarding Figure 3, part (b), after structured pruning, the empty parts should have no values. Why are zeros assigned to these parts? Additionally, structured pruning usually doesn't achieve weight-wise pruning, so what does \"structured\" mean in this context?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The paper is well-organized and easy to follow, with a clearly stated problem.\n+ It introduces a new metric to assess weight importance, avoiding expensive second-order gradient computations and mitigating the impact of extreme values.\n+ It is interesting that separate binarization for non-salient weights retains crucial information in this segment, enhancing model performance.\n+ The approach is logical and rigorous, discussing the method from various perspectives and fully validating its effectiveness through comprehensive experiments.", "weaknesses": "- In the zero-shot experiment, the paper mentions seven zero-shot tasks. It would be helpful to include a brief description of each task to provide readers with a clearer understanding of the evaluation scope.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2TlxIhI12x", "reviewer": "Reviewer_yy1T", "summary": "This paper presents a sparse and binarized compression method for large language models (LLMs), achieving an average bit count of less than one bit. Specifically, in terms of sparsity, a new metric matrix is proposed to represent the importance of different weights, along with a method for calculating the sparsity level for each layer based on this metric. This allows for effective sparsification of the model weights. For quantization, weights are grouped and binarized within each group, thereby reducing quantization error. Experiments on models such as LLaMA-1/2/3 demonstrate that this method achieves superior performance at higher compression ratios.", "questions": "- Regarding the calculation of average bit count:\n   - Could you clarify whether the overhead from indices associated with sparsity has been factored into the calculation?\n   - It would be helpful if you could provide a concrete example illustrating the calculation methodology, including both the weight bits and any additional storage requirements.\n\n- In Algorithm 1, there appears to be some ambiguity regarding the Semi-Structured function:\n   - Is this function performing sparsification based on SI?\n   - Neither the main text nor the appendix provides details about this function's implementation. Could you please elaborate on its mechanism?\n\n- The term \"OBC\" in Algorithm 1 requires clarification:\n   - While BiLLM mentions this as an abbreviation from another work, it would be beneficial to provide the full reference and explanation for completeness.\n\n- Regarding computational requirements:\n   - Could you provide an estimate for the computational time required for the 65B model, perhaps through theoretical scaling analysis?\n\n- In Figure 3, there appears to be an overlap between Salient Weight and Non-salient Weight distributions:\n   - Could you explain the underlying reasons for this overlap?\n   - How does this overlap affect the overall performance of the method?\n\n- Concerning Tables 5 and 7:\n   - There seems to be redundancy as Table 5 appears to be a subset of Table 7's Wikitext2 results. Could you justify the inclusion of both tables?\n   - The manuscript lacks discussion of Table 7's results, particularly regarding:\n     * Why does SI perform worse than SparseGPT on PTB and C4 datasets?\n     * What factors contribute to the different performance patterns across datasets?\n     * Could you provide insights into these performance variations?", "limitations": "", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This approach integrates sparsity with quantization, achieving a significantly higher compression ratio by reducing both the number of active weights and the bit precision required to represent them. The sparsity aspect not only reduces storage needs but also opens up additional acceleration opportunities, as sparse models can skip unnecessary computations, leading to more efficient inference.\n- A dedicated CUDA kernel was developed to optimize the performance of the sparse and quantized model on GPU hardware. This kernel was specifically tailored to exploit the structure of the sparse and binarized weights, enabling efficient memory access patterns and computation. The actual runtime performance of the model was measured using this custom kernel, providing a practical assessment of speedup gains achieved through the combined compression and acceleration strategy.", "weaknesses": "- While the proposed quantization methodology shows promise, the performance improvements over the baseline BiLLM implementation appear to be incremental. I would encourage the authors to further highlight the distinctive advantages of their approach and potentially explore additional optimization strategies to achieve more substantial gains.\n- The manuscript would benefit from enhanced clarity in several sections. Of particular importance is the need for a more comprehensive explanation of the average bit count calculation methodology. I suggest:\n  - Including a detailed step-by-step breakdown of the calculation process\n  - Providing specific examples to illustrate the computational procedure at inference time\n  - Clarifying how this calculation relates to the overall system performance on speedup or memory reduction", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)