HBLLM: Wavelet-Enhanced High-Fidelity 1-Bit Quantization for LLMs
Authors: Ningning CHEN
Weicai Ye
Ying Jiang
Venue: NeurIPS 2025 spotlight ()
Keywords: 1-Bit Quantization；Haar-based
Primary area: deep_learning
TL;DR: We introduce HBLLM, a wavelet-enhanced high-fidelity $1$-bit post-training quantization method for Large Language Models (LLMs).

Abstract: We introduce HBLLM, a wavelet-enhanced high-fidelity $1$-bit post-training quantization method for Large Language Models (LLMs). By leveraging Haar wavelet transforms to enhance expressive capacity through frequency decomposition, HBLLM significantly improves quantization fidelity while maintaining minimal overhead. This approach features two innovative structure-aware grouping strategies: (1) frequency-aware multi-parameter intra-row grouping and (2) $\ell_2$-norm-based saliency-driven column selection. For non-salient weights, a shared mean is employed across quantization groups within each frequency band to optimize storage efficiency. Experiments conducted on the OPT and LLaMA models demonstrate that HBLLM achieves state-of-the-art performance in $1$-bit quantization, attaining a perplexity of $6.71$  on LLaMA$2$-$13$B with an average weight storage of only $1.08$ bits. Code available at: https://github.com/Yeyke/HBLLM.

HBLLM: Wavelet-Enhanced High-Fidelity 1-Bit Quantization for LLMs
Ningning Chen chenm2276mai22.sy7 edu. cn
Weicai Yel cai_rcy@16e com
Ying Jiang! 1.2t
jiangy328mail. Sategy32mmailsyyuuu.uu edu. sysu.edu.co cn
Sun Yat-sen University
2 Guangdong Province Key Laboratory of Computational Science
Abstract
We introduce HBLLM, a wavelet-enhanced high-fidelity 1-bit post-training quanti- zation method for Large Language Models (LLMs). By leveraging Haar wavelet transforms to enhance expressive capacity through frequency decomposition, HBLLM significantly improves quantization fidelity while maintaining minimal overhead. This approach features two innovative structure aware grouping strate- gies: (1) frequency-awand multi-parameter intra-row grouping and (2) f--norm- based saliency-driven column selection. For non-salient weights, shared mean is employed across quantization groups within each frequency band to optimize storage efficiency. Experiments conducted the OPT and LLaMA models demon- strate that HBLLM achieves state-of-the-art performance in 1-bit quantization, attaining a perplexity of 6.71 LLaMA2-13B with average weight storage of only 1.08 bits. Code available at: http:///hhbbbooo/yoy///////BB..
Introduction
In recent years, Large Language Models (LLMs)
+W IP2E
RS-LIM
have achieved remarkable progress in natural lan- guage processing tasks. However, their massive pa- rameter sizes-often reaching tens or even hundreds of billions pose significant challenges for deploy- ment edge devices and in low-resource environ- ments. To reduce the computational and memory burden of these models, a variety of compression techniques have been proposed, including quantiza- tion 212 33 35), pruning [IT BI], and knowledge distillation 19 30). Among them, Post- -raanirann Quantization (PTQ) is widely adopted for its effi- ciency, requiring no additional training and having low deployment cost, especially in 1-bit quantization, which is considered a key approach for achieving extreme inference efficiency (13).
AZD LU
HELLU--Ontours)
Avg. Relative Perplexity
HBLLU--olierra)
225-
150
1.5
78
308
Model Size
Figure I: Average relative perplexity (normal- ized to FP16) on PTB, WikiText2,, and C4 for LLaMA-1 family models, comparing LLM binarization methods and our HBLLM.
Although existing 1-bit PTQ methods have achieved some success on base models such as GPT-2 and OPT, they tend to suffer from significant performance degradation or even complete failure-wher applied to more complex modern architectures like LLaMA3-SB [16). To address this, recent studies have introduced several strategies to improve quantization fidelity:
Equual contribution. Corresponding Author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Group quantization: divides the weight matrix into multiple groups for separate quantization. For instance, outlier-aware partitioning handles critical columns independently but can be constrained by partition design and scalability 15
Residual approximation: adds residual terms on top of primary quantization to partially recover errors (6). though this provides limited fidelity gains and introduces extra computation;
Low-Rank Adaptation (e.g., Lo inserts low-rank modules to absorb quantization errors with some flexibility, like 34 but often shows sensitivity to hyperparameters:
Global orthogonal transformations: apply global rotations in 22 [ before model compression
to enhance representational capacity, but require expensive inverse transforms (e.g., matrix mul- tiplications (d²)) complexity for d-dimension linear layer), leading to increased inference latency and energy consumption, making them impractical for deployment.
To overcome the structural trade-off between expressiveness and efficiency, we propose novel 1-bit This method is the first to integrate localized orthogonal transformations (i.e., Haar wavelets) into Bil Mis-Lle quantization process. Combined with structure-aware group- ing, HBLLM significantly enhances expressive power under ultra-low bit budgets while maintaining negligible inverse transform cost and excellent compatibility with hardware-efficies inference.
PTQ Tramework-HBLL..
Our main contributions are as follows:
A localized orthogonal transformation mechanism: we apply a single Haar wavelet transform to decompose the weight matrix into high- and low-frequency components, improving binary expressiveness while reducing transform computation;
Frequency-aware multi-parameter intra-row grouping: we introduce intra-row grouping in the frequency domain to capture structural patterns;
l2- fg-norm-based saliency-driven column selection: we propose an & norm-based ranking method to retain key columns using saliency metrics, effectively reducing quantization error;
antra-frequency- -aan mean sharing: for non-salient components, introduce mechanism that shares the mean across groups within the same row and wavelet band, reducing storage without sacrificing fidelity.
We conduct extensive experiments on OPT 37). LLaMA family 32) of LLMs. Results show that HBLLM achieves state-of-the-ant performance under 1-bit quantization: Across language modeling tasks (C4, PTB, WikiText2),, the perplexity ratio between HBLLM and the original FP16 model re- mains within the range of 1.2 2.2, shown in Fig] outperforming the next- next-best methods by 33%-66%; On 9 zero-shot QA benchmarks, HBLLM retains 73.8%-88. 88.8% of the original model's accuracy; On modern architectures such LLaMA3-8B, HBLLM remains stable with performance collapse: Even with a lower average bit rate and memory usage than BiLLM and ARB-LLMRe 18, HBLLM outperforms both in overall task accuracy.
These results demonstrate that HBLLM significantly extends the applicability of 1-bit quantization, balancing extreme compression with high fidelity, and offers new paradigm for deploying large-scale language models efficiently.
Related Work
2.1 1-Bit Post- Fost-Training Quantization
1-bit PTQ has emerged as a critical promising solution for deploying LLMs under extremely low bit budgets. Representative methods such as BiLLM [15] adopt salient column separation mechanism, in which salient weights are quantized independently, while non-salient weights are grouped based on magnitude and quantized row-wise. ARB-LLMx [18] further introduces column- grouping and alternating refined binarization, achieving notable improvements in fidelity. Unlike [10] 34], BiLLM can accomplish PTQ tasks without intensive computation for knowledge distillation with multi-GPUs.
However, current methods face several key limitations: (1) They heavily rely on fixed thresholds or simple & bbsed heuristics for salient column selection, which are insufficient to capture sparse but significant activation outliers; (2) They fail to account for the structural asymmetry between and

---

column dimensions in weight matrices, limiting their adaptability to complex model architectures; (3)
They completely neglect frequency-dome domann information.
2.2 Evolution and Limitations of Grouping Strategies
To improve quantization flexibility and fidelity, some studies have proposed learnable or adaptive grouping strategies. For example, Mixture of Scales (17] introduces Mixture-of-Expent (MoE) mechanism to assign scaling factor groups, and OneBitGPT (34) uses frequency masks to control quantization range sensitivity, and AWQ identifies weights with the greatest impact model predictions only based on activation outputs. However, these methods are generally effective only on unstructured tensors, rely on fine- Ain-grrined distillation, and lack explicit frequency-dond -omann awareness.
In addition, existing grouping strategics [5
often apply uniform partitioning rules across the entire weight matrix, ignoring variations across different rows. This lead to degraded expressiveness when quantizing models with significant inter-row diversity.
[8]
2.3 Comparison Between Global Orthogonal Transforms and Local Wavelet Transforms
Orthogonal transforms have recently been adopted to improve LLM quantization. FrameQuant 1 and QulP [5] utilize orthogonal transforms to enhance fidelity, but inference with such global transforms incurs high overhead, requiring (d²) matrix multiplications [T] that cannot be fused into
linear layers, leading to increased latency and energy cost.
By contrast, local orthogonal transforms such the Haar wavelet 20 offer localized spectral sensitivity and have been widely applied in image compression, denoising, and edge detection 14 They can be efficiently implemented via lightweight local convolutions with negligible inference cost, making them well-suited for low-bit compression and edge deployment.
3 HBLLM: A Quantization Framework with Wavelet Transform and Frequency-Domalle Grouping
3.1 Motivation and Core Challenges
Current mainstream 1-bit quantization methods face three key challenges in practice: (1) limited numerical expressiveness leading high reconstruction error; (2) insufficient accuracy salient column selection, failing to capture critical activation columns; (3) lack of structure-aware grouping strategies that adapt to heterogeneous model structures.
To characterize expressiveness under ultra-low bit settings, we introduce a new metric: the cardinality of the Inverse Quantization Set (CIQ), which measures the size of the discrete set of dequantized values within a row. CIQ serves as unified indicator of how the above challenges constrain model fidelity. acts both as a theoretical tool analyze the limits of existing methods and as empirical evidence of the advantage of our proposed method.
Under 1-bit quantization, the CIQ of BiLLM and ARB-LLMx is 8 and 10, respectively. When block size sets to 128, the CIQ upper bound of ARB-LLMx can reach 128. In contrast, our method achieves CIQ of up to 1024 after applying the Haar wavelet transform, significantly improving theoretical expressiveness. For more information on the benefits introduced by applying Haar transform, please refer the appendix and C
Based on aboved analysis, we propose: (1) Haar wavelet transform to enhance expressive capacity by frequency decomposition; (2) fo-norm-basse
dalicney-driver column selection prioritize critical columns; (3) trequency-awarr multi callti-prammter intra-row grouping to capture structural patterns. We also introduce an feerraareueecc--eee -band mean sharing strategy and local convolution optimization to reduce storage and inference cost, thus forming a 1-bit PTQ framework HBLLM.
3.2 Method Overview
We define the objective of HBLLM under the binary quantization setting for LLM weights. Specifi- cally, the quantization targets the full-precision weight matrix WTP E R where binary diagonal mask matrix M 10, 1] indicates which columns selected salient. The salient and
3

---

Preparation Salient Column Selection
Non-Salee San--at Region Quantization
Salient Region Quantization
W
X
Col-HQuant
sal
Wann-sal
FillAvg
W filled
Row-HQuant
H
W non-sal
OR
Col-HQuant
Wgon
Row-Haar Quant
J
Col-Haar Quant
Wren Wg
Overview of our HBLLM.The HBLLM quantization process consists of four steps: prepa- ration, salient column selection, haar transform and quantization for the non-salient part, and quanti- zation for the salient part. Since the salient columns excluded from the Haar transform of the non-salient part, their positions must be filled before performing row-wise Haar transforms. This is handled by process refer FillAvg. where each missing column is filled with the average of its adjacent non-salient columns. For the non-salient part, HBLLM supports flexible choice between row-wise (HBLLL-row) and column-wise (HBLLM-col) transforms. The salient part undergoes column-wise Haar transformation followed by HaarQuant for quantization.
Figure 2:
non-salient parts are quantized in the Haar domain, and their respective quantized Haar coefficients denoted by Wasl and W Wonon-sll These are then reconstrueted using inverse Haar transforms H and H²!!
The reconstruction objective of HBLLM is twofold. For the quantization of matrix layer W, the objective expressed in the Frobenius norm is formulated as:
min w
WX
-xX F'
where X is the input of the matrix layer. For quantization of matrix block WFP of W, the object is:
min Ma. Wasl Wineo
WFP M
W
[I - M,,) Wronn-sal B
When H, H2 are fixed Haar transforms, this formulation simplifies quantization problem entirely in the Haar domain. In this case, the objective is the same to that of BiLLM. Layer-level quantization is commonly tackled with the GPTQ algorithm [12.
We emphasize that our approach does not aim to solve this objective function via explicit optimization. Instead, this formulation serves as a conceptual framework that guides our method design. The actual quantization process is based on a set of heuristics and structure- strategies that approximate this objective computationally efficient and scalable manner.
Quantization Pipeline Overview. HBLLM integrates the Haar transform into a BiL.MM-style quantization pipeline (see Algorithm I and Figure 2) consisting of the following key steps:
1. Preparation Phase: Compute the column-wist importance scores using Hessian-based saliency metric.

---

2. Salient Column Selection and Quantizatio(SALLEEN))
Sort columns by their & norm significance.
Select top-K salient columns and determine M
Wasl HaarQuaa Quant (Mal
Choose the subset with the lowest quantization error.
3. Non-Salient Region Quantization:
Fill the missing values in salient columns using adjacent averages (FillAvg).
Warn-sal HaarQuant (Maal W filled + (I - Maa) W), where Willed is from FillAvg.
4. Adjustment and Refinement:
W = Maal W
-H-1
Wnon-ssl
W
HaarQuant
w
Algorithm Framework of HBLLM: Details of each function shown in Algorithm E.
fune (BLLLMW,X,,B1)
fune Row Now-HaarQuamn W rows(-})
Input: W E R= weight matrix
1: W Allled <- FillAvg( (W. i.jerows, rows
X gxxd calibration data
2: Balled &- HaarQuant( W MiledRRM
3 block size
3: W x- W Billed
À hessian regularizer
4: Bsalient HaarQuant W :fforos: COL)
Output: B haared binarized weights
B <- Baalien + Biiled
1: + 2XX
l error hessian matrix
6: return
2: H +- Chollsky(HH (I)) 11)
func Col-HaarQuant W rows(.))
3: B <- --0nxm
1: Bunsalient & HaarQuant W :.eemws COL)
4: for 0, 3,23,...... do
W W Bulled
5: W +- W 1bb+0
3: Baatient <- HaarQuant W :.eooss: COL)
6: rows(-} + SALIENT(W 16b43, H
4: B &- Bratient + B unsalient
7: if Row-HBLLM then
5: return B
8:
B. MBb+ < Row-HaarQuant w (-w)(.))
9:
else Col-HBLLM then
10:
B.b+ - Col-HHarQuant W r.)ws(.))
11:
E + .bbbb - B.bbbb+ H 8,
12:
W. Wb++0 + W.++3: E H 0843,004
13:
return B
3.3
HaarQuant: One-Bit Quantization in the Wavelet Domain
To boost expressiveness, apply Haar wavelet transform to the weight matrix of linear layers, generating frequency-domaii coefficient matrix, followed by group-wise 1-bit quantization.
To address limited numerical expressiveness, HBLLM introduces the HaarQuant algorithm. Haar- Quant consists three stages.
Haar Transform. A row of weights W is decomposed into low- and high-frequency coefficients via ID Haar transform Ht:
H (W)- = [Hoowwppsss (W). W Hnigh-pass (W)).
where Wi the Haar coefficient of W, Hoowppss (W) and Hnigh (W) are low- and figh-freeuency coefficients, respecively.
Frequency-Aware Multi-Pann Hulti-Parameter Intra-Row Grouping. For each row, boundary candidates determined by the row are enumerated, and the best grouping with minimal quantization error is selected. Furthermore, we split the rows by frequency bands. This adaptive strategy captures intra-row structural differences better than global uniform boundaries used in BiLLM.
5

---

Coefficient Quantization. Each group W is quantized using sign base- binarization centered on its mean:
W B sign FT (4);
where E R is the row-wise scaling factor and j is the group-wish wise mean and WB y is the result.
3.4 Structure-aware Grouping Strategies
To enhance the fidelity and adaptability of binary quantization under structural constraints, HBLLM introduces two structure-aware grouping strategies that operate along both column and row dimensions of the weight matrix.
Saliency-Driven Column Selection via & Norm. This strategy is used during salient column identification and quantization to overcome the limitations of prior heuristics based on fixed thresholds simple magnitude criteria.
Columns are ranked by their f2-norm scores, which correlate with their overall contribution to activation magnitude.
The top- K columns selected as salient and quantized in the Haar-transformed domain using column-wise transforms.
This approach helps preserve activation-critical directions, especially those dominated by outlier weights.
Frequency-Awan Multi-Parameter Intra-Row Grouping. This strategy is used during Haar domain quantization, where conventional row grouping lacks sensitivity to structural variations in weight distributions.
Each row is first decomposed into high- and low-frequency components based on Haar subbands.
Within each frequency band, coefficients are adaptively split into dense and sparse groups using band-specific, data-driven thresholds.
This grouping effectively doubles the number of quantization subgroups per row, enabling finer granularity and better error control.
Together, these strategies facilitate fine-grained, structure-preserving quantization across both dimen- sions of the weight matrix. To further guide saliency-based partitioning, we adopt the parameter importance metric used in BiLLMM, defined as: 8; = w/ ( 12, where H denotes the Hessian matrix of the layer, w; is the full full-precision value of the i-th parameter, and H 1] is the i-th diagonal entry of the inverse Hessian.
This metric reflects the relative sensitivity of the loss to changes in each parameter: higher val- ues indicate greater influence on the model's output, and thus prioritize that weight for accurate reconstruction.
3.5 Intra-freqqeenc--and Mean Sharing
To reduce storage overhead, HBLLM shares a single mean value among 2 groups in the same frequency band within each row: /ssuredd = ø1tn2 E - Σド v5) not only reduces per- parameter storage by 0.25 bits, but also maintains accuracy even slightly improving downstream task performance. This optimization achieves trade-off between compression and accuracy, improving deployment viability.
3.6 Efficient Haar Implementation via Local Convolutions
Instead of costly matrix multiplication, HBLLM implements Haar transform using fixed local convolutions. There are only two predefined 1D kernels, [1/2, 1/2] and [1/2, 11/2), whose kernel size is 2. Furthermore, it can be hardcoded into the model for zero runtime initialization and no training or storage is needed. convolutional sliding window, while FrameQuant needs O(d²) operations. As a result, HBLLM significantly lowers inference cost and is ideal for edge deployment.
complexity comparison, HBLLM needs (d) operatons via

---

Table I: Comparison of perplexity and average accuracy across models and methods
LLaMA1
Perplexity
LLaMA3
Perplexity
AvgQAT
AugQAT
Siiee
Method
W-bits
C4
Wiki2
PTB
Size
Method
W-bits
C4
Wiki2
PTB
FullPrecision
16.00
6.71
5.68
35.80
65.62
FullPrecision
16.00
11.90
8.29
13.07
68,94
FramsQuant
2.20
10.89
9.96
104.7
56.19
FramoQuant
2.20
28.44
23.36
40.33
52.27
PB-LLM
1.70
90.19
113.4
830.0
35.71
PB-LLM
1.70
111.7
141.5
171.1
36.83
7B
1.09
44.85
369.3
40.01
8B
BiLLM
1.06
56.24
81.27
ARB-IJMX
1.09
22.80
24.70
240.5
45.65
ARR-L.MX
1.06
48.45
37.90
52.59
43,40
ARB-LLMac
1.09
15.13
13:45
155.8
52.23
ARB-LLMac
1.06
34.44
30.24
45.23
49.08
HBLLM-FOW
1.09
9.49
8.82
88.86
57.48
HBLLM-roW
1.06
20.09
16.18
22.83
54.80
HBLLLMccol
1.00
10.38
9.67
117.7
54,03
HIBLLM col
1.00
22.18
17.80
26.38
51.43
FullPrecision
16.00
6.24
5.09
25.36
68.09
FullPrecision
16.00
6.61
2.85
7.74
74.62
PrameQuant
2.20
8.79
7.84
50.69
60.69
FrameQuani
2.20
NAA
NIA
N/A
N/A
PB-LLMM
1.70
38.41
46.02
190.2
40.39
PB PB-LLMM
1.70
33.56
28.93
44.38
47.45
13B
BiLLM
1.10
13.93
14.99
69.75
50.80 89.89
70B
BiLLM
1.09
385.8
137.6
129.5
34.18
ARB-LLMx
1.10
NIA
N/A
NIA
N/A
ARB-LLMx
1.09
N/A
N/A
N/A
N/A
ARB-LLMec
1.10
10.68
10.19
43.85
MRB-LLMMe
1.09
12,80
10.24
12.76
63.90
HBLLM-row
1.09
7.62
6.68
34.94
62.57
HBLLM-FOW
1.08
10.87
8.08
11.44
56.45
HBLLM-col
1.00
7.77
6,98
37.62
61.25
HBLLM-col
1.00
13.69
9.09
14.26
55.89
PallPrecision
16.00
5.62
4.10
21.35
71.06
OPT
Perplexity
AvgQAt
Prame
2.20
7.35
6.32
28.69
65.13
Size
Method
W-bits
C4
Wiki2
PTB
PB-LLM
1.70
21.73
25.87
127.1
47.22
30B
BiLLM
1.11
10.27
10.55
41.76
58.07
FallPrecision
16.00
13:45
14.62
16.41
52.54
ARB-LLMx
1.11
NIA
N/A
N/A
NIA
ARB-LLMac
1.11
7.79
30.98
64.49
FrameQuant
2.20
24.29
27.15
30.45
44,48
HBLLMMccol
--ow
1.10
7.03
6.88
5,82
25.95
26.65
66.76
1.3B
PR.II.M
BiLLM
1.09
1.70
56.24
186.9
68.43
309.0
286.3
119.2
33.44
38.39
ARB-LLMx
1.09
43.23
53.55
67.96
41.42
PallPrecision
16.00
5.31
3.53
21.11
72.27
ARB-LLMsc
1.09
24.23
28.77
33.32
45.28
HBLLM-1Ow
1.07
19.30
21.68
25.34
46.35
FrameQuant
2,20
6.69
5.55
27.48
68.58
HBLLM-coo
1.00
21.92
24.08
27.28
44.70
PB-LIM
1.70
12.66
12.76
99.67
65B
BiLLM
1,10
9.26
8.58
41.93
62.05
FallPrecision
16.00
12.06
12:47
14.61
54.95
ARB-LLMX
1.30
N/A
N/A
NIA
N/A
FraneQuantt
2.20
17.86
18.24
22.60
49.58
ARB-LLMRC
1.10
7.48
6.47
29.14
68.53
HBLLM-roW
IIBLMM
1.09
6.28
5.26
5.07
24.11
30.38
69.18
67.83
2.7B
ARB-LLMx
PB-LIM
BILLM
1.10
1.70
1.10
42.92
30.02
165.
216.8
55.75
34.15
41.35
103.2
160.4
37.62
40.02
44,60
LLaMA2
Perplexity
ARB-LLMac
18.02
19.53
24.46
AvgQAT
HBBLLM row
1.09
15.70
16.85
19.54
48.80
Method
W-bits
Wikiz
M-col
17.28
PallPrecision
16.00
8.66
6.94
37.86
65.54
FallPrecision
16.00
10.68
10.86
12.73
58.95
FrameQuant
2.20
14.66
13.34
177.1
52.75
FrameQuant
2.20
14.53
14.59
18.71
53.77
PB-LLM
1.70
63.95
55.40
486
36.54
PB-LLM
1.70
122.9
206.7
222.3
34.87
7B
BILLM
1.08
33.97
31.38
373.0
42.11
6.7B
BILLM
1.11
39.96
54.91
90.10
37.40
ARB-LLMX
1.08
26.55
21.74
314.2
45.41
ARB-LLM,
1.11
19.39
19.50
24.78
49.79
ARB-LLMMR
1.08
17.87
15.85
462.2
46.71
ARB.LLMMc
1.11
14.29
15.16
17.92
53.76
HBLLM-roW
1.07
11.75
10.52
89.23
57.74
HBLLM -row
1.10
12.56
13.04
15.26
56.17
HBLLM
11.33
150.6
54.09
HBLLLM-co col
1.00
13.29
13.67
15.70
FullPrecision
16,00
6.18
4.88
43.02
69.18
FallPrecissio
16.00
10.16
10.13
11.89
58.41
FrameQuant
2.20
9.40
7,80
109.3
61.35
FrameQuant
2.20
12.26
12.51
14.59
55.42
PR-LIM
1.70
313.4
289
934.4
32.91
PB-IIM
42.89
94,98
13B
BiLLM
1.08
22.17
19.57
303.4
46.76
13B
BiLLM
1.13
17.01
18.34
21.56
49.82
ARB-LLMx
1.08
N/A
NA
NIA
NIA
ARB-LLMs
1.13
N/A
N/A
N/A
NIA
ARB LLMBc
1.08
11.90
10.98
151.8
57.35
ARB- LLM
1.13
12.60
13.14
15.14
55.35
1.07
7.82
6.71
61.75
63.61
HBLLM-mo
1.12
11.47
11.72
13.78
55.91
HBLLM-col
1.00
8.28
7.00
69.77
62.04
M-LLM
1.00
11.71
12.34
14.13
55.66
FollPrecision
16,00
|
5.24
3.32
21.49
|
72.96
FallPrecissin
16.00
9.60
9.56
11.50
62.09
FrameQuant
2.20
NIA
N/A
N/A
NIA
FrameQuant
2.20
10.92
11.15
13.25
59.62
PB-LLM
1.70
N/A
NIA
54.26
PB-LLM
21.60
28.62
45.63
46.14
70B
BiLLM
1.09
15.57
15.86
71.03
55.81
30B
BiLLM
1.06
13.43
13.44
16.66
54.22
ARB-LLMXx
1.09
N/A
NIA
N/A
ARB- A.B.LLMx
N/A
N/A
ARB-LLMgc
1.09
7.26
6.00
28.43
68.77
ARB-LLMBc
1.06
11.18
10.94
13.27
58.59
IBLLMM row
1.08
6.18
4.82
24.69
70.01
HIBLIM row
1.06
10.41
10.13
12.58
60.04
HBL.M-col
1.00
6.63
5.04
26.31
68.61
M-co
1.00
10.53
10.29
12.75
58.91
Note: All methods are calibrated on C4 with 128 samples and a sequence length of 2048, A block size
128
is used for channel-wise quantization, as commonly done in prior work. N/A: ARB-LLMy method cannot run on single 3090 GPU 24GB. W-bits is the average weight overhead per weight. For more details, please refer to the appendix
7

---

Experiments
4.1 Experimental Settings
Models and Evaluation Datasets. In our study, we evaluate HBLLM on various models, including those from the OPT, LLaMA-1, 2LaMA-2, and LLaMA-3, well the recently introduced BRI-Distill-LJam--81 Specifically, we utilize the OPT models with 1.3B and 2.7B parameters, the LLaMA-1 and L-LMA-22 models with 7B and 13B parameters for evaluations, and the LLaMA-3 model with 8B parameters. We measure language modeling capabilities by evaluating their perplexity the C4(26), Wiki'Tex12 22 and PTB datasets. Additionally, we assess zero-shot accuracy on various Common Sense Reasoning Tasks such as PIQA(4), BoolQ OpenBookQA [23], WinoGrande(28), ARC-e, ARC-c(8), HellaSwap wag 36), which are commonly used for evaluating the performance of LLM quantization methods. To further enhance evaluation coverage, we also include COPA (7) for causal reasoning and LAMBADA (25) for long-context language modeling. All evaluations are conducted using the open-source LLM evaluation framework, Conaluation-Harness 24).
these models
Details of Experiments. All experiments are conducted with PyTorch on NVIDIA GeForce RTX 3090 GPUs with 24GB of memory. For the calibration data, follow the settings adopted in GPTQ and BiLLM, selecting 128 samples from the C4 dataset, with a sequence length of 2048. During quantization, we set the block size 128 in BiLLM, PB-LLM, ARB-LLM, and HBLLM. Activations are kept in full precision (FP16).
Baselines. We compare HBLLM against several state-of-the -ae--o--theerrtt art LLM binarization methods, including BiLLM, ARB-LLM and PB-LLM, ensuring that all implementations adhere to the details provided in their respective papers. BiLLM, L and PB- PB-LLM all utilize the PTQ approach for model calibration through OBQ based method of GPTQ. For ARB-LLM, we evaluate two of its best- performing variants, ARB-LIMX and ARB-LLMRc. Both ARB-LLM, and ARB-LLMMc employ the salient column bitmap and group bitmap (CGB) for better performance. For PB-ILLM, which allows variable ratios of salient weights enhance accuracy, have the ratio of salient weights 10% to ensure the average bit width of weight parameters remains below 2 bits. Given the sigmificant accuracy improvements demonstrated by HBLLM over traditional binarization techniques, we also include a comparison with a leading method using orthogonal transforms: FrameQuant. For Frane quantization is performed not in the original weight space but in the structured orthogonal basis constructed through Fusion Frames. We evaluate two configurations: FrameQuant (r 1.0) and FrameQuant (r = 1.1), where the redundancy factor controls the amount of redundancy introduced during the transformation.
4.2 Perplexity and Accuracy Results of 1-2 Bit Quantized Models
The perplexity and zero-shot accuracy results of previous 1-2 bit quantization methods and the proposed HBLLM are presented in Table[] HBLLM consistently outperforms existing 1-2 bit quantization techniques across all evaluation metrics.
Specifically, HBLLM reduces the language modeling perplexity by 33%-66% compared to previous methods, while achieving substantial improvements QA task accuracy, with relative gains ranging from 73% to +1 1..3%. In our experiments, HBLLM slightly outperforms FrameQuant, a 2.2- bit quantization method, and exhibits a particularly significant advantage on the LLaMA-3-8B model. Moreover, when compared with BiLLM and ARB-LLMx. HBLLM-col, demonstrates a clear advantage in both perplexity and accuracy, despite operating at comparable lower bit-widths. These results indicate that HBLLM effectively narrows the performance gap between quantized models and their 6 counterparts, achieving 1.22x to 2.48x of the original perplexity and retaining 13.8%-88.88 88.8% of the original QA accuracy.
4.3 Ablation Study
Salient Column Selection Criterion. To evaluate the impact of selection criteria salient column screening quantization effectiveness, we compare two strategies: the column & norm and the column Ea norm as significance indicators. Experimental results in Table 2a reveal that the column & norm consistently achieves lower quantization error and superior performance in downstream tasks,
8

---

Table 2: Ablation study on LLaMA2-7B. Results are measured by perplexity, with final results highlighted in bold.
(a) Suidy of salient column selection criterion
(b) Study grouping granularity
Selection eriterion
Group Partition
Method
Wiki2
PTB1
Method
Wiki2,
PTB/
E
10.78
143.7
global 16.32
1990
HBLLM- row
HBLLM -row
l
10.52
89.23
row-wise
11.08
95.58
&1 11.45
308.2
global
13.99
1546
HBIJLM-col
MBLLMMcool
l 11.33
150.6
row-wise
12.02
146.1
(c) Effectiveness of shared mean
(d) Study of partitioning candidates number
Shared mean
Candidate number
Method
Wiki2
PTB1
Method
Wiki2|
PTB]
X
11.08
95.58
10
11.16
108.8
HBLLM-row
10.52
89,23
20
11.32
165.8
HBLLM-row Mooo row
X
12.02
146.1
40
11.08
95.58
HBLL-M-col
11.33
150.6
80
11.13
113.8
indicating its greater effectiveness in capturing energy distribution across columns and enhancing quantization quality.
Granularity of Group Quantization. To explore the influence of grouping granularity on model performance, we compare global grouping with row-wise grouping strategies, evaluating both quantization error and perplexity, as shown in Table 2b The results reveal that row-wise grouping significantly reduces quantization error and achieves lower perplexity compared to global grouping. This suggests that finer-grained row-wise partitioning better preserves local data fidelity, leading to improved quantized inference performance.
Shared Mean Strategy. Under the standard dual-partition quantization setting, we further explore compression strategy that shares the quantization center across two partitions within each row. By unifying the mean for both partitions, the storage overhead of quantization coefficients can be significantly reduced. Experimental results in Table 2 demonstratee that the shared mean strategy even slightly reduces quantization eтror without degrading perplexity, verifying its effectivvnesss
Choice of Partitioning Number. We investigate the impact of varying the number of partition candidates on final quantization performance under the row-wise grouping setting. Specifically, for each row, we generate partition candidates based on absolute value percentiles ranging from 10% to 90%, and evaluate the corresponding quantization error and perplexity, as shown in Table 2d Experimental results indicate that moderately increasing the number of partition candidates can effectively reduce quantization error and further lower perplexity, while excessive partitioning yields diminishing returns and increases computational cost. Consequently, we adopt 40 partition candidates as the default setting to balance performance and efficiency.
4.4 Time and Memory Analysis
Time Comparison. As binary PTQ frame- work, HBLLM eliminates the need for finetun- ing, The introduction of Haar wavelet trans- forms requires additional computation during quantization, yet this overhead remains fully acceptable. As shown in Table 3, HBLLM in- creases the quantization time by approximately 20%-30% compared to BiL.MM across different model sizes. It is worth noting that ARB-LLMy and FrameQuant fail to complete quantization for LLaMA-1-13B and LLaMA-1-30B under the single-GPU-24 GB setting, while HBLLM successfully completes the process, demonstrating better scalability.
Table 3: Time comparison between LLM binariza- tion methods and our HBLLM on LLaMA-1 with different model sizes.
Method
7B
13B
30B
BiLLM
36min
71min
142min
ARB-LLM,
88min
X
ARB-LLMRC
76min
119min
239min
PB-LLM
18min
29min
57min
FrameQuant
14min
22min
HBLLM
44min
98min
173min

---

Memory Comparison. As shown in Table 4 HBLLM-col achieves better performance while occupying a storage size comparable to ARB- L.M. By employing grouped shared-mean strategy, HBLLM improves compression effi- ciency without sacrificing performance. Specif- ically, HBLLM-col applies Haar transforms along the column dimension, such that only one grouped quantization operation is required per row on the transformed coefficients. Compared to HBLLM-row, this leads to reduced data fi- delity but provides clear advantages storage cost. Notably, the reported memory usage is measured at runtime in our setup and may be influenced by model variants and implementation choices, leaving for further engineering optimizations. The detailed storage calculation formulas can be found in the appendix
Table 4: Memory comparison LLM binarization methods and our HBLLM on LLaMA-I I with dif- ferent model sizes.
Method
7B
13B
FP16
13.48GB
26.03GB
BiLLM
2.93GB
5.36GB
ARB-LLM,
3.23GB
5.95GB
ARB-LLMRC
2.83GB
5.17GB
PB-LLM
2.91GB
5.33GB
FrameQuant
11.36GB
16.08GB
HBLLM-row
3.29GB
6.07GB
HBLLM-col
2.86GB
5.22GB
4.5 Inference Latency Estimation
To evaluate the inference latency of HBLLM, we conduct an experiment that combines direct measurement with estimation. Due there is existing inference framework that fully supports the dequantization algorithm used in HBLLM, we test GEMV on layers from the OPT-175B model an NVIDIA P100 GPU following the GPTQ benchmark setup [1]. Our
instead. The tests estimation results show that the inference latency of HBLLM is approximately 31.8% of the FP16 baseline inference time. For details, please refer the appendix G
Conclusion
We introduce 1-bit weight only quantization HBLLM, which applies Haar transform to BILLM pipeline. Besides quantifying the coefficients on frequence domain, HBLLM integrates two innovative structure-aware grouping strategies enhance fidelity. Furthermore, HBLLM optimize storage efficiency. As results, HBLLM outperforms SOTA QAT quantization methods of LLM at 1-bit across different LLM families and tests. The current HBLLM supports only quantized dense models. Next, we will focus on the MoE PTQ algorithm.
Acknowledgements
We thank all constructive comments from anonymous reviewers. This work is partially supported by the National Key Research and Development Program of China under Grant No. No.2023YFB3000
References
[1] Adepu H, Zeng Z, Zhang L, et al. FrameQuant: flexible low-bit quantization for transformers. Proceedings of the 41st International Conference Machine Learning. 2024: 203-227.
[2] Ashkboos S, Croci L. Nascimento M et al. SliceGPT: Compress Large Language Models by Deleting Rows and Columns. The Twelfth International Conference on Learning Representations. 2024.
[3] Lin J, Tang Tang H, et al. AWQ: Activation weight quantization for on-device L.M compression and acceleration. Proceedingy of Machine Learning and Systems, 2024, 6: 87-100.
[4] Bisk Y, Zellers R, Gao J, et al. PIQA: Reasoning about physical commonsense in natural language. Proceedings of the AAAI Conference on Artificial Intelligence. 2020: 7432-7439.
[5] Chee J, Cai Y, Kuleshov V, et al. QulP: 2-bit quantization of large language models with guarantees. Advances in Neural Information Processing Systems, 2023, 36: 4396-4429.
[6] Chen H. Lv C, Ding L. et al. DB LL- Accurate dual- binarization for efficient LLMs. Findings of the Association for Computational Linguistics: ACL 2024. 2024: 8730.
10

---

[7] Clark C. Lee K. Chang M W. al. BoolQ: Exploring the surprising difficulty of natural yes/no questions. Proceedings NAACL-HILT. 2019: 2924-2936.
[8] Clark P. Cowhey I, Etzioni O, et al. Think you have solved question answering? Try ARC, the A12 reasoning challenge. arXiv preprint 1803.::800. 2018.
[9] Duan Y. Liu F, Jiao L, et al. SAR image segmentation based on convolutional neural network and Markov random field. Pattern Recognition, 2017, 64: 255-267.
[10] Edalati A, Ghaffari A, Nejad M G, ct al. OAC: Output-adaptive calibration for accurate post-training quantization. Proceedings of the AAAI Conference on Artificial Intelligence. 2025, 39(16): 16453-16461.
[11] Frantar E, Alistarh D. SparseGPT: Massive language models can be accurately pruned in one-shot. International Conference on Machine Learning. 2023: 10323-103371
[12] Frantar E, Ashkboos S. Hoefler T, et al. OPTQ: Accurate post-training quantization for generative pre-trained transformers. 11th International Conference on Learning Representations. 2023.
[13] Gong R, Ding Y, Wang Z, et al. A survey of low-bit large language models: Basics, systems, and algorithms, arXiv preprint arXiv:2409. 16694, 2024.
[14] Huang H. He R. Sun Z. et al. Wavelet-SRNet: A wavelet-based CNN for multi-scale face super resolution. Proceedings of the IEEE International Conference on Compuder Vision. 2017: 1699-
[15] Huang W, Liu Y. Qin H. et al. BiLLM: Pushing the limit of post-training quantization for LLMs. Proceedings of the 41st International Conference on Machine Learning. 2024: 20023-20042.
[16] Huang W, Zheng X. Ma X, al. An empirical study of LLLMA3 quantization: From LLMs MLLMs. Visual Intelligence, 2024, 2(1); 36.
[17] Jo D, Kim T, Kim Y. Mixture of scales: Memory-efficient token-adaptive binarization for large language models. Advances in Neural Information Processing Systems, 2024, 37: 137474-137494.
[18] Li Z. Yan X. Zhang T, al. Arb-LLM: Alternating refined binarizations for large language models. arXiv
preprint arXiv:2410.03123 2024.
[19] Liang C, Zuo S, Zhang Q. al. Less is more: Task-aware layer-wise distillation for language model compression. International Conference on Machine Learning. 2023: 20852-20867.
[20] Mallat S. A Wavelet Tour of Signal Processing. Elsevier, 1999.
[21] Marcus M, Kim G, Marcinkiewicz M A, et al. The Penn Treebank: Annotating predicate argument structure. Human Language Technology: Proceedings of a Workshop held at Plainsboro. 1994.
[22] Merity S, Xiong C, Bradbury J, et al. Pointer Sentinel Mixture Models. International Conference on Learning Representations. 2017.
[23] Mihaylov T, Clark P. Khot T, et al. Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 2018: 2381-2391.
[24] Mukherjee S, Liu X, Zheng G, et al. Few-shot learning evaluation in natural language understanding. Thirty-fifth Conference Neural Information Processing Systems Datasets and Benchmarks Track. 2021.
[25] Paperno D. Kruszewski G, Lazaridou A, et al. The LAMBADA dataset: Word prediction requiring a broad discourse context. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. 2016: 1525-1534.
[26] Raffel C, Shazeer N, Roberts A, et al. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 2020, 21 (144)) 1-67.
[27] Roemmele M, Bejan C A, Gordon A S. Choice of Plausible Alternaties:: An Evaluation of Commonsense Causal Reasoning. AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning. 2011: 90-95.
[28] Sakaguchi K. Bras R L, Bhagavatula C, et al. Winogrande: An adversarial Winograd schema challenge at scale. Communications of the ACM, 2021, 64(9): 99-106.
[29] Shang Y. Yuan Z, Wu Q. et al. PB- PB- Partially binarized large language models, The Twelfth Interna- tional Conference on Learning Representations. 2024.

---

[30] Shridhar K. Stolfo A. Sachan M. Distilling reasoning capabilities into smaller language models. Findings the Association for Computational Linguistics: ACL 2023, 2023: 7059-7073.
[31] Sun M, Liu Z, Bair A, et al. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306. 11695, 2023.
[32] Touvron H. Lavril T, Izacard G, et al. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13977 2023.
[33] Xiao G, Lin J. Seznee M, al. SmoothQuant: Accurate and efficient post-training quantization for large language models. International Conference Machine Learning. 2023: 38087-38099,
[34] Xu Y, Han X, Yang Z, et al. OneBit: Towards extremely low-bit large language models. The Thirty-eighth Annual Conference Neural Information Processing Systems. 2024: 66357-66382.
[35] Yao Z, Yazdani Aminabadi R, Zhang M, et al. ZeroQuant: Efficient and affordable post-training quan- tization for large-scale transformers. Advances in Neural Information Processing Systems, 2022, 35:
271 27168-27183.
[36] Zellers R, Holtzman Bisk Y, HellaSwag: Can machine really finish your sentence? Proceedings of the 57th Annual Meeting of Association for Computational Linguistics. 2019: 4791-4800.
[37] Zhang S. Roller S, Goyal N, al. OPT: Open pre-trained transformer language models, arXiv preprint arXiv:2205.0Im 00068, 2022.
12

---

A Introducing Haar Wavelet Transform into LLM Quantization
Haar wavelet transform converts original vector into matrix containing low- and eegefrreeency information by computing averages differences. This tramsformation can extended to larger vectors and multidimensional data and is commonly used in image processing and signal analysis.
Applying the Haar wavelet transform LLM quantization offers three advantages:
The inverse quantization set becomes richer in representation, which can be demonstrated via the CIQ metric.
The data distribution becomes more concentrated: with approximately 65% probability, the variance of the high- and low-frequency coefficient sets in each
smaller than before the transformation.
The additional computational cost for inference is Odd), where d is the input length, the Haar transform be implemented using local convolutional layers, resulting in lower cost than methods such as FrameQuant.
A.1 Definition of Haar Transform
Let the one-dimensional input signal be z. output after wavelet transformation
f = [01, bi...., bul.
(5)
where a, denotes low-frequency coefficients denotes high-frequener coefficients.
This process can also expressed matrix form:
x² H(x)) = Udug
(6)
where
U
U
Uding =
(7)
Because the Haar matrix is orthogonal, the inverse Haar transform is:
U T
(8)
dag
A.1.1 Row-wise and Column-wise Haar Transforms on Matrices
Define row-wise column-wise Haar transforms matrix as:
How (W)=
WUT Hosl (W) Unag W.
(9)
Below simple example showing how apply Haar transforms matrix. Given 4 4 matrix defined
18
22
12
14
10
8
A
(10)
24
26
30
28
20
22
18
16/
result after applying row-wise Haar transform denoted as
(17
-1
21
1
Arow v2x
13
-1
(1I)
21
and after applying column-wii Haar transform is denoted as
15
A v2x
2
0
22
(12)
15
0
23
-2
0
-2
where each element of matrices is multiplied the coefficient v2.
13

---

B Relationship Between Richness of Inverse Quantization Set and Model Fidelity
B.1 Definition and Application of CIQ Metric
The cardinality of an inverse quantization set (CIQ) measures the number of distinct values that can be recovered from quantized weights in cach For lincar quantization without partitioning. CIQ equals the bit- width of quantized weights, since linear quantization evenly distributes several points When partitioning strategies are applied, it characterize expressiveness of the quantization algorithm.
the original data range.
HBLLM, which follows the GPTQ quantization scheme, quantization done per row each matrix block. We define CIQ in terms of a single row of a matrix block in the following discussion without loss of generality. Furthermore,
CIQ min fow length, maximum recovery ability under given quantization parameters).
holds.
To study the composition IQ, introduce mapping from the quantized weights the set dequantized weights. Several mappings from quantized weights to inverse quantized values exist:
Identity mapping
Group merging mapping
Residual merging mapping
Inverse transformation mapping
For example, inverse quantization brought by residual merging mapping is defined
1Q ressdall
inViesidull
(X1, Xx) = { (x + v), X1, Vy X2).
(13)
Furthermore,
CIQ residual
S X X2]:
(14)
where | is the cardinality of set.
An inverse quantization set brought by Haar inverse transformation is defined
IQ
inVitar
(Xim, Xtigh)
12:2
(ar y) or
V2 (ar v). Vr E Xim, Vy Xtigh
(15)
And then,
CIQ taars
S 2/
[
(16)
Let inverse quantization set produced by BiLLM algorithm be denoted IQ According BiLLLM algorithm, TQBILLM he expressed in the following form:
IQ
IQui
IQ non-aal
(17)
inVresidual
X X
son
where IQual Iisstttol and (Quee-sal represent inverse quantization sets of the salient and non-salient parts, respectively.
Lemma 1. BiLLM has at most 8 different dequamtized values row.
Proof. According to and the definition of CIQ, we can infer
CIQUILLM S CIQ
(18)
Since salient part adopts residual approximation strategy, follows
CiQui Wiitial
S X
X2 ssl
(19)
Given that X = |
Xmenssal
2, substituting
19 into
18) yields:
CIQ
x
x
x
000-u1
x non-sal
= x 2 8.
(20)
14

---

Next, analyze theoretical upper bounds of CIQ for HBLLMM algorithms. We first consider the case of HBLLMMccol. Let inverse quantization produced by HBLLM-col be denoted as HUBLIM out According HBLLM-col algorithm, IUULLIM-col can be expressed the following form:
HBLILMaa = IQ U H
non
(21)
iilV Hazs
X
U UnVVHar
Ligh
where IQ and Hiar parts employ group quantization strategy under Haar transform. so the upper bound for each part is the product of the cardinality of the two Haar sub-band quantization sets. Additionally, within HBLLM-col algorithm, each sub-band has two groups. resulting a total of quantized values. Based the above analysis, we arrive at the second conclusion.
ronsaal represent the dequantized sets of the salient and non-salient parts, respectively. Both
Lemma 2. HBLLM-col has at most 64 different dequantized values per row.
Prouf.
CIQHELLM 2
Fasa low
Sesal high
+ 2
Tenon sal low
X
non sal
high
64.
(22)
The CIQ upper bound of HBLLM-row algorithm is significantly larger than that of HBLLM-col algorithm. Let inverse quantization set produced by HBLLM-row algorithm be denoted THBLLMMOW By HBLLM-row algorithm, IQ IQHBLIM.OW can shown following form:
1BLLM
1
Q
Taar
son-sal
(23)
where IQ row is defined
IQ FBLL
(IQ Thur
IQ Coa--sal
Haar
(24)
Unlike HBLLM-col the non non-salien salient part of HBLLM-row encompasses the entire matrix area, resulting in overlap with the salient part. Therefore, HBLLM-rot incorporates residual approximation on the salient part, further increasing the CIQ upper bound. Additionally, within HBLLM algorithm, each sub-bant has two groups, resulting total of four quantized values. Based on the above analysis, arrive the third conclusion.
Lemma 3. HBiLLM-row can have over 1024 different dequantized values per row.
Proof.
CIQ
sal HBLL M-eow
IQ
IQ
mon-sal
inVi
X
low
Xaal High
x inv
Samnssal high
(25)
2 sall
Scal high
x 2 Rann
Snun-sal
high
1024.
CIQ CIQHBLIM-RWW
< 1024 32
1056.
(26)
Lemma is result that holds under the assumption that there sufficiently many columns in the salient part. practical algorithms, due the dual constraints of the quantized matrix size and the total hitrate, the upper hound of HELLM is much less than 1024. The specific upper hound can he described in Lemma
Lemma 4. Let the size of a quantized matrix block be d d, where d < 256, and let the proportion of the number of colunns the salient part to the total number of columns be p (where 0 p < 1). Then, we have
CIQ CIQUILLM. S p.d.
(27)
and
CTQ HILLE 32 min(p d,32).
(28)
Proof. It is easy to get by Lemma and Lemma
Theorem 1. Under the same proportion p of the salient part. CIQ
HILLLM row CIQ cut holds.
Proof.
This follows from Lemma 4
15

---

Distrouttion inverse quanbzation Values (Unique: 421
(a) HBLLM-row
Uistrizution quartication (Uriquee:
oac usar ose 110 1266
(b) HBLLM-col
x10-2
Figure B.I: CIQ and Distribution of inverse quantization Values
B.2 Measured Values of the CIQ Metric for the HBLLM Algorithm
To validate theoretical analysis of CIQ. we present two representative examples under MBL.LM-row and HBLLM-col schemes, respectively. These examples illustrate how the CIQ values measured in practice align with the theoretical bounds derived earlier.
Example 1: HBLLM-row. As shown in B.la consider row of length 128 where 14 elements are marked salient. After quantization and reconstruction, the total number of distinct values observed is 42. Among these, 30 values originate from the non-salient part (i.c., [IQ Hlazz 30) and 7 values from the salient part (i.c.,
[IQ Hanz 7).
Although the salient part involves residual merging, the final reconstructed set still satisfies:
CIQ 42 30 14 44,
(29)
which confirms that the practical CIQ value stays within the theoretical upper bound in Lemma
Example HBLLM-col. shown in Bblb second example under HBLLM-col scheme, row consists non-salient and salient elements. The measured CIQ is 23, with values from the non-salient part and 7 values from the salient part. This result again satisfies:
CIQURIIMON 23 < 16
25.
(30)
These observations demonstrate that in practice, the effective size of the inverse quantization set significantly below the worst-case bounds, especially when some quantized values are shared or overlap. They also confirm the effectiveness of the HBLLM decomposition strategies in maintaining a compact and expressive representation of quantized weights.
B.3 Limitations of CIQ Metric Analysis and the Necessity of Introducing Structure-Aware Grouping Strategy
Although HBLLM significantly outperforms BiLLM in the CIQ metric. it lacks a clear advantage in quantization performance without the structure-aware grouping strategy introduced in this paper.
To assess the performance differences before and after implementing this strategy, we conduct experiments evaluating perplexity and QA accuracy. Experiment data he found in Table B.I.
BiLLM+ Employing ba-ba salicncy-driven column selection together with multi parameter intra-row grouping.
HaLM: This refers method obtained by removing the f--nor based saliency-driven column selection and marti-parramttrr intra-row grouping strategies from HBLLM. This method integrates the one-dimensional discrete Haar wavelet transform applied row-wise or column-wise, thereby deriving two approaches: Row-Haar+Bil LL and hol-Harr-Bii.IMM. respectively.
DCT+BiLLM: This approach applies the BiLLM algorithm to coefficient matrices obtained from the one- dimensional Discrete Cosine Transform (DCT) applied row-wise column-wise on weight matrices, resulting Row-DCT+BiLLM Col-DCT-BiLLM. Unlike Haar+BiLLM method, DCT+BiLLM uses global transformation strategy by first mapping the entire matrix to the Fourier domain before quantization. In contrast, Haar+BiLLM, applics the Haar transform to matrix blocks, with quantization following the BillM process, making it local orthogonal transformation.
HBiLLM+: These method. refer those derived from Haar+BiLLM, utilizing strategies proposed our paper. They include HBLLM-row+&s, HBLLM-col+&s, HBLLM-col+(, and HBLLM-col+!!.
16

---

Table B.1: Perplexity (L. C4, Wiki2, PTB) and AvgQA accuracy (f. AvgQA over zero-shot tasks) of BiLLM variants with Haar.
OPT-1.3B
LLaMA2-7B
Method
AvgQAt
AvgQA1
C4.
Wiki24
PTB.
C41
Wiki2
PTB.
BiLLM
56.24
68.43
119.2
38.39
33,97
31.38
373.0
42.11
BiLLM+&2
55.95
72.42
105.9
37.95
33,46
31.34
695.8
41.11
B
56.88
70.48
92.16
39.28
28.17
25.08
226.3
41.77
Row-Haar+BBiln BiLLM
47.45
52.81
62.81
39.57
25.77
25.12
138.0
44.67
Hol-Haar+BiLLM
95.56
128.8
171.3
36.92
41.03
37.25
5193
39.60
BOw-DCT+BiLLM
8010
11517
6729
31,36
45358
49395
26888
34.48
Col-DCT+BiLLM
107.1
150.5
250.1
34.19
26.54
24.64
1202
44.82
26.47
33.68
41.17
41.17
16,26
19.86
87.90
47.90
HBLLM-col+
26.37
29.99
36.24
42.13
15.04
13.99
154.6
49.38
IBLLM-row+e)
19.55
26.95
25.70
46.87
13.00
13.20
85.50
50.86
HBLLM-col+
21.98
23.69
27.39
45.21
13.18
12.02
146.1
51.34
Note: &z denotes activation of Ez-norm-based saliency- driven column selection; f denotes activation of frequency-awarr aware multi-pu parameee intra-row grouping.
main experimental results are summarized follows:
Directly applying Haar transform into BiLLM pipeline does not significantly improve 1-bit quantization performance.
Now-Haar-H
HaLLL BiLLM shows slight improvement.
Col- Col-Haar++Bl BiLLM decreases performance.
Implementing the 'aliency-driven column selection via & norm' strategy leads
Significant improvements for HBL.M-row-f2 and HBLLM-col++ compared to their predecessors.
Perplexity tests show:
32-64% reductions on C4 and Wiki2 test sets.
HBLLM-col+ shows notable improvements, but still lags behind BiLLM on the PTB set.
QA testing accuracy improves by 3-10%.
Further introducing ffrequency-awar multi-parameter intra-row grouping' results in:
-IBLLM-row++ wise--rpgggg the best.
26-45% reductions in perplexity on C4 and Wiki2.
Significant improvements the PTB set, surpassing BiLLM.
Cumulative accuracy QA testing increases 2-8%.
This experimental result demonstrates that introducing a structure-aware grouping strategy essential for effectively combining the Haar transform with the BiLLM algorithm.
B.4 Effectiveness of Haar Transform and the Importance of Local Orthogonality
As shown in Table although HBLLM integrates multiple strategics, is important to disentangle the specific contribution of the Haar-based frequency dccomposition from other components such as saliency selection and structure-aware grouping. To this end, we conduct dedicated ablation studies to quantify the standalone effectiveness of the Haar transform and contrast it with global orthogonal alternatives such Discrete Cosine Transform (DCT).
We summarize our key observations below:
While incorporating either the bz-based saliency selection or the structure- aware grouping alone yields only modest improvements BiL.IM, introducing the Haar transform leads to consistently substantial gains in both perplexity and QA accuracy. Notably, under partial activation (e.g.. HBBLLM-co++$5 the models outperform their BiLLM counterparts.
Effectiveness of Haar Transform:
Both HBLLMM Morww+82 and HBI.IM-col+f.) significantly outperform BiLLM BiL.LM+ underscor- ing the crucial role of Haar in preserving the rtequency-domate structure weights.
Now-Haar+Bii L,,LM,, even without & nord--bbeedd saliency-driverr column selection or grouping, shows consistent performance gains, confirming that Haar decomposition independently enhances quantization representation capacity.
17

---

Local vs. Global Orthogonal Transforms: We fürther analyze the impact of replacing Haar transform with global DCT.
Applying the global row-wise DCT results in severe degradation across all benchmarks, with Row- DCT+B performing significantly worse than BiL.M.
Applying the global column-wise DCT offers moderate improvement on LLaMA2-7B; however, Col- LDC still lags behind BiLLM in all OPT-1.3B tests.
These results highlight that global transforms struggle to capture local variations in weight distributions. which effectively preserved by block-wise Haar decomposition.
As shown in Table B.2 global transforms such as DCT also incur substantial computational overhead compared local Haar transtorms:
Table B.2: Time comparison between BiLLM, DCT+BiLLM, and HBLLM on LLaMA-I with
different model sizes. The DCT implementation used in this test is from pytorch.
Method
7B
13B
30B
BiLLM
36 min
71 min
142 min
DCT+BiLLM
211 min
414 min
1012 min
HBLLM
44 min
98 min
173 min
conclusion: Haar transform independently and robustly contributes to quantization fidelity, even the absence of auxiliary strategies such as f2-norm- saliency-driven column selection or grouping; local orthogonal transforms like Haar are consistently more effective than global ones like DCT in preserving localized peqquencydommate structures-an essential property for stable and expressive 1-bit quantization.
C Analysis of the Correlation Between Data Concentration and Model Fidelity
In this section, we explores the positive correlation between improved data concentration and enhanced model fidelity after quantization, following the application of the Haar transform and structure-aware grouping strategy proposed by HBLLM. This correlation provides theoretical foundation for the effectiveness of the HBLLM quantization method. We first observe that the concentration of cocfficient distribution improves with 65% probability after applying Haar row transform. Based on this observation, we mathematically model probability of variance improvement in data concentration. We then apply this variance improvement probability to the HBLLM-row and HBLL.Mcol methods to validate the correlation between enhanced data concentration and improved model fidelity after quantization.
C.1 Improvement of Data Concentration by Haar Transform
We discusses the improvement in data concentration resulting from the Haar transform, which is typically deseribed by variance porance-lowwr variance indicates higher data concentration. We examine the impact of the Haar row transform on distribution characteristics the weight matrix. Specifically, we select a matrix block from the OPT-1.3B model and compare the variance each original weight vector with the variances of the low-frequency and high-feqqency subhands ohtained after Haar decomposition. The variances of each row from different methods are arranged in ascending order, shown in Figure C.I Notably, approximately 65% of the exhibit a variance in least subband that lower than the original value, indicating that the Haar transform generally enhances data concentration.
5,0 600*
4.5
4.0-
Varlance
3.0
25
Hear High
2.0
Figure C.1: Row-wise Variance Comparison Before and After Haar Transform
18

---

Therefore, we anticipate that the application of HBLLM's structure-await grouping strategy may further improve
data concentration.
C.2 Mathematical Modeling of Variance Improvement Probability
To quantify the improvement in data concentration achieved by the Haar transform combined with the structure- aware grouping strategy, we introduce random variable to describe the probability of improvement in data concentration for each row after applying the strategy, well as the expected probability of improvement E(p) for each matrix block. Let the matrix block size be d x d, and the j-th row, after the Haar row transform, be divided into .M subgroups, cach containing ny coefficients, with an intra-grouu group variance of V.'. The variance of the entire row before transformation denoted as Vo. The sample value Py for the j-th row is calculated as follows:
Σ n sign max (Vo v3)1)
Pj
(31)
d
where, the sign function sign(-) takes the value of only when the subgroup variance is less than the original row variance; otherwise, it is O. The value p3 represents the proportion of coefficients in the j-th row that belong to subgroups with improved concentration. Based on this, we define the expected value Epp) for the entire matrix black to characterize the average level of overall data concentration improvement:
Σj=1Ps d
E(p)
(32)
C.3 Analysis of the Relationship Between Data Concentration and Quantization Fidelity
To further investigate the impact of improved data concentration on model fidelity, we collected data on the probability of variance improvement, the relative & error of matrix blocks before and after quantization, and the corresponding model fidelity, as shown in Figure The relative &2 error serves as the optimization criterion for HBLLM quantization, while model fidelity is measured by perplexity-lower perplexity indicates higher fidelity after quantization.
We analyzed the distribution changes of relative le emrors for all matrix blocks across different models before and after quantization, and we plotted the perplexity performance under various grouping strategies. The experimental results demonstrate that HBLLM not only significantly enhances data concentration but also effectively mitigates the growth of quantization error, thereby better preserving the model's original performance.
Figure C.2a displays the proportion E(p) of Haar coefficients across all matrix blocks that meet the variance improvement criterion after combining the wavelet transform and grouping strategy. The points the graph represent (p) for matrix block a quantized model, with each graph showing the values of E(p) arranged ascending order. As illustrated, after applying the strategy, over 65% of Haar coefficients achieved an improvement in data concentration, with median improvements of 67% and 68%, respectively.
matrix blocks
The results from Figure C.2 lead to the following conclusion: after applying Haar transform combined with the structure-aware grouping strategy proposed by HBLLM, there is a positive correlation between the improvement in data concentration and the enhancement of model fidelity after quantization.
C.4 Empirical Analysis of Saliency Ratio Distribution Across Layers and Blocks
To better understand the structure-awand quantization behavior of our method, we analyze how saliency ratio is distributed across different types of layers and transformer blocks. A saliency ratio is defined as the proportion of weights selected by our Hessian-baseed criterion during quantization.
We visualize the results using two plots: histogram showing the saliency ratio distribution across different layer types, and a line plot depicting the evolution of block-wise saliency ratio across the network. As shown in Figure C.3a and Figure C.3h query and key projections exhibit low saliency (mostly helow 5%), while value, gate, and up- up-projection layers tend to have significantly higher saliency. Additionally, we observe a gradual increase in saliency ratio from shallow to mid transformer blocks, followed by stabilization.
These results confirm that saliency ratio is not uniformly distributed, but highly dependent on layer type and block depth. This validates our strategy of adapting quantization granularity according the structure of the model.
D Storage Cost Analysis and Inference Execution Process
To comprehensively assess compression effectiveness, we divide the stored data into three types: weight overhead coefficient overhead, and flag overhead.
19

---

Ralltttv Quentttaatti
HEL-MoL $3881
HILP
sEL so gooa
HELL No rovi
IEL 0030)
HELL goo
HELL movile
(a) Variance Improvement Score E(p) per block
(b) Relative &a quantization error per block
Perplentt
W0.55
710
1BLL gloud HELL rovile
Perplexity Comparison
Figure C.2: Comparative Evaluation of Grouping Strategies Data Concentration, Quantization Error, and Model Perplexity.
SALLON (354200)
Modia
14
Preportion
Sallency
transformer
Block
block
(a) Saliency Ratio Distribution Across Layers and Blocks
(b) Saliency Statistics Across Layers and Blocks
Figure C.3: The Distribution of Saliency between Different Types of Weight Layers and across Different Transformer Blocks.
Weight Overhead (W-bits). This refers to the number of bits used to store binarized weights. In standard 1-bit schemes, cach weight requires only 1 bit. However, in schemes that retain salient weights, these may be stored with higher precision (c.g.. 2-bit or 8-bit), increasing the overall weight overhead.
To increase fidelity, some weights would values with residual approximation. As result, the average weight overhead per weight (denoted W-bits) of Billm results would become fractional, such 1.08 bits.
than bit. For example, the salient part employs two 1-bit
Coefficient Overhead (C-bits). This refers to additional bit-width required store scaling factors and means. For example, OneBit introduces two scaling vectors per row or column, while ARB-LLMac further computes scaling factors for both rows and columns. Although these parameters are smaller in size than the weight matrix, they must be tightly controlled in precision-critee applications.
Flag Overhead (F-bits). These bits used store indicators for salient/non saantet salient weights (such the "salient column" tag PB-LLM bitmap/group masks BiLLM), or group affiliation information group IDs).
objectively evaluate the compression efficiency various methods, we use "average hit-width per weight" (Average-Bit as a unified metric. This metric characterizes not only storage cost but also the handwidth required from memory to GPU registers. The average bit-width is computed
Total storage bits
AvgBit
x Structure expansion factor.
(33)
Total number of parameters
here, the total number of parameters refers to the product of the matrix dimensions. The structure expansion factor accounts for mismatches in the number of stored units and original parameters (e.g., for restrruttugturing methods, > for methods like FrameQuant).
20

---

F-bits
W-bits
C-bits
Binary Weight
High-Precision (e.g. INT8) Weight
Non bent
Outjut
Safent
Sabert
(a) PB-LLM
(b) BiLLM
Sileet
(c) ARB-LLM
buhent
Von
Misc
Searrt
Mag Aokk
(d) Row-HBLLM
(e) Col-HBLLM
Figure D.1: Overview storage and inference procedure across different LLM binarization methods.
Table D.I: Storage data composition of different LLM binarization methods.
Method
W-Bits
F-Bits
C-Bits
Average-Bits
PB-LLM
1.70
1.000
0.500
3,200
BiLLM
1.09
1.008
0.875
2.973
ARB-LLM
1.09
1.008
1.25
3.348
HBLLM-rowt
1.08
1.088
1.25
3.418
HBLLM-col
1.00
1.008
0.875
2.883
D.1 Data Distributions of Various Binarization Methods
Taking LLaMA1-7B as an example, Figure D. and Table Jillustrate how different LLM binarization methods distribute their storage data. PB-LLM uses hybrid-precision quantization strategy that encodes 10% of weights with 8-bit asymmetric linear quantization and binarizes the remaining 90%. A 1-bit flag differentiates salient from non-salient weights. Non-salient parts are recovered using shared scaling and mean vectors, while salient weights use separate parameters and zero-points All coefficients are stored in FP16 and shared per output channel, resulting in 0.5 bits of cocfficient overhead per weight. PB-LLM's AvgBit is 3.2 bits.
BiLLM extends basic binarization with residual approximation for salient weights. Columns are divided into salient and non-salient parts. 90% of weights use standard 1-bit encoding, while the salient portion is enhanced with residual binarization. This yields weight overhead of 1.09 bits per weight. Additional structure information (bitmap, grouping) accounts for 1.008 bits per weight. With FP1 FP16-stored coefficients and two sets of residual parameters for salient parts, total coefficient overhead is 0.875 bit per weight. Hence, BiLLM's Avg is 2.973 bits.
ARB LLM introduces group modeling for salient columns based on BiLLM to better capture complex distribu- tions. weight overhead remains 1.09 bits per weight. Structure metadata (CGB) takes L,008 bits per weight. All
21

---

coefficients are stored in FP16. Salient columns are grouped and assigned two sets of second-order coefficients. bringing coefficient overhead 1.25 bits per weight. ARB-LLM's AvgBit reaches 3.348 bits.
IIBLLM extends BiLLM by introducing structure-aware grouping strategies improve quantization fidelity the Haar domain. The weight matrix is split into salient and non-salient parts. Salient columns undergo column-wis Haar transforms, while non-salient parts are transformed along either row or column directions, followed by grouped binarization. All weights are quantized using standard 1-bit encoding. HBLLM-cn Mocool has weight overhead of 1.00 bit per weight; HBLLM-row employs neighborhood averaging strategy (FillAvg) reconstruct missing values, increasing the weight overhead 1.08 bits/weight. and frequency-awand grouping metadata add 1.008 hits per weight. All reconstruction coefficients are stored in FP16. HBLLM-row forms four subgroups per row with independent scale and mean values, resulting in a coefficient overhead of 1.25 bits per weight. HBLLM-col shares four subgroups across two rows, averaging two groups per row, and applies intra-band mean sharing to reduce coefficient overhead to 0.875 bit per weight. The final average bit-widths of HBL.LM-row and HBLLM-col are 3.418 and 2.883 bits, respectively.
bitmaps
D.2 Details of Average Bit- Width Calculation
The average bit-width of a quantized matrix W <Rax is defined as the total memory cost (in bits) divided by the number of elements in the original matrix:
M
AvgBit
(34)
n m
For W R"x"" block size k. the memory of W after standard row-wise binarization
mihtipl blocks
row-wise EP16 and
M MM
m
[m/k]
x
2 2 16
(35)
Moreover, second-order row-wise binarization can be represented as
Bi B2
multiple blocks
.a2. and
M 2nd
X m
[m/k]
3 x X 16
(36)
since row-wise μn and μla can be combined together PI + Thus, the memory required by BiLLM be formulated
second
MRAIM 2x [m/k]
3n 16
(37)
binarizaiion
grop hilma
stlient column bimaa
+n (m c) +
[m/k] 2n 16 x
+
(38)
groups
where e is number of salient columns for W. Similarly, we can formulate the memory occupation of first-order row-column-wise binarization and ARB-RC as
second-orderr binarizationn
MARD RC
[m/k] 2n 2c) x
(39)
groups
fist-arder binaeizaiion
group bitmap
salient column hitmap
+u x (m c) + [[m/k]] n + (m - c)) 16 x
+ n
X m +
(40)
2 grronss
In addition, since CGB is used in the experiments, the total memory of ARC-RC + CGB
second binarization
MARBARCACOR = 2x2xxxe ([m/k] x 2n 2c) x 16 x
(41)
3 groups
firs-ander binarization
group bitmap
saliens column bitmap
tn (m c) + ([m/k] n (m c)) 16 x
m
(42)
groups
22

---

Furthermore, we formulate the memory cost of PBLLM by considering both unsalient weights and salient weights
unsalient weights
MPBLLM
Thimury
X
[m/k] 2n 16
(43)
salseni weights
group bitmap
Mhiiarr) X X x 8 + [m/k] x 2n x 16 X m
(44)
where Thinary denotes the ratio the binarized weights.
HBLLM-row adopts four subgroups per row with independent a and μi, intra-band band mean sharing, and a neighborhood-bast reconstruction strategy (FillAvg), which increases the group bitmap cost.
unsalient weights
MIBLLM + mm/k x 3n x 16 2
(45)
groups
group brttnap
bitmap
+
m//k
2n 16 x n (m c)+
(46)
groups
HBLLM-col shares four subgroups across two rows and applies intra-band mean sharing.
unsalient weights
MHELIMoal = n (m c) + m/ x 1.5n x 16 x
(47)
groups
solieut weights
bitmap
salient column hitmap
+n
[m/k]
x
m
(48)
group
Example: Average Bit-width of HBLLM-row and HBLLM-col
Assume W m block size 128 and the number of salient columns 0.08m. The total memory cost HBLLM-row can expressed as:
MHBLIM-00 nm +
m
3n
16-
+ nc +
m7
16.2 n + c) + m
(49)
k
2nm + 2nc 160n
m.
(50)
k
Thus, the average bit-width is:
MARIIMAOW
160n Tm
m7 mm
AvgBit
2.16
3.418 bits.
(51)
mm
Similarly, for HBLLM-col:
MIBLIMeal = ra(m- c)
+
1.5n 16 2 nc +
2n 16 2 + nm + m
(52)
k
m
2nm + 112n + m.
(53)
Then, the average bit-width becomes:
MABLIM-dd = 2
112m m
AvgBit
Rs 2.883 bits.
(54)
10m
E HBLLM Implementation
The implementation details of the HBLLM quantization pipeline provided in Algorithm E.1
23

---

Algorithm E.1 HBLLM: Detailed functions process
func SALIENT W,
H²)
fune BINARY
1: S <- W
[ b,b+ // salient matrix
1: μ
25=1 TM
W // row-wise mean
2: rows(-}
topk([|S||2. dim = 0) searching error
2: W
W
// centered matrix
3: e e Do
IW
3: a
row-wise scale
4: K +-0 -
// optimal number of salient columns
4: t- a sign( W) + p
5: for = .,2,.... ..2,,...Iee(((ows) do
5: return B
6:
Bi BINARY W :.j (rows((())
7:
B2 F BINARY W *.j [/oows(s:)
fune BEG_RROWSSEARRE SEARCH(W)
8:
if I (Bi U ) < then
n number of rows in W
9:
< W [B) B2)
2: E - to0 1xx1
// row-wisc crror
K+i
3: p' + --Ox11
end
// optimal break-point of each row
12:
end for
4: for 0.1, ... 0.9 do
13: return rows|: K]
5:
p <- max(abs(( W) (aaaasssWWdi = 1))
fune HAARQUANT(W mode E (COL, ROW))
6:
Bi < BINARY( w.c().l S p)
7:
B2 < BINARY(W) Weetu)al > p)
1: Now +- Hlow (W)
2: pi < SEG_ROW WEIRRROWSSSRRREE SSAARE(EAW...... (ow)
8:
if [ (B) B2) || then
3: Blo +- BINARY (b)
9:
e W (B1 + B2)
1w, Low) +
10:
p" p
(W (b)
| 1>p;.low/
11: end if
4: Waiff W
H 7--1-1111 (Bion 0)
12: end for
5: Whigh + Hnight ( airr)
13: return p²
6: P2 < SEG ROW
LOW_SSEAR SEARCH Whigh high)
7: Buigh BINARY swsss high
BINARY(WW (b)
8: B Blow + Bnigh
9: return B
Table F.1: Perplexity and zero-shot accuracy results of DeepSeek-R1-De RI--Dstill-LLamm-
Perplexity
Model
Method
Wbits
AvgQAt
C4
Wiki2
PTB
FullPrecision
16.00
18.40
13.14
20.57
63.80
FrameQuant
2.20
59.60
46.71
70.79
43.95
PB-LLM
1.70
316.3
224.6
344.3
34.52
DeepSeek-RI-Diie Litistill-Llaaa---ee -BB
BiLLM
1.06
234.4
219.5
442.9
35.91
ARB-LLMX
1.06
74.31
54.73
69.35
42.92
ARB-LLMge
1.06
67.77
54.27
92.37
43.00
HBLLM-row
1.05
40.88
29.26
45.00
47.06
HBLLM-col
1.00
55.82
35.80
62.80
45.71
Additional Experimental Results
E1
Experimental Results for DeepSeck-RI-Distillle RI-Distill-LLamma--
Table provides further experimental results DeepSeek-Ri-De -iieeek----Disttie Ri-Dissill--lamm-Sl) model. In line with the trends observed Table 1, HBLLM consistently outperforms existing 1-2 bit quantization techniques across all these evaluation metrics. Remarkably, even when applied to the more complex and modern DeepSeek-RR RI- Distill-Llama 8B architecture, HBLLM preserves the same performance advantages previously demonstrated on LLaMA3-SB. This assessment highlights the effectiveness of HBLLM for LLMs.

---

Table F.2: Accuracy of 9 QA datasets on LLaMAl family models. We compare the results among FrameQuant PB-LLM, BiLLM, ARB-LLM and HBLLM to validate the quantization effect.
LLaMA1
Zeroshot Accuracyt
AvgQA
Size
Method
Wbits
PIQA
BoolQ
OBQA
WinoG
ARC-e
ARC-c
HSwag
COPA
LAMBD
FullPrecision
16.00
78.67
75.02
34,20
70.01
75.34
41.89
56.94
85,00
73.51
65.62
PrameQuant
71.16
23.60
66.54
29.78
81.00
58.70
PB- LLM
1.70
$4.62
58.04
13.20
49.17
29.38
21.25
27.61
61.00
7.10
35.71
7B
59.63
15.00
71.00
21.15
ARB-LLMx
1.09
63.55
64.10
16.80
56.75
42.21
21.93
34.21
73.00
38.27
45.65
ARB-LLMac
1.09
69.15
64.10
22.40
61.25
52.90
26
39.26
78.00
57.71
52.23
HBLLM-row
1.08
73.67
71.07
24.60
62.75
64.81
30.80
47.60
79.00
63.03
57.48
HBLLM-col
1.00
71.93
62.60
22,00
62.27
61.15
28.58
44.68
78.00
55.09
54.03
PullPrecision
16.00
79.16
77.92
33,20
72.61
77.36
46.42
59.93
90.00
76. 19
68.09
FrameQuant
2.20
75.14
73.64
26,20
68.67
66.71
34.64
48.58
83,00
69.59
60.69
PR-LIM
1.70
58.00
62.02
14.60
52.96
33.25
18.26
29.99
69.00
25, .4
40.39
13B
67.19
ARB-IIMX
1.10
N/A
N/A
N/A
N/A
N/A
N/A
N/A
NIA
N/A
N/A
ARB-LLMac
1.10
72.03
24.40
68.11
64.73
31.14
45.54
$8.00
69.
HBLLM-row
1.09
76.28
68.93
28.60
68.90
70,37
38,31
52.54
86,00
73.22
62.57
HBLLLM-col
1.00
75.03
70.46
24.80
69.69
69.02
34.39
51.01
87.00
69.82
61.25
FullPrecision
16,00
80.96
82.69
36,00
75.93
80.35
52.73
63.35
90.00
77.55
71.06
PrameQuant
2.20
76.39
73.58
31,20
72.85
72.18
38.14
53,97
92,00
75.84
65.13
PR-LIM
1.70
63.60
64.34
16.00
60.30
44.78
20.90
34.80
74.00
46.30
47.22
30B
BiLLM
L.11
71.49
68.87
24.40
67.96
63.59
29.69
44.51
84.00
68.10
58.07
ARB-LLMx
L.11
IIAA
N/A
NIA
N/A
NIA
NNA
N/A
N/A
NIA
NIA
ARB-LLMar
74.86
29.60
HBLLM-row
L.10
77.09
79.72
32.20
71.67
74.24
43.34
55,79
91.00
75.80
66.76
HBLLM-col
1.00
76.66
72.23
29.80
71.35
73.91
39.68
53.81
90.00
76.29
64.86
PallPrecision
16.00
81.34
84.86
38.00
77.43
81.31
52.82
64,56
91.00
79.12
72.27
FrameQuant
2.20
79.00
83.27
32,00
74.66
77.23
45.56
56.58
90.00
78.92
68.58
PB-LLM
L,70
71.98
79.45
28.
74.98
67.34
34.47
46.57
89.00
70.37
62.48
65B
BiLIM
1.10
74.05
80.40
26,20
70.40
69.32
36.60
48.15
85,00
68.35
62.05
ARB-LLMx
L.10
IN/A
N/A
N/A
N/A
NIA
N/A
N/A
N/A
N/A
N/A
ARB-LLMac
77.97
32.60
44.28
HBLLM--ro row
1.09
78.89
81.96
33.80
75.77
76.94
45.65
59.03
91.00
79.58
69.18
HBLIM-cR
79.00
57.98
F22
Comparison on zero-shot QA datasets
In the main paper. we report the average accuracy (AvgQA) across 9 zero-shot QA datasets to provide high-level comparison of different quantization methods, In this appendix, present the detailed accuracy each individual dataset. As shown in Tables F2 F5 our method consistently delivers strong performance across all datasets, further validating its effectiveness and robustness in diverse zcro-shot question answering tasks.
F.3 Comparison of Avg. Relative Perplexity and Avg. Relative QA Accuracy across Models
To provide a unified view of model performance across both language modeling and common sense reasoning tasks, we compute two metrics: Avg. Relative Perplexity and Avg. Relative QA. Avg. Relative Perplexity is calculated as the average over three language modeling datasets: C4, Wiki2 and PTB. Avg, Relative QA is computed as the average across zero-shot QA datasets.
Relative Perplexity defined as:
Speplesity
RSS. Pesplexity
(55)
SFP Perplesity
sFP where Sperplexit and Perpleiit are the perplexities of the models after and before quantization, respectively.
Relative QA defined as:
Soa
RS
(56)
S#P
QA
where SQx and SiA are the QA scores the models after and before quantization, respectively.
Figure presents the comparison of these two metrics across LLaMA-L, -1, LLaMA-2, LLaMA-3, and OPT family models. HBLLM consistently shows lower relative perplexity and higher relative QA accuracy compared to prior 1-2 bit quantization methods (PB-LLM, BiLLM, and RRB-LLMsc). demonstrating both effectiveness and scalability across diverse LLMs.
25

---

Table F.3: Accuracy of 9 QA datasets LLaMA2 family models. We compare the results among FrameQuant, PB-LLM, BiLLM, ARB-LLM and HBLLM to validate the quantization effect.
LLaMA2
Zero-shot Accuracyt
AvgQAT
Size
Method
Wbits
PIQA
BoolQ
OBQA
WinoG
ARC-e
ARC-e
HSwag
COPA
LAMBD
PullPrecision
16.00
76.44
79.72
33.40
66.46
71.91
44.20
57.80
87.00
70.95
65.54
FrameQuant
2.20
68.55
67.13
19.00
61.80
56.61
27.56
42.20
78.00
53.89
52.75
PR-ILM
1.70
54.84
61.90
11.80
48.07
27.90
19:45
27.78
50.00
17.14
36.54
7B
BiLLM
1.08
59.85
63.82
12.80
54.14
21.67
31,66
64.00
30.93
42.11
ARB ARB-LLMMy
1.08
61.97
66.42
16.20
57.70
43.27
22. 02.78
33.81
68,00
38.54
45.41
ARB-LLMec
61.15
59.51
59.27
23.46
50.61
46.71
HBLLM-row
1.07
72.96
73.58
25,60
60.93
61.87
32.17
47.50
84.00
61.03
57.74
HBLLM-col
1.00
71.11
69.66
21.40
61.33
59.22
29.35
45.33
74.00
55.44
54.09
FullPrecision
16.00
79.05
80.55
35.20
72.14
79.42
48.46
60.05
91.00
76.73
69.18
Quant
2.20
73,83
76.30
25.60
69.61
69.02
33.79
47.15
85,00
71.84
61.35
PB-LLM
1.70
54.30
41.25
12.60
50.12
27.27
20.14
27.00
59.00
4.52
32.91
13B
RiLLM
1.08
61.81
66.51
18.00
56.35
43.64
21.84
33.33
75.00
44.34
46.76
ARB-LLMx
1.08
NNA
N/A
N/A
N/A
N/A
INAA
N/A
NIA
N/A
NIA
ARB-LLMac
HIBLLM row
1.07
75.63
77.74
29.80
68.27
71.21
37.12
52.54
88.00
72.19
63.61
HBLLM-col
74.86
77.61
26.80
69.14
70.66
35.75
50.46
84.00
69.12
62.04
PallPrecision
16.00
82.26
83.76
37.20
77.98
82,74
54.35
64.77
94.00
79.60
72.96
FrameQuant
2,20
N/A
N/A
NIA
N/A
NIA
N/A
N/A
N/A
N/A
N/A
PB-LLM
64.04
74.77
21.20
64.72
27.22
38.77
80.00
70B
BiLLM
1.09
68.39
70.64
24.40
65.98
64.23
32.34
41.85
82.00
52.47
55.81
ARB-LLMx
1.09
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
ARB-LLMac
77.53
80.21
34.40
75.85
44.80
92.00
80.98
HBLLM-FOW
L.08
79.65
81.56
33.40
75.06
79.29
50.51
59.02
91.00
80.56
70.01
IIBLMM
78.59
Table F:4: Accuracy of 9 QA datasets on LLaMA3 family models. We compare the results among FrameQuant PB-LLM, BiLLM, ARB-LLA BRB-LLM and HBLLM to validate the quantization effect.
LLaMA3
Zero-shot Accuracyt
AvgQAt
Size
Method
Whits
PIQA
RoolQ
OBQA
WinoG
ARC-
ARC-e
HSwag
COPA
LAMBD
PollPrecision
16:00
78.35
83.18
34,20
71.67
52.90
57.66
89.00
71.88
FrameQuant
2.20
65.40
71.87
19.20
59.04
60.19
29.69
47.06
78.00
46.94
$2.27
PB-LLM
57.18
62.26
11.60
50.36
31.31
18.00
28:59
55.00
17.16
36.83
BiLLM
1.06
59.14
64.46
15.20
53.75
37.54
19:45
32.01
65.00
30.04
41.84
ARB-LLMRc
1.06
62.73
69.72
18.80
56.91
50.34
25.94
35.18
74.00
49.08
49.08
HBLLM-row
1.05
67.68
72.94
23,80
63.54
56.52
28.07
43.55
81.00
56.14
54,80
IIBLIM col
1.00
67.03
69.94
21.20
60.93
49.79
85
42.79
81,00
44.30
51.43
FullPrecision
82.15
60.07
93:00
FrameQuant
2.20
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
PR-LIM
1.70
$7.89
68.07
17.60
58:96
37.37
19.62
38,86
80.00
48.67
47.45
BiLLM
1.09
52.39
42.26
12.60
51.78
25.34
20.73
28.11
65.00
9.41
34.18
N/A
N/A
NIA
N/A
ARB LLMgc
1.09
74.86
80.89
25.80
72.69
72.69
40.44
$1.99
85.00
70.76
63.90
HBLLM-col
1.00
56.04
77.83
29.00
73.40
33.63
18.34
53.19
91.00
70.56
55.89
Table F.5: Accuracy of 9 QA datasets on DeepSeek-R1-Distill-.e among FrameQuant, PB-LLM, BiL.M,, ARB-LIM and HBLLM to validate the quantization effect.
We compare the results
DeepSeck-RI R1-Distill--lam
Zero-shot Accuracyt
AvgQAt
Size
Method
Whits
PIQA
BoolQ
OBQA
WinoG
ARC-
ARC-c
HSwag
COPA
LAMBD
FullPrecision
16.00
76.33
82.94
31.40
67.48
70.54
40.27
55.54
89.00
60.72
63.80
FromsQuant
2.20
61.48
63.50
17.00
55.17
22.10
35.42
67.00
29.81
43,95
PR-IIM
1.70
55.01
59.54
12.60
48.07
29.08
16.98
27.56
52.00
9.82
34.52
SB
BILLM
1.06
55.01
60.37
12.80
26.89
19.62
29.20
57.00
12.87
35.91
ARB-LLMX
1.06
60.34
67.55
15,60
55.64
39.90
22.35
32.97
68.00
23.97
42.92
ARB L.MMcC
1.06
60.50
63.15
15.40
53.51
40.24
21.76
34.55
64.00
33.90
43,00
HALLM..ro
1.05
63.71
72.26
18.00
56.20
41.75
23.38
39.74
74.00
34.47
47.06
HBLLM-col
1.00
64.53
68.53
18.40
56.35
42.42
22. 22.44
39.75
70.00
28.99
45.71
26

---

Table F.6: Accuracy of 9 QA datasets OPT family models. We compare the results among Frame FrameQuant, PB-LLM, BiLLM, ARB-LLM and HBLLM to validate the quantization effect.
Accuracyt
AvgQAT
Size
Method
Whits
PIQA
BoolQ
OBQA
WinoG
ARC-e
ARC-
HSwag
COPA
LAMBD
PullPrecision
16.00
71.65
51.77
23:40
59.27
57.03
23.38
41.53
81.00
57.85
52.54
FrameQuant
2.20
64.91
57.00
16.40
55.
46.51
20.73
34.01
71.00
34.58
44.48
PB-LLM
1.70
53.81
48.38
12.60
50.67
28.49
19.97
26.20
59.00
1.88
33.44
1.38
BiLIM
1.09
59.41
61.07
13.80
52.49
35.65
17.06
29.53
62.00
14.54
38.39
ARB-LLMy
1.09
60.39
60.61
14.60
52.72
39.98
18.09
30.65
64.00
31.73
41.42
ARB-LLMac
1.09
86
53.75
47.22
33.58
42.27
45.28
HBLLM-row
1.07
66.49
62.14
17.80
56.20
47.94
21.42
35,70
69.00
40.42
46.35
PullPrecision
16.00
73.83
60.34
25.00
61.33
60.73
26.79
45.88
77.00
63.63
54.95
FrameQuant
2.20
67.36
61.93
18.20
57.22
52.23
22.70
36.88
75.00
54.71
49.58
PR-LLM
50.67
30.13
27.35
37.62.
2,7B
BILLM
1.10
60:45
62.08
13.20
53.59
35.90
20.56
30.54
63.00
20.88
40.02
ARB-LLMx
1.10
63.00
62.35
15.40
54.78
42.85
19.45
32.20
69.00
42.38
44.60
ARB LLMc
1.10
67.74
57.03
18.20
58.17
51.05
22.61
37.33
74.00
59.62
49.53
HBLLM-Fow
1.09
68.66
58.35
20.20
56,91
52.86
22.87
39.49
72.00
47.86
48.80
48.56
FullPrecision
16.00
75.84
65.75
27.00
65.19
67.09
32.94
52.45
81.00
58.41
FramoQuant
2.20
72.85
66.15
22.60
63 63.69
62.54
28.07
45.49
82.00
68.25
55,42
PR-LIM
39.50
13B
BiLLM
1.13
67.25
63.91
18.20
58.64
51.73
24.66
38,18
76.00
$4.03
49.82
ARB-LLMx
1.13
N/A
NIA
NIA
NIA
N/A
N/A
N/A
NIA
N/A
N/A
ARB ARB-LLMac
1.13
73.34
67.92
23.40
64.25
60.14
27.30
44.41
82.00
67.44
55.35
TIBLLM now
1.12
73.94
67.98
22.60
61.64
62.50
29.10
46.55
83.00
66.64
55,91
HBLIM-col
1.00
73.56
66.51
22.00
64.56
62.04
28.92
45.65
82.00
68.06
55.66 56.66
PullProcision
16.00
77.58
70.40
68.35
70.03
34.47
54.28
71.47
62.09
FrameQuant
2.20
75.08
70.49
26.60
64.48
66.79
30.72
48.37
$2.00
72.04
59.62
PB-LLM
1,70
64.47
62.94
53.67
44.65
35.95
70.00
45.37
16.14
BiLLM
1.13
71.33
63.91
21.80
61.80
$7.87
25.09
41.94
80,00
64.23
$4.22
MLMX
N/A
ARR-T1MMc
1.13
74.27
67.37
27
64.33
63.76
30.03
48.03
78.00
74.15
58.59
HBLLM-rOW
1.12
75.46
70.80
26.80
66.22
66.12
30.97
49.76
82.00
72.23
60.04
HBLIM-cO
1.00
74.70
68.90
25.00
64.96
66.16
30.03
48.90
81.00
70.52
58.91
Belp Salotiie Perpeeddyy
pap. Dalebve OK
LLaMA-1
LLaMA-2
LLAMA-3
OPT
FP16
PR-LLM
BILLM
ARB-L1Mgc
HBLL M-row(ours)
Figure F.1: Average relative perplexity and average relative QA accuracy (normalized to FP16) for
LLaMA-1/22 1/2// family models, comparing LLM binarization methods and our HBLLM.
27

---

ROlATti Detanttoon Earer
Releeti Ooann Quasnttennig Ereer Qu
OPT-1:58
LLOMA2-78
OPT-1:38
LLaMMA2-78
16
4
HELF-Oo--1
FELLX 1666
HELL
HBLN uhiba
YELLS
(a) Study of salient column selection criterion
(b) Study of grouping granularity
Relattve Quantization [N]
28
35.30
3034
3433
OPT:1.38
LLaMA2-78
(c) Effectiveness of shared mean
(d) Study of partitioning candidates number
Figure F.2: Ablation study on OPT-1.3B and LLaMA2-7B. Results are measured by & Relative Quantization Error.
F4
Ablation Study from the Perspective of & Relative Error
To complement the ablation studies presented in the main paper, provide an additional analysis from the perspective relative quantization & error each quantized weight block, defined the Frobenius norm of the quantization residual normalized by the original weight norm. We then examine the distribution these errors across different layers and quantization methods to assess fidelity at the matrix level.
of the Frobenius norm. Specifically, compute the relative
The relative error of quantized weight matrix is defined as:
|WFp W
RE weight
(57)
||
where Wir and Wa denote the full-preciie precision and quantized weight matrices, respectively.
Figure F22 visualizes the sorted relative quantization errors under different ablation settings, including (a) salient column selection criterion, (b) grouping granularity. (c) shared mean strategy, and (d) number of partitioning candidates. Across all cases, our proposed HBLLMM consistently achieves lower median relative compared prior approaches, further confirming the effectiveness each component.
we observe that using the & norm as column selection criterion leads to significantly lower quantization errors than ε,, validating its stronger ability to capture the energy distribution weights. In Figure
Figure F2a
F2bb row-wise grouping substantially outperforms global grouping. indicating the importance of fine-grained adaptivity. In Figure F2c the shared mean strategy slightly reduces quantization error while offering better compression efficiency. Finally, Figure F22d inereasing the number of partition candidates reduces error up a point, with 40 candidates offering the best trade-off.
These observations align well with perplexity-bar trends, and jointly confirm the robustness and precision of HBLLM under various quantization design choices.
G Inference Latency Estimation and Efficiency Analysis
To evaluate the computational efficiency of HBLLM in practical inference scenarios. we design estimation- based experiment to analyze its inference latency. This is necessary because no current inference framework fully supports the dequantization algorithm used in HBL.MM, and constructing such backend involves significant engincering effort beyond the current scope.
Our findings suggest that the inference latency HBLI.M is approximately 31.8% of the FP16 baseline, indicating strong computational benefits despite the algorithm's structural complexity.
Latency Estimation Methodology. as:
We define the relative inference time of HBLLMM compared to FP16
R(p. 1) = TiMLIM
- p)
(58)
Tepi6
28

---

where is the portion of time spent matrix-vector multiplication (GEMV) during inference. and | the acceleration factor of GEMV after quantization. Following prior works such as GPTQ [ use p 0.78.
To compute
Twrch
1
(59)
Tique
where Tocch denotes the runtime of FP16 GEMV using Py PyTorch, and Thyny refers to our quantized matrix-vector multiplication kernel (HQMV).
Since the current hqmv implementation does not support Intra- Frequ-eey Grouping (IFG), use the runtime of HBLLM-col without IFG (denoted HBLLM-col w/o IFG) proxy estimate the runtime HBLLM-col with full IFG. Notably, applying doubles number of groups, potentially leading to: roughly 2x more CUDA warp divergence, 90% increase in average data loading (Avgbit increases to 2.88 bit).
However, since data loading and computation can overlap on GPU, we conservatively estimate that the runtime with IFG should not exceed twice the runtime without IFG.
Table G.I: GEMV Runtime and Inference Latency Estimation for HBLLM (OPT-175B Linear Layer, P100 GPU).
Method
Trucch (s)
Tegme (8)
R(p. I)
FP16 Baseline
1.35 10-3
1.00
HBLLM-col (w/ IFG)
8.54 x 10-6
HBLLM-col (with IFG)
1.70
10
0.318
Sctup: We benchmark a single linear layer of OPT-175B under the same GEMV input/output shape used in GPTQ [2]. All timing experiments are conducted on NVIDIA P100 GPU.
Measured: Twurch 1.35 10-3s
Measured: Thgue for HBLLM-CR col w/ IFG = 8.54 10 1070s
Estimated: Tagms for full HBLLM-col (with IFG) R 1.70 x 10
This yields:
1.35 x 10-3
0.78
1.70 x 10 7.94,
(1 0.78) + 7.94
~ 0.318
(60)
Discussion: This result indicates that HBLLM inference, despite involving additional processing (e.g.. dequantization and grouping), maintains high efficiency due to:
Lightweight binary matrix operations;
Reduced memory handwidth consumption;
Effective overlap of memory access and computation in GPU execution.
Overall, HBLLM achieves strong acceleration FP16 without relying highly specialized hardware handcrafted fusion kernels, making it viable candidate practical deployment in low-bit LLM inference systems.
H Licenses for Existing Assets
Ilama, Ilama2, Ilama3
https://wgginngicce.o///eeeeeee Ilama, Ilama2, llama3
33
OPT-175B LICENSE AGREE-
MENT
15
http://giihbb.ommm/ammmmunnng77s MIT License
18
http://gghhbbbbooo//iiiggggggggu Apache License 2.0
https///jttbbboom///urrEE/LL///
29 https:/github.com//oootrc//LLLL Creative Commons Zero v1.0 Universal
4
https//ghhh....oooooooooeessssss quecstio
http::/ingggiiggee...oooocccccaa Creative Commons Attribution Share Alike 4.0
2
text http://tt__ee__olyl_____eee Dataset provided for research purposes only
2
http:///nggngaace..ccccccccnnnnn Apache License 2.0
Creative Commons AttributionSSarrAAlke License.
https://ggtubb.comm/lluueeerr//t Creative Commons Attribution 4.0 International
harness, MIT License
http:://enndd.rrg///ooo/zzzzzz/
26
https://githubb. httptiuww.ooooooooogge
wtwp.////ooooooo..eeee//eeeeeeee to-texxttt-o-exttt-ee transfer- tranoorrrtransffmmeuutttttttttt Apache License 2.0
27
http://uoorrroogguuuoooooooooooe BSD 2-Clause License
29

---

28
http:///ghhbbbooommmlmm///w/oaao Apache License 2.0
30

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiiaton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The abstract and introduction clearly state our main contributions, which are supported by the theoretical and empirical results presented in the Section and Section 4.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
31

---

Justification:
Please refer to Section 5,
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: Please refer Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please refer Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
32

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. w
For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The code will be made publicly available upon acceptance of this paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (nttps://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ 7/nips. /ttppp.cc/pbliccguideeee http://oeeuuiiiiiooooooooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
33

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
Please refer to Section 4.1.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported because it would be computationally expensive. Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
show in tables
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Quantization for models <30B run on 4xRTX 3090 (24GB), and for models 330B on A800-80GB. See Section 4.1.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
34

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics hattp::/neurrpp:
Answer: [Yes]
Justification: We carefully read and follow the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: The paper is purely fundamental research and does not involve social impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [No]
Justification:
This paper does not release new models or datasets.
Guidelines:
The answer NA means that the paper poses such risks.
35

---

Released models that have high risk for misuse or dual use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Please refer Appendix.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
36

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the L.M used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: LLMs were not involved as core components in the development of the method or results in this paper.
Guidelines:
The answer NA means that the core method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: /ttp::://///rpie iips / MLMLLLL for what should or should be described.
37


Reviews:
[{"review_id": "KNG4OoIakd", "reviewer": "Reviewer_3GV9", "summary": "This paper proposes HBLLM, a 1 bit PTQ method for LLM. The key idea is to apply a Haar wavelet transform to decompose weight matrices into frequency components, enabling structure-aware quantization. The method further introduces two grouping strategies: frequency-aware intra-row grouping and saliency-driven column selection, alongside a shared-mean quantization scheme for non-salient components. Experiments show improved perplexity and QA performance compared to prior 1-bit quantization methods.", "questions": "(1) I encourage the authors to disentangle the contribution of Haar-based frequency decomposition from other components (e.g., grouping, saliency selection) to provide a clearer understanding of where the benefits originate.", "limitations": "The authors did not discuss the limitations of the work.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n(1) Introduces the Haar wavelet transforms as a localized orthogonal transformation to increase binary expressiveness.\n(2) Introduces two innovative grouping strategies, frequency-aware multi-parameter intra-row grouping and l2-norm saliency-driven column selection, and also employs a shared mean for non-salient weights, to better preserve important weight structures.\n\nWeaknesses\n\n(1) While the Haar transform is well-motivated theoretically, its contribution is not isolated. The authors do not compare their method with a baseline where grouping and saliency selection are applied directly in the original weight space without the Haar transform. This leaves unclear whether the Haar decomposition meaningfully improves quantization results beyond what grouping and saliency alone achieve.\n\n(2) Fig.2 is not clear enough, maybe adding more description in the caption would help understanding. Also, if could add a figure to illustrate the weight decomposetion (before and after) would greatly enhance the reader's understanding of the core idea.", "was_revised": true, "final_justification": "The authors' response addressed my concern. Thus, I decided to raise my rating of this paper.", "rebuttal": "We sincerely thank the reviewer for the thorough evaluation and insightful suggestions. We fully agree that verifying the individual contribution of each component and improving the clarity of our results are both important and necessary.\n\n**Q1 (W1):** I encourage the authors to disentangle the contribution of Haar-based frequency decomposition from other components (e.g., grouping, saliency selection) to provide a clearer understanding of where the benefits originate.\n\n**Response:** To address this, we conducted additional experiments to explicitly isolate and validate the performance gain brought by the Haar transform in our method.\n\nThe results confirm that the Haar transform indeed lead to significant improvements in HBLLM performance — gains that cannot be achieved by other strategies such as grouping or saliency selection alone.\n\nSpecifically, starting from the original BiLLM baseline, we independently activated the following two strategy components:\n\n- L2-norm-based saliency-driven column selection,\n\n- Frequency-aware multi-parameter intra-row grouping.\n\nWe then compared their performance with HBLLM variants that incorporate the Haar transform, with or without these two additional strategies.\n\nAs shown in the Table 1, although introducing either the $\\ell_2$-based saliency or the aware grouping alone brings minor improvements to BiLLM, the addition of the Haar transform in HBLLM consistently yields more substantial gains in perplexity reduction and QA accuracy improvement on both OPT-1.3B and LLaMA2-7B, even under partial strategy activation (e.g., HBLLM-col+$\\ell_2$).\n\n**Table 1: Perplexity (↓, C4, Wiki2, PTB) and AvgQA accuracy (↑, AvgQA over 9 zero-shot tasks) of BiLLM variants with Haar. OPT-1.3B (cols 2–5), LLaMA2-7B (cols 6–9).**\n\n|Method|C4↓|Wiki2↓|PTB↓|AvgQA↑|C4↓|Wiki2↓|PTB↓|AvgQA↑|\n|-|-|-|-|-|-|-|-|-|\n|BiLLM|56.24|68.43|119.2|38.39|33.97|31.38|373.0|42.11|\n|BiLLM+$\\ell_2$|55.95|72.42|105.9|37.95|33.46|31.34|695.8|41.11|\n|BiLLM+$\\ell_2^\\dagger$|56.88|70.48|92.16|39.28|28.17|25.08|226.3|41.77|\n|Row-Haar+BiLLM|47.45|52.81|62.81|39.57|25.77|25.12|138.0|44.67|\n|Col-Haar+BiLLM|95.56|128.8|171.3|36.92|41.03|37.25|5193|39.60|\n|HBLLM-row+$\\ell_2$|26.47|33.68|41.17|41.17|16.26|19.86|87.90|47.90|\n|HBLLM-col+$\\ell_2$|26.37|29.99|36.24|42.13|15.04|13.99|154.6|49.38|\n|HBLLM-row|**19.55**|26.95|**25.70**|**46.87**|**13.00**|13.20|**85.50**|50.86|\n|HBLLM-col|21.98|**23.69**|27.39|45.21|13.18|**12.02**|146.1|**51.34**|\n\n*Notes: $\\ell_2$ denotes $\\ell_2$-norm-based saliency-driven column selection; $\\dagger$ denotes multi-parameter intra-row grouping. In Table 1, the results of HBLLM-row and HBLLM-col (our proposed method) already include these strategies.*\n\nIn summary, the Haar transform itself plays a decisive role in improving the final performance. We will include the corresponding experiments and analysis in the final version of the paper.\n\n**W2:** Fig.2 is not clear enough, maybe adding more description in the caption would help understanding. Also, if could add a figure to illustrate the weight decomposetion (before and after) would greatly enhance the reader's understanding of the core idea.\n\n**Response:** We appreciate the reviewer’s feedback about the lack of clarity in Figure 2. Due to rebuttal constraints, we cannot update the figure at this stage, but we provide the following clarification in text. \n\nThe HBLLM quantization process consists of four steps: preparation, salient column selection, haar transform and quantization for the salient part, and quantization for the non-salient part.\n\nSince the salient columns are excluded from the Haar transform of the non-salient part, their positions must be filled before performing row-wise Haar transforms. This is handled by a process we refer to as FillAvg, where each missing column is filled with the average of its adjacent non-salient columns.\n\nFor the non-salient part, HBLLM supports flexible choice between row-wise (HBLLM-row) and column-wise (HBLLM-col) transforms. The salient part undergoes column-wise Haar transformation followed by HaarQuant for quantization.\n\nWe will further revise the caption in the final version.\n\nIn the final version, we will also include additional visualizations (e.g., energy distribution before and after Haar transform, examples of row/column grouping structures) to aid understanding."}, {"review_id": "OAmukEDiWP", "reviewer": "Reviewer_VPLv", "summary": "The submission proposes a new post-training LLM quantization technique based on frequency-domain lossy weight compression. It shares the main idea with other lossy frequency-domain compression schemes that were applied successfully to media data types in the past. Instead of exploiting different perception sensitivity levels of humans depending on frequency bands (works for media types), this method uses the Haar-transform, a saliency selection method of matrix columns and a new metric called CIQ that is used for the optimization.", "questions": "* Please add measurements showing inference speed (latency / throughput) especially as the technique requires a sequence of more complex math operations.\n\n* Line 177: is K a single global parameter? Would multiple K work as well and if yes, how (maybe based on the energy of the Haar-spectrum)?\n\n* Table 3 (FrameQuant), close to line 291: is it possible to get at least an estimate for the total amount of memory FrameQuant would have allocated?", "limitations": "Yes", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "The strength of the submitted work is the idea. It seems to deliver consistently better performance at 1 bit-per-weight or close to that compared to >2 bits-per-weight of the next best performing technique which is FrameQuant.\n\nMinor weaknesses are missing experimental results showing inference speed (latency / throughput) and a few details that are left unclear (see Questions).", "was_revised": true, "final_justification": "All my issues were resolved by the rebuttal.", "rebuttal": "We sincerely appreciate the reviewer's positive acknowledgment of our work and their detailed suggestions. We would like to respond to the following \nthree questions one by one.\n\n**Q1:** Add measurements showing inference speed (latency / throughput) especially as the technique requires a sequence of more complex math operations.\n\n**Response:** To address this point, we designed an experiment that combines direct measurement with estimation to evaluate the inference latency of HBLLM. This is necessary because, at present, there is no existing inference framework that fully supports the dequantization algorithm used in HBLLM. Building a highly optimized inference framework requires considerable engineering effort and cannot be completed in the short term.\n\nOur estimation results show that the inference latency of HBLLM is approximately $31.8$% of the FP16 baseline inference time.\n\n**Estimation Procedure**\n\nWe define the relative inference time of HBLLM to the FP16 baseline as:\n\\begin{equation}\n R(p,l):= \\frac{T_{HBLLM}}{T_{fp16}}=\\left[ (1-p)+p/l \\right]. \n\\tag{1}\n\\end{equation}\nwhere:\n- $ T_{HBLLM} = p*T_{fp16} $ corresponds to the portion of inference time attributed to General Matrix–Vector multiplication (GEMV) ($p$ is the proportion of inference time spent on GEMV, which depends on the LLM architecture. Following GPTQ’s benchmark [1], we set $p=0.78$)\n- $l$ is the GEMV acceleration factor after quantization, defined as:\n \\begin{equation}\n l:= \\frac{T_{torch}}{ T_{hqmv}}. \n\\tag{2}\n\\end{equation}\nwhere $T_{torch}$ is the runtime of FP16 GEMV using PyTorch, and $T_{hqmv}$ is the runtime of our quantized GEMV implementation (denoted hqmv), which varies with the quantization strategy.\n\nSince the current hqmv implementation does not support Intra-Frequency Grouping (IFG), we use the runtime of HBLLM-col without IFG (denoted HBLLM-col w/o IFG) as a proxy to estimate the runtime of HBLLM-col with full IFG. Notably, applying IFG doubles the number of groups, potentially leading to:\n- roughly 2× more CUDA warp divergence,\n- a $90$% increase in average data loading (Avg bits per weight increases to 2.88).\n\nHowever, since data loading and computation can overlap on GPU, we conservatively estimate that the runtime with IFG should not exceed twice the runtime without IFG.\n\n**Experiment Steps**\n\nThe full estimation process involves the following steps:\n\n1. GEMV Runtime Measurement: Measure $T_{torch}$ and $T_{hqmv}$ using the HBLLM-col w/o IFG strategy.\n\n2. Runtime Estimation with IFG: Estimate the runtime of hqmv with full HBLLM-col (i.e., with IFG), and compute the acceleration factor $l$.\n\n3. Final Latency Ratio: Plug $p=0.78$ and the estimated $l$ into Equation (1) to compute the relative inference time \n$R$.\n\n**Experimental Results**\n\nFollowing the GPTQ benchmark setup [1], we tested GEMV on a single linear layer from the OPT-175B model. Experiments were run on an NVIDIA P100 GPU.\n\n\nThe result from a single run showed $ T_{torch} = 1.35e-03s $ , and $ T_{hqmv} $ with HBLLM-col w/o IFG was $ 8.54e-05s $.\nThen, we estimated\n$T_{hqmv}$ with HBLLM-col w/o IFG to be $1.70e-04s$. Finally, substituting $p=0.78$ and $l=7.94$ into the equation yields\n$R=0.318.$\n\nThis means that the estimated inference latency of HBLLM is only $31.8$% of that of the FP16 baseline, demonstrating its significant computational efficiency.\n\n[1] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, Dan Alistarh.GPTQ: Accurate Post-training Compression for Generative Pretrained Transformers. arXiv preprint arXiv:2210.1732, 2022.\n\n**Q2:** Line 177: is K a single global parameter? Would multiple K work as well and if yes, how (maybe based on the energy of the Haar-spectrum)?\n\n**Response:** In HBLLM, the significant column count K is not a globally fixed hyperparameter; instead, it is adaptively determined as the optimal value within each matrix block. Therefore, there are multiple values of K.\n\nSpecifically, the process for selecting significant columns is illustrated in  Appendix Algorithm E.1.\n\nFor each block, we sort the columns based on their saliency scores in descending order and gradually accumulate the candidate significant columns, calculating their corresponding binary reconstruction errors. Ultimately, we select the number of significant columns n* that results in the minimum error as the optimal K for that block. In Algorithm E.1, n* should be changed to K to maintain consistency with the main text and eliminate any misunderstanding.\n\nRegarding the suggestion to \"select salient columns based on the energy of the Haar spectrum,\" we find this to be very valuable. The saliency-driven algorithm presented in this paper does not currently consider this direction. In the future, we will explore incorporating Haar energy spectrum features into the criteria for selecting significant columns.\n\n**Q3:** Table 3 (FrameQuant), close to line 291: is it possible to get at least an estimate for the total amount of memory FrameQuant would have allocated?\n\n**Response:** Yes, the memory usage of FrameQuant can be estimated. We provide experimental results for smaller models and an extrapolated estimate for the larger one.\n\nSpecifically, we conducted experiments to measure the actual GPU memory consumption during the quantization process of LLaMA1-7B and LLaMA1-13B using FrameQuant. The results are shown in Table 1.\n\n**Table 1: GPU Memory Usage(GB) during FrameQuant Quantization**\n\n|Model|Memory(GB)|\n|-|-|\n|LLaMA1-7B|11|\n|LLaMA1-13B|16|\n\nBased on these experimental results, we estimate the memory usage for LLaMA1-30B to be approximately **40 GB**. During the quantization of the 30B model, this estimated requirement exceeded available GPU memory, leading to runtime failures such as the one mentioned near line 291 in the main text.\n\n\n**Paper Formatting Concerns:** Additionally, regarding the layout position and citation format of Figure 2, we appreciate the reviewer’s detailed suggestions. We will adjust the position of Figure 2 in the final version, moving it closer to Section 3.2 to enhance the readability of the corresponding process description.\n\nAs for the \"C\" / \"J\" designations in the references, they represent conference and journal sources, respectively, and are primarily used for distinguishing between document types. In the revised version, we will remove these suffixes to improve formatting consistency and avoid confusion."}, {"review_id": "5rkuMPzuvW", "reviewer": "Reviewer_cU7V", "summary": "The paper proposes the HBLLM approach to enhance 1-bit post-training quantization for LLMs. HBLLM uses Haar wavelet transforms to decompose weights by frequency, enabling more effective quantization. The proposed method introduces structure-aware grouping and saliency-based selection to improve accuracy and storage efficiency. Results on the OPT and LLaMA models show that HBLLM achieves state-of-the-art performance for 1-bit quantization.", "questions": "1. What is the difference between the proposed saliency-driven selection via L2 norm and AWQ [1] ?\n\n2. The objective function in the paper does not consider calibration data. However, the authors use calibration data to compute the parameter importance metric. What would happen if the reconstruction loss also considered the calibration data?\n\n3. As far as I know, the frequency domain is often used to capture the characteristics of data in the feature space rather than the weight space. Please correct me if I am wrong. Why does transforming weights to the frequency domain using the Haar Transform help capture the structural differences of weights?\n\n4. In Table 1, could you provide the equation used to compute the average bit-width in detail? Since element-wise saliency detection creates overhead, why is the bit-width using HBLLM-col 1.00?\n\nRef: Lin, Ji, et al. \"AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration\". CoRR abs/2306.00978 (2023). 2023.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- Good empirical improvements.\n- The paper is easy to follow.\n- The paper provides extensive experiments across multiple LLMs and tasks.\n\nWeaknesses:\n\n- The paper detects the saliency parameters in an element-wise manner, which could result in more overhead compared to a channel-wise approach. Did the authors consider this?\n- The motivation for using the Haar Transform to capture the structural differences of weights is not clear.\n- The paper does not number the equations.", "was_revised": true, "final_justification": "Good paper. Most of my concerns have been resolved. Therefore, I increase the score after the rebuttal.", "rebuttal": "We sincerely thank the reviewer for the careful reading and constructive feedback. Below we provide point-by-point responses to the Weaknesses and Questions, supported by both experiments and theoretical analysis.\n\n**W1:** The paper detects the saliency parameters in an element-wise manner, which could result in more overhead compared to a channel-wise approach. \n\n**Response:** Our method does not perform element-wise saliency detection. Instead, we evaluate saliency column-wise, which incurs much lower theoretical overhead than element-wise. Specifically, as described in Algorithm E.1 of the Appendix, we first compute a saliency score for each column. Then, within each block, we perform a structured search over the sorted saliency scores to select the optimal subset of salient columns. The strategy is adapted from BiLLM [1], with the difference that we replace the original $\\ell_1$-based metric with an $\\ell_2$-based one.\n\n**W2:** The motivation for using the Haar Transform to capture the structural differences of weights is not clear.\n\n**Response:** Applying the Haar wavelet transform to LLM quantization offers three main advantages:\n\n* The inverse quantization set becomes richer in representation, which can be demonstrated via the CIQ metric. Please refer to the details in in the Appendix B.\n\n* The data distribution becomes more concentrated: with approximately $65\\%$ probability (from Appendix Figure C.1), the variance of the high- and low-frequency coefficient sets in each row is smaller than before the transformation. Please refer to the details in in the Appendix C.\n\n* The additional computational cost for inference is $\\mathcal{O}(d)$ [2], where $d$ is the input length, as the Haar transform can be implemented using local convolutional layers, resulting in lower cost than methods such as FrameQuant.\n\n[1] Huang W, Liu Y, Qin H, et al. BiLLM: pushing the limit of post-training quantization for LLMs. Proceedings of the 41st ICML. 2024: 20023-20042.\n\n[2] Meyer, Y. and Ryan, R.D., Wavelets: Algorithms & Applications, Society for Industrial and Applied Mathematics, 1993.\n\n**Q1:** What is the difference between the proposed saliency-driven selection via L2 norm and AWQ?\n\n**Response:** The key difference lies in how saliency is defined and computed.\n\nAWQ selects salient weights based on the output of activation functions — it analyzes the activation-induced importance of weights during inference and identifies those with the greatest impact on model outputs. In contrast, our method in HBLLM performs analysis on the weight matrix itself: it computes the $\\ell_2$ norm of each column in a saliency factor matrix derived directly from the weights, without relying on input-dependent activations.\n\nThis makes HBLLM activation-free and efficient, as it avoids collecting runtime statistics, while still effectively identifying structurally significant components for preserving accuracy under quantization.\n\nWe thank the reviewer for pointing out AWQ and will include a discussion comparing the two methods in the revised manuscript.\n\n**Q2:** What would happen if the reconstruction loss also considered the calibration data?\n\n**Response:** We tried incorporating calibration-data-based reconstruction loss. However, the results show no consistent improvements and sometimes even lead to degradation. The results are in the Table 1:\n\n**Table 1: Perplexity (↓, C4, Wiki2, PTB) and AvgQA accuracy (↑, AvgQA over 9 zero-shot tasks) of BiLLM variants with Haar. OPT-1.3B (cols 2–5), LLaMA2-7B (cols 6–9).**\n\n|Method|C4↓|Wiki2↓|PTB↓|AvgQA↑|C4↓|Wiki2↓|PTB↓|AvgQA↑|\n|-|-|-|-|-|-|-|-|-|\n|HBLLM-row|**19.55**|26.95|**25.70**|**46.87**|**13.00**|13.20|**85.50**|50.86|\n|HBLLM-col|21.98|**23.69**|27.39|45.21|13.18|**12.02**|146.1|51.34|\n|HBLLM-row+calib-loss|23.03|26.55|29.96|44.93|14.65|13.45|282.6|51.45|\n|HBLLM-col+calib-loss|24.95|28.47|36.89|44.27|14.19|13.41|247.3|**52.43**|\n\n**Q3:** Why does transforming weights to the frequency domain using the Haar Transform help capture the structural differences of weights?\n\n**Response:** While frequency transforms are commonly applied to features (e.g., images), our focus is the structure of weights. We assume different frequency bands encode different structural patterns. Applying Haar to weights enables selective grouping and quantization with minimal accuracy loss. This choice is motivated by two factors:\n\n* **Better grouping.** Frequency-domain representation increases the separability for grouping, enlarging the inverse quantization set (CIQ). As shown in Table 2, this improves fidelity over BiLLM. See Appendix B.3.\n\n* **Higher quantization efficiency.** Due to frequency sparsity, accurate weight recovery is possible using few components, reducing 1-bit quantization error. See Appendix C.3.\n\n**Table 2: Perplexity (↓, C4, Wiki2, PTB) and AvgQA accuracy (↑, AvgQA over 9 zero-shot tasks) of BiLLM variants with Haar. OPT-1.3B (cols 2–5), LLaMA2-7B (cols 6–9).**\n\n|Method|C4↓|Wiki2↓|PTB↓|AvgQA↑|C4↓|Wiki2↓|PTB↓|AvgQA↑|\n|-|-|-|-|-|-|-|-|-|\n|BiLLM|56.24|68.43|119.2|38.39|33.97|31.38|373.0|42.11|\n|BiLLM+$\\ell_2^\\dagger$|56.88|70.48|92.16|39.28|28.17|25.08|226.3|41.77|\n|HBLLM-row|**19.55**|26.95|**25.70**|**46.87**|**13.00**|13.20|**85.50**|50.86|\n|HBLLM-col|21.98|**23.69**|27.39|45.21|13.18|**12.02**|146.1|**51.34**|\n\n*Notes: $\\ell_2$ denotes $\\ell_2$-norm-based saliency-driven column selection; $\\dagger$ denotes multi-parameter intra-row grouping. In Table 1, the results of HBLLM-row and HBLLM-col (our proposed method) already include these strategies.*\n\n**Q4:** In Table 1, could you provide the equation used to compute the average bit-width in detail? Since element-wise saliency detection creates overhead, why is the bit-width using HBLLM-col 1.00?\n\n**Response:** The average bit-width of a quantized matrix \\\\( \\widehat{\\mathbf{W}} \\in \\mathbb{R}^{n \\times m} \\\\) is defined as the total memory cost \\\\( \\mathcal{M} \\\\) divided by the number of elements:\n\n\\\\[\n\\textbf{AvgBit} = \\frac{\\mathcal{M}}{n \\times m}.\n\\tag{1}\n\\\\]\n\nLet \\\\( \\mathbf{W} \\in \\mathbb{R}^{n \\times m} \\\\), \\\\( k \\\\) be block size , and \\\\( c \\\\) be the number of salient columns in \\\\( \\mathbf{W} \\\\). For each method, $\\mathcal{M}$ in eq. (1)  is defined by:\n\n\\\\[\n\\mathcal{M}_{\\text{BiLLM}} = 2nc + \\left\\lceil \\frac{m}{k} \\right\\rceil \\cdot 3n \\cdot 16 + n(m - c) + \\left\\lceil \\frac{m}{k} \\right\\rceil \\cdot 2n \\cdot 16 \\cdot 2 + nm + \\widetilde{m},\n\\tag{2}\n\\\\]\n\n\\\\[\n\\mathcal{M}_{\\text{ARB-RC}} = 2nc + \\left\\lceil \\frac{m}{k} \\right\\rceil \\cdot (2n + 2c) \\cdot 16 \\cdot 2 + n(m - c) + \\left\\lceil \\frac{m}{k} \\right\\rceil \\cdot (n + m - c) \\cdot 16 \\cdot 2 + nm + \\widetilde{m},\n\\tag{3}\n\\\\]\n\n\\\\[\n\\mathcal{M}_{\\text{PBLLM}} = rnm + \\left\\lceil \\frac{m}{k} \\right\\rceil \\cdot 2n \\cdot 16 + (1 - r)nm \\cdot 8 + \\left\\lceil \\frac{m}{k} \\right\\rceil \\cdot 2n \\cdot 16 + nm,\n\\tag{4}\n\\\\]\n\n\\\\[\n\\mathcal{M}_{\\text{HBLLM-row}} = nm + \\left\\lceil \\frac{m}{k} \\right\\rceil \\cdot 3n \\cdot 16 \\cdot 2 + nc + \\left\\lceil \\frac{m}{k} \\right\\rceil \\cdot 2n \\cdot 16 \\cdot 2 + n(m + c) + \\widetilde{m},\n\\tag{5}\n\\\\]\n\nand\n\n\\\\[\n\\mathcal{M}_{\\text{HBLLM-col}} = n(m - c) + \\left\\lceil \\frac{m}{k} \\right\\rceil \\cdot 1.5n \\cdot 16 \\cdot 2 + nc + \\left\\lceil \\frac{m}{k} \\right\\rceil \\cdot 2n \\cdot 16 \\cdot 2 + nm + \\widetilde{m},\n\\tag{6}\n\\\\]\nwhere \\\\( r \\\\) denotes the ratio of the binarized weights.\n\n**Example: Average Bit-width of HBLLM**\n\nAssume \\\\( k = 128 \\\\), \\\\( c = 0.08m \\\\). For HBLLM-row and HBLLM-col, AvgBit will be:\n\n\\\\[\n\\mathbf{AvgBit} = \\frac{\\mathcal{M}_{\\text{HBLLM-row}}}{nm} = 2.16 + \\frac{160n \\left\\lceil \\frac{m}{k} \\right\\rceil}{nm} + \\frac{\\widetilde{m}}{nm} \\approx 3.418~\\text{bits},\n\\tag{7}\n\\\\]\n\nand\n\n\\\\[\n\\mathbf{AvgBit} = \\frac{\\mathcal{M}_{\\text{HBLLM-col}}}{nm} = 2 + \\frac{112n \\left\\lceil \\frac{m}{k} \\right\\rceil}{nm} + \\frac{\\widetilde{m}}{nm} \\approx 2.883~\\text{bits},\n\\tag{8}\n\\\\]\n\n respectively.\n\nIn HBLLM-col, saliency selection is performed at the column level rather than element-wise. Within each block, a few salient columns are identified via structured search and processed separately from the non-salient ones. Unlike HBLLM-row, HBLLM-col does not require FillAvg to complete the matrix, thus avoiding additional storage overhead.\n\nIn summary, the only overhead in HBLLM-col is the **1-bit** binarization per weight matrix.\n\nRegarding the **equation numbering issue**, we will standardize the numbering of all equations in the final version."}, {"review_id": "ZefM3z3miz", "reviewer": "Reviewer_1RBF", "summary": "This paper proposes HBLLM, a 1-bit post-training quantization method for large language models that uses Haar wavelet transforms to improve expressiveness and accuracy. It introduces two structure-aware grouping strategies: one based on frequency and the other on saliency (L2-norm). For less important weights, it shares group means to save space. Experiments on OPT and LLaMA models show its good results, with only 1.08 bits per weight and strong performance.", "questions": "See weakness.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strength**\n\n1. The primary contribution is that this method is the first to integrate \"localized\" orthogonal transformations (i.e., Haar wavelets) into a BiLLM-style quantization process. With other 3 auxillary strategies, Frequency-aware multi-parameter intra-row grouping/L2-norm-based saliency-driven column selection/Intra-frequency-band mean sharing, is achieves comprehensive good results.\n\n2. Experiments on comprehensive models, baselines, and evaluation benchmarks demonstrate that the proposed approach is effective.\n\n**Weakness**\n\n1. The main difference and contribution of this paper compared to BiLLM lies in the introduction of local orthogonal transforms, corresponding to lines 7–10 of the HBLLM algorithm. However, the actual impact of these local transforms on performance (e.g., PPL and accuracy) is not clearly demonstrated. It would be helpful to show in ablation studies what happens when this component is removed or replaced by a global orthogonal transform.\n\n2. The authors mention in the introduction that global orthogonal transforms are computationally expensive, thus motivating the use of local transforms. However, only the overall runtime is reported. A more detailed analysis comparing the computational cost of local vs. global transforms would provide useful insight.\n\n3. There is no evaluation or discussion of computational efficiency, such as matrix multiplication speed or end-to-end inference latency. This analysis would be important for understanding the practical benefits of the proposed method.", "was_revised": true, "final_justification": "Some confusion has been clarified.", "rebuttal": "Thank you very much for your careful review and constructive feedback. Here we provide point-by-point responses to your comments and questions.\n\n**W1:** The main difference and contribution of this paper compared to BiLLM lies in the introduction of local orthogonal transforms, corresponding to lines 7–10 of the HBLLM algorithm. However, the actual impact of these local transforms on performance (e.g., PPL and accuracy) is not clearly demonstrated. It would be helpful to show in ablation studies what happens when this component is removed or replaced by a global orthogonal transform.\n\n**Response:**\nLocal orthogonal transforms (such as Haar) indeed significantly enhance the performance of HBLLM. Removing these transforms or replacing them with global transforms clearly leads to degraded performance. To systematically demonstrate the effectiveness of the Haar transform, we have conducted the following ablation studies and compared the results against HBLLM and BiLLM (see Table 1):\n\n1. BiLLM+$\\ell_2^\\dagger$: Employing $\\ell_2$-norm-based saliency-driven column selection together with multi-parameter intra-row grouping (results in Table 1 under BiLLM+$\\ell_2^\\dagger$).\n\n2. Haar+BiLLM: This refers to the method obtained by removing the $\\ell_2$-norm-based saliency-driven column selection and multi-parameter intra-row grouping strategies from HBLLM (see the results of Row-Haar+BiLLM and Col-Haar+BiLLM in Table 1).\n\n3. DCT+BiLLM: Applying the BiLLM algorithm to the coefficient matrices obtained by performing 1D DCT transform row-wise or column-wise on the weight matrices (results in Table 1 under Row-DCT+BiLLM and Col-DCT+BiLLM).\n\n**Table 1: Perplexity (↓, C4, Wiki2, PTB) and AvgQA accuracy (↑, AvgQA over 9 zero-shot tasks) of BiLLM variants with Haar. OPT-1.3B (cols 2–5), LLaMA2-7B (cols 6–9).**\n\n|Method|C4↓|Wiki2↓|PTB↓|AvgQA↑|C4↓|Wiki2↓|PTB↓|AvgQA↑|\n|------|----|------|-----|-------|----|------|-----|-------|\n|BiLLM|56.24|68.43|119.2|38.39|33.97|31.38|373.0|42.11|\n| BiLLM+$\\ell_2^\\dagger$|56.88|70.48|92.16|39.28|28.17|25.08|226.3|41.77|\n|Row-Haar+BiLLM|47.45|52.81|62.81|39.57|25.77|25.12|138.0|44.67|\n|Col-Haar+BiLLM|95.56|128.8|171.3|36.92|41.03|37.25|5193|39.60|\n|Row-DCT+BiLLM|8010|11517|6729|31.36|45358|49395|26888|34.48|\n|Col-DCT+BiLLM|107.1|150.5|250.1|34.19|26.54|24.64|1202|44.82|\n|HBLLM-row|**19.55**|26.95|**25.70**|**46.87**|**13.00**|13.20|**85.50**|50.86|\n|HBLLM-col|21.98|**23.69**|27.39|45.21|13.18|**12.02**|146.1|**51.34**|\n\n*Notes: $\\ell_2$ denotes $\\ell_2$-norm-based saliency-driven column selection; $\\dagger$ denotes multi-parameter intra-row grouping. In Table 1, the results of HBLLM-row and HBLLM-col (our proposed method) already include these strategies.*\n\nFrom Table 1, we summarize the following observations:\n\n- BiLLM+$\\ell_2^\\dagger$ slightly improves performance compared to the original BiLLM.\n\n- The HBLLM method demonstrates substantial performance improvement compared to BiLLM+$\\ell_2^\\dagger$ (see BiLLM+$\\ell_2^\\dagger$, HBLLM-row, and HBLLM-col rows in Table 1).\n\n- Row-Haar+BiLLM notably outperforms BiLLM and BiLLM+$\\ell_2^\\dagger$, yet still lags significantly behind HBLLM.\n\n- Col-Haar+BiLLM shows clearly inferior performance compared to BiLLM.\n\n- Both HBLLM-row and HBLLM-col outperform BiLLM and BiLLM+$\\ell_2^\\dagger$ substantially, highlighting the critical role of an effective quantization strategy that leverages the Haar transform to preserve the frequency-domain structural features of weights.\n\n- Using the global row-wise DCT leads to severe performance degradation, significantly underperforming BiLLM. While the global column-wise DCT shows some performance improvements in certain tests for LLaMA2-7B, it still significantly underperforms BiLLM for all tests with OPT-1.3B.\n\nOverall, local orthogonal transforms are indeed beneficial in preserving structural fidelity during quantization. However, replacing them with global transforms would necessitate re-designing the quantization strategy to maintain the model's performance.\n\n**W2:** The authors mention in the introduction that global orthogonal transforms are computationally expensive, thus motivating the use of local transforms. However, only the overall runtime is reported. A more detailed analysis comparing the computational cost of local vs. global transforms would provide useful insight.\n\n**Response:** Quantization algorithms leveraging local orthogonal transforms demonstrate a clear advantage in runtime efficiency. To substantiate this, we compared the runtime and computational complexity of model quantization using Haar (local transform) versus DCT (global transform), as shown in Table 2 below.\n\n**Table 2: Time comparison between BiLLM, DCT+BiLLM and HBLLM on LLaMA-1 with different model sizes.**\n\n|Method|7B|13B|30B|\n|-|-|-|-|\n|BiLLM|36min|71min|142min|\n|DCT+BiLLM|211min|414min|1012min|\n|HBLLM|44min|98min|173min|\n\nThe experimental results indicate that the computational overhead introduced by global transforms is substantial. HBLLM's runtime is moderately higher ($<25$%) than BiLLM’s, whereas the DCT-based transform incurs a runtime cost approximately 5–7 times greater (using NumPy’s DCT implementation).\n\nRegarding computational complexity:\n\n- Haar (local transform) involves only pairwise operations between adjacent elements. When a fixed number of decomposition levels is used (e.g., one-level transform), it results in linear computational complexity $O(n)$, where $n$ is the number of elements in a weight matrix. Such locality enables these operations to be fused into GPU kernels for concurrent execution without introducing significant additional memory overhead.\n\n- DCT (global transform) requires computation across the entire dimension, with complexity of $O(n \\log_2 n)$ using an FFT-based implementation. In contrast, applying the row-wise (or col-wise) 1D Haar transform results in a total complexity of $O(n)$. For instance, consider the fully-connected layers in LLaMA-30B, which have dimensions of $6656\\times6656$. Applying a 1D DCT in this scenario increases computational complexity approximately 13-fold compared to the row-wise 1D Haar transform.\n\nIn summary, both our theoretical analysis and experimental results clearly demonstrate that local orthogonal transforms (such as Haar) provide a superior balance between accuracy and efficiency, exhibiting significant practical advantages for real-world deployment.\n\n\n**W3:** There is no evaluation or discussion of computational efficiency, such as matrix multiplication speed or end-to-end inference latency. This analysis would be important for understanding the practical benefits of the proposed method.\n\n**Response:** To address this point, we designed an experiment that combines direct measurement with estimation to evaluate the inference latency of HBLLM. This is necessary because, at present, there is no existing inference framework that fully supports the dequantization algorithm used in HBLLM. Building a highly optimized inference framework requires considerable engineering effort and cannot be completed in the short term.\n\nOur estimation results show that the inference latency of HBLLM is approximately $31.8$% of the FP16 baseline inference time.\n\n**Estimation Procedure**\n\nWe define the relative inference time of HBLLM to the FP16 baseline as:\n\\begin{equation}\n R(p,l):= \\frac{T_{HBLLM}}{T_{fp16}}=\\left[ (1-p)+p/l \\right]. \n\\tag{1}\n\\end{equation}\nwhere:\n- $ T_{HBLLM} = p*T_{fp16} $ corresponds to the portion of inference time attributed to General Matrix–Vector multiplication (GEMV) ($p$ is the proportion of inference time spent on GEMV, which depends on the LLM architecture. Following GPTQ’s benchmark [1], we set $p=0.78$)\n- $l$ is the GEMV acceleration factor after quantization, defined as:\n \\begin{equation}\n l:= \\frac{T_{torch}}{ T_{hqmv}}. \n\\tag{2}\n\\end{equation}\nwhere $T_{torch}$ is the runtime of FP16 GEMV using PyTorch, and $T_{hqmv}$ is the runtime of our quantized GEMV implementation (denoted hqmv), which varies with the quantization strategy.\n\nSince the current hqmv implementation does not support Intra-Frequency Grouping (IFG), we use the runtime of HBLLM-col without IFG (denoted HBLLM-col w/o IFG) as a proxy to estimate the runtime of HBLLM-col with full IFG. Notably, applying IFG doubles the number of groups, potentially leading to:\n- roughly 2× more CUDA warp divergence,\n- a $90$% increase in average data loading (Avg bits per weight increases to 2.88).\n\nHowever, since data loading and computation can overlap on GPU, we conservatively estimate that the runtime with IFG should not exceed twice the runtime without IFG.\n\n**Experiment Steps**\n\nThe full estimation process involves the following steps:\n\n1. GEMV Runtime Measurement: Measure $T_{torch}$ and $T_{hqmv}$ using the HBLLM-col w/o IFG strategy.\n\n2. Runtime Estimation with IFG: Estimate the runtime of hqmv with full HBLLM-col (i.e., with IFG), and compute the acceleration factor $l$.\n\n3. Final Latency Ratio: Plug $p=0.78$ and the estimated $l$ into Equation (1) to compute the relative inference time \n$R$.\n\n**Experimental Results**\n\nFollowing the GPTQ benchmark setup [1], we tested GEMV on a single linear layer from the OPT-175B model. Experiments were run on an NVIDIA P100 GPU.\n\n\nThe result from a single run showed $ T_{torch} = 1.35e-03s $ , and $ T_{hqmv} $ with HBLLM-col w/o IFG was $ 8.54e-05s $.\nThen, we estimated\n$T_{hqmv}$ with HBLLM-col w/o IFG to be $1.70e-04s$. Finally, substituting $p=0.78$ and $l=7.94$ into the equation yields\n$R=0.318.$\n\nThis means that the estimated inference latency of HBLLM is only $31.8$% of that of the FP16 baseline, demonstrating its significant computational efficiency.\n\n[1] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, Dan Alistarh.GPTQ: Accurate Post-training Compression for Generative Pretrained Transformers. arXiv preprint arXiv:2210.1732, 2022."}]

Author rebuttal:

Meta-review: This paper proposes HBLLM, a wavelet-enhanced high-fidelity 1-bit post-training quantization method for Large Language Models that leverages Haar wavelet transforms and innovative structure-aware grouping strategies to improve quantization fidelity with minimal overhead. All the reviewers posted positive comments After discussion, most of the concerns have been well resolved. The method demonstrates significant strengths, including the novel integration of Haar wavelet transforms to enhance binary expressiveness, innovative structure-aware grouping strategies that effectively preserve important weight structures. Therefore, the AC recommends to accept this paper.
Decision: Accept (spotlight)