Title: HBLLM: Wavelet-Enhanced High-Fidelity 1-Bit Quantization for LLMs
Venue: NeurIPS 2025
Accepted: yes
Keywords: 1-Bit Quantization；Haar-based
Abstract: We introduce HBLLM, a wavelet-enhanced high-fidelity $1$-bit post-training quantization method for Large Language Models (LLMs). By leveraging Haar wavelet transforms to enhance expressive capacity through frequency decomposition, HBLLM significantly improves quantization fidelity while maintaining minimal overhead. This approach features two innovative structure-aware grouping strategies: (1) frequency-aware multi-parameter intra-row grouping and (2) $\ell_2$-norm-based saliency-driven column selection. For non-salient weights, a shared mean is employed across quantization groups within each frequency band to optimize storage efficiency. Experiments conducted on the OPT and LLaMA models demonstrate that HBLLM achieves state-of-the-art performance in $1$-bit quantization, attaining a perplexity of $6.71$  on LLaMA$2$-$13$B with an average weight storage of only $1.08$ bits. Code available at: https://github.com/Yeyke/HBLLM.

HBLLM: Wavelet-Enhanced High-Fidelity 1-Bit Quantization for LLMs

Abstract

We introduce HBLLM, a wavelet-enhanced high-fidelity 1 -bit post-training quantization method for Large Language Models (LLMs). By leveraging Haar wavelet transforms to enhance expressive capacity through frequency decomposition, HBLLM significantly improves quantization fidelity while maintaining minimal overhead. This approach features two innovative structure-aware grouping strategies: (1) frequency-aware multi-parameter intra-row grouping and (2) ℓ 2 -normbased saliency-driven column selection. For non-salient weights, a shared mean is employed across quantization groups within each frequency band to optimize storage efficiency. Experiments conducted on the OPT and LLaMA models demonstrate that HBLLM achieves state-of-the-art performance in 1 -bit quantization, attaining a perplexity of 6 . 71 on LLaMA 2 -13 B with an average weight storage of only 1 . 08 bits. Code available at: https://github.com/Yeyke/HBLLM.

1 Introduction

In recent years, Large Language Models (LLMs) have achieved remarkable progress in natural language processing tasks. However, their massive parameter sizes-often reaching tens or even hundreds of billions-pose significant challenges for deployment on edge devices and in low-resource environments. To reduce the computational and memory burden of these models, a variety of compression techniques have been proposed, including quantization [12, 33, 35], pruning [11, 31], and knowledge distillation [19, 30]. Among them, Post-Training Quantization (PTQ) is widely adopted for its efficiency, requiring no additional training and having low deployment cost, especially in 1 -bit quantization, which is considered a key approach for achieving extreme inference efficiency [13].

Although existing 1 -bit PTQ methods [15, 17, 34]

20

FP16

PB-LLM

15

+

BiLLM

ARB-LLMRC

→

10

HBLLM-row(ours)

HBLLM-col(ours)

3.0

2.75

2.50

2.25

2.00

1.75

1.50

1.25

1.00

?

7B

13B

30B

have achieved some success on base models such as GPT2 and OPT, they tend to suffer from significant performance degradation-or even complete failure-when applied to more complex modern architectures like LLaMA3-8B [16]. To address this, recent studies have introduced several strategies to improve quantization fidelity:

Group quantization : divides the weight matrix into multiple groups for separate quantization. For instance, outlier-aware partitioning handles critical columns independently but can be constrained by partition design and scalability [15];

Residual approximation : adds residual terms on top of primary quantization to partially recover errors [6], though this provides limited fidelity gains and introduces extra computation;

Low-Rank Adaptation (e.g., LoRA) : inserts low-rank modules to absorb quantization errors with some flexibility, like [34], but often shows sensitivity to hyperparameters;

Global orthogonal transformations : apply global rotations in [1, 2, 5] before model compression to enhance representational capacity, but require expensive inverse transforms (e.g., matrix multiplications at O ( d 2 ) complexity for a d -dimension linear layer), leading to increased inference latency and energy consumption, making them impractical for deployment.

To overcome the structural trade-off between expressiveness and efficiency, we propose a novel 1 -bit PTQ frameworkHBLLM . This method is the first to integrate localized orthogonal transformations (i.e., Haar wavelets) into a BiLLM-style quantization process. Combined with structure-aware grouping, HBLLM significantly enhances expressive power under ultra-low bit budgets while maintaining negligible inverse transform cost and excellent compatibility with hardware-efficient inference.

Our main contributions are as follows:

A localized orthogonal transformation mechanism : we apply a single Haar wavelet transform to decompose the weight matrix into high- and low-frequency components, improving binary expressiveness while reducing transform computation;

Frequency-aware multi-parameter intra-row grouping : we introduce intra-row grouping in the frequency domain to capture structural patterns;

ℓ 2 -norm-based saliency-driven column selection : we propose an ℓ 2 norm-based ranking method to retain key columns using saliency metrics, effectively reducing quantization error;

Intra-frequency-band mean sharing : for non-salient components, we introduce a mechanism that shares the mean across groups within the same row and wavelet band, reducing storage without sacrificing fidelity.

We conduct extensive experiments on OPT [37], LLaMA family [32] of LLMs. Results show that HBLLM achieves state-of-the-art performance under 1 -bit quantization: Across language modeling tasks (C4, PTB, WikiText2), the perplexity ratio between HBLLM and the original FP16 model remains within the range of 1 . 2 -2 . 2 , shown in Fig 1, outperforming the next-best methods by 33% -66% ; On 9 zero-shot QA benchmarks, HBLLM retains 73 . 8% -88 . 8% of the original model's accuracy; On modern architectures such as LLaMA3-8B, HBLLM remains stable with no performance collapse; Even with a lower average bit rate and memory usage than BiLLM and ARB-LLMRC [18], HBLLM outperforms both in overall task accuracy.

These results demonstrate that HBLLM significantly extends the applicability of 1 -bit quantization, balancing extreme compression with high fidelity, and offers a new paradigm for deploying large-scale language models efficiently.

2 Related Work

2.1 1-Bit Post-Training Quantization

1 -bit PTQ has emerged as a critical promising solution for deploying LLMs under extremely low bit budgets. Representative methods such as BiLLM [15] adopt a salient column separation mechanism, in which salient weights are quantized independently, while non-salient weights are grouped based on magnitude and quantized row-wise. ARB-LLMX [18] further introduces column-wise grouping and alternating refined binarization, achieving notable improvements in fidelity. Unlike [10, 34], BiLLM can accomplish PTQ tasks without intensive computation for knowledge distillation with multi-GPUs.

However, current methods face several key limitations: (1) They heavily rely on fixed thresholds or simple ℓ 1 -based heuristics for salient column selection, which are insufficient to capture sparse but significant activation outliers; (2) They fail to account for the structural asymmetry between row and

column dimensions in weight matrices, limiting their adaptability to complex model architectures; (3) They completely neglect frequency-domain information.

2.2 Evolution and Limitations of Grouping Strategies

To improve quantization flexibility and fidelity, some studies have proposed learnable or adaptive grouping strategies. For example, Mixture of Scales [17] introduces a Mixture-of-Experts (MoE) mechanism to assign scaling factor groups, and OneBitGPT [34] uses frequency masks to control quantization range sensitivity, and AWQ [3] identifies weights with the greatest impact on model predictions only based on activation outputs. However, these methods are generally effective only on unstructured tensors, rely on fine-grained distillation, and lack explicit frequency-domain awareness.

In addition, existing grouping strategies [15, 18] often apply uniform partitioning rules across the entire weight matrix, ignoring variations across different rows. This can lead to degraded expressiveness when quantizing models with significant inter-row diversity.

2.3 Comparison Between Global Orthogonal Transforms and Local Wavelet Transforms

Orthogonal transforms have recently been adopted to improve LLM quantization. FrameQuant [1] and QuIP [5] utilize orthogonal transforms to enhance fidelity, but inference with such global transforms incurs high overhead, requiring O ( d 2 ) matrix multiplications [1] that cannot be fused into linear layers, leading to increased latency and energy cost.

By contrast, local orthogonal transforms such as the Haar wavelet [20] offer localized spectral sensitivity and have been widely applied in image compression, denoising, and edge detection [9, 14]. They can be efficiently implemented via lightweight local convolutions with negligible inference cost, making them well-suited for low-bit compression and edge deployment.

3 HBLLM: A Quantization Framework with Wavelet Transform and Frequency-Domain Grouping

3.1 Motivation and Core Challenges

Current mainstream 1 -bit quantization methods face three key challenges in practice: (1) limited numerical expressiveness leading to high reconstruction error; (2) insufficient accuracy in salient column selection, failing to capture critical activation columns; (3) lack of structure-aware grouping strategies that adapt to heterogeneous model structures.

To characterize expressiveness under ultra-low bit settings, we introduce a new metric: the cardinality of the Inverse Quantization Set (CIQ) , which measures the size of the discrete set of dequantized values within a row. CIQ serves as a unified indicator of how the above challenges constrain model fidelity. It acts both as a theoretical tool to analyze the limits of existing methods and as empirical evidence of the advantage of our proposed method.

Under 1 -bit quantization, the CIQ of BiLLM and ARB-LLMX is 8 and 10 , respectively. When block size sets to 128 , the CIQ upper bound of ARB-LLMX can reach 128 . In contrast, our method achieves a CIQ of up to 1024 after applying the Haar wavelet transform, significantly improving theoretical expressiveness. For more information on the benefits introduced by applying Haar transform, please refer to the appendix B and C.

Based on aboved analysis, we propose: (1) Haar wavelet transform to enhance expressive capacity by frequency decomposition; (2) ℓ 2 -norm-based saliency-driven column selection to prioritize critical columns; (3) frequency-aware multi-parameter intra-row grouping to capture structural patterns. We also introduce an intra-frequency-band mean sharing strategy and local convolution optimization to reduce storage and inference cost, thus forming a 1 -bit PTQ framework HBLLM .

3.2 Method Overview

We define the objective of HBLLM under the binary quantization setting for LLM weights. Specifically, the quantization targets the full-precision weight matrix W FP ∈ R d × d , where a binary diagonal mask matrix M sal ∈ { 0 , 1 } d × d indicates which columns are selected as salient. The salient and

① Preparation

Non-Salient Region Quantization

② Salient Column Selection

Salient Region Quantization

Weal

X

Col-HQuant

Wsal

B

FillAvg

Row-HQuant

W

non-sal

OR

Col-HQuant

Row-Haar Quant

6

6

H

0

Col-Haar Quant

Smin

smax

wmin

amin

non-salient parts are quantized in the Haar domain, and their respective quantized Haar coefficients are denoted by ̂ W sal B and ̂ W non-sal B . These are then reconstructed using inverse Haar transforms H -1 1 and H -1 2 .

The reconstruction objective of HBLLM is twofold. For the quantization of a matrix layer W , the objective expressed in the Frobenius norm is formulated as:

where X is the input of the matrix layer. For quantization of a matrix block W FP of W , the object is:

When H 1 = H 2 are fixed Haar transforms, this formulation simplifies to a quantization problem entirely in the Haar domain. In this case, the objective is the same to that of BiLLM. Layer-level quantization is commonly tackled with the GPTQ algorithm [12].

We emphasize that our approach does not aim to solve this objective function via explicit optimization. Instead, this formulation serves as a conceptual framework that guides our method design. The actual quantization process is based on a set of heuristics and structure-aware strategies that approximate this objective in a computationally efficient and scalable manner.

Quantization Pipeline Overview. HBLLM integrates the Haar transform into a BiLLM-style quantization pipeline (see Algorithm 1 and Figure 2), consisting of the following key steps:

Preparation Phase: Compute the column-wise importance scores using a Hessian-based saliency metric.

2. Salient Column Selection and Quantization(SALIENT):

Sort columns by their ℓ 2 norm significance.

Select topK salient columns and determine M sal .

̂ W sal B = HaarQuant ( M sal W ) .

Choose the subset with the lowest quantization error.

3. Non-Salient Region Quantization:

Fill the missing values in salient columns using adjacent averages (FillAvg).

̂ W non-sal B = HaarQuant ( M sal W filled +( I -M sal ) W ) , where W filled is from FillAvg.

4. Adjustment and Refinement:

̂ W sal B = HaarQuant ( ˜ W ) .

Algorithm 1 Framework of HBLLM: Details of each function are shown in Algorithm E.1

func HBLLM ( W , X , β, λ )

Input:

W ∈ R n × m - weight matrix

X ∈ R r × d - calibration data

β - block size

λ - hessian regularizer

Output: B - haared binarized weights

1: H ← 2 XX ⊤ // ℓ 2 error hessian matrix

3: B ← 0 n × m

5: W b ← W : ,b : b + β

6: rows {·} ← SALIENT ( W : ,b : b + β , H c )

func Row-HaarQuant( W , rows {·} )

1: W filled ← FillAvg ( W : ,j / ∈ rows , rows {·} )

2: B filled ← HaarQuant ( W filled , ROW )

3: ̂ W ← W -B filled

4: B salient ← HaarQuant ( ̂ W : ,j ∈ rows , COL )

5: B ← B salient + B filled

6: return B

func Col-HaarQuant ( W , rows {·} )

1: B unsalient ← HaarQuant ( W : ,j / ∈ rows , COL )

2: ̂ W ← W -B filled

3: B salient ← HaarQuant ( ̂ W : ,j ∈ rows , COL )

7: if Row-HBLLM then

4: B ← B salient + B unsalient

5: return B

9: else if Col-HBLLM then

10: B : ,b : b + β ← Col-HaarQuant ( W b , rows {·} )

12: W : ,b + β : ← W : ,b + β : -E · H c b : b + β,b + β :

13: return B

3.3 HaarQuant: One-Bit Quantization in the Wavelet Domain

To boost expressiveness, we apply Haar wavelet transform to the weight matrix of linear layers, generating a frequency-domain coefficient matrix, followed by group-wise 1 -bit quantization.

To address limited numerical expressiveness, HBLLM introduces the HaarQuant algorithm. HaarQuant consists of three stages.

Haar Transform. A row of weights W is decomposed into low- and high-frequency coefficients via 1D Haar transform H :

where ̂ W is the Haar coefficient of W , H low-pass ( W ) and H high-pass ( W ) are low- and high-frequency coefficients, respecively.

Frequency-Aware Multi-Parameter Intra-Row Grouping. For each row, boundary candidates determined by the row are enumerated, and the best grouping with minimal quantization error is selected. Furthermore, we split the rows by frequency bands. This adaptive strategy captures intra-row structural differences better than global uniform boundaries used in BiLLM.

Coefficient Quantization. Each group ̂ W FP is quantized using sign-based binarization centered on its mean:

where α ∈ R d is the row-wise scaling factor and µ is the group-wise mean and ̂ W B is the result.

3.4 Structure-aware Grouping Strategies

To enhance the fidelity and adaptability of binary quantization under structural constraints, HBLLM introduces two structure-aware grouping strategies that operate along both column and row dimensions of the weight matrix.

Saliency-Driven Column Selection via ℓ 2 Norm. This strategy is used during salient column identification and quantization to overcome the limitations of prior heuristics based on fixed thresholds or simple magnitude criteria.

Columns are ranked by their ℓ 2 -norm scores, which correlate with their overall contribution to activation magnitude.

The topK columns are selected as salient and quantized in the Haar-transformed domain using column-wise transforms.

This approach helps preserve activation-critical directions, especially those dominated by outlier weights.

Frequency-Aware Multi-Parameter Intra-Row Grouping. This strategy is used during Haar domain quantization, where conventional row grouping lacks sensitivity to structural variations in weight distributions.

Each row is first decomposed into high- and low-frequency components based on Haar subbands.

Within each frequency band, coefficients are adaptively split into dense and sparse groups using band-specific, data-driven thresholds.

This grouping effectively doubles the number of quantization subgroups per row, enabling finer granularity and better error control.

Together, these strategies facilitate fine-grained, structure-preserving quantization across both dimensions of the weight matrix. To further guide saliency-based partitioning, we adopt the parameter importance metric used in BiLLM, defined as: s i = w 2 i / [ H -1 ] 2 ii , where H denotes the Hessian matrix of the layer, w i is the full-precision value of the i -th parameter, and [ H -1 ] ii is the i -th diagonal entry of the inverse Hessian.

This metric reflects the relative sensitivity of the loss to changes in each parameter: higher values indicate greater influence on the model's output, and thus prioritize that weight for accurate reconstruction.

3.5 Intra-frequency-band Mean Sharing

To reduce storage overhead, HBLLM shares a single mean value among 2 groups in the same frequency band within each row: µ shared = 1 n 1 + n 2 ( ∑ n 1 i =1 x i + ∑ n 2 j =1 y j ) . It not only reduces perparameter storage by 0 . 25 bits, but also maintains accuracy even slightly improving downstream task performance. This optimization achieves a trade-off between compression and accuracy, improving deployment viability.

3.6 Efficient Haar Implementation via Local Convolutions

Instead of costly matrix multiplication, HBLLM implements Haar transform using fixed local convolutions. There are only two predefined 1D kernels, [1 / 2 , 1 / 2] and [1 / 2 , -1 / 2] , whose kernel size is 2 . Furthermore, it can be hardcoded into the model for zero runtime initialization and no training or storage is needed. In complexity comparison, HBLLM needs O ( d ) operatons via convolutional sliding window, while FrameQuant needs O ( d 2 ) operations. As a result, HBLLM significantly lowers inference cost and is ideal for edge deployment.

Note: All methods are calibrated on C4 with 128 samples and a sequence length of 2048 . A block size of 128 is used for channel-wise quantization, as commonly done in prior work. N/A: ARB-LLMX method cannot run on a single 3090 GPU - 24GB. W-bits is the average weight overhead per weight. For more details, please refer to the appendix D.

4 Experiments

4.1 Experimental Settings

Models and Evaluation Datasets. In our study, we evaluate HBLLM on various models, including those from the OPT, LLaMA1 , LLaMA2 , and LLaMA3 , as well as the recently introduced f-R1-Distill-Llama8 B. Specifically, we utilize the OPT models with 1 . 3 B and 2 . 7 B parameters, the LLaMA1 and LLaMA2 models with 7 B and 13 B parameters for our evaluations, and the LLaMA3 model with 8 B parameters. We measure language modeling capabilities of these models by evaluating their perplexity on the C 4 [26], WikiText 2 [22] and PTB[21] datasets. Additionally, we assess zero-shot accuracy on various Common Sense Reasoning Tasks such as PIQA[4], BoolQ[7], OpenBookQA[23], WinoGrande[28], ARC-e, ARC-c[8], HellaSwag[36], which are commonly used for evaluating the performance of LLM quantization methods. To further enhance evaluation coverage, we also include COPA[27] for causal reasoning and LAMBADA[25] for long-context language modeling. All evaluations are conducted using the open-source LLM evaluation framework, LM-Evaluation-Harness[24].

Details of Experiments. All experiments are conducted with PyTorch on NVIDIA GeForce RTX 3090 GPUs with 24 GB of memory. For the calibration data, we follow the settings adopted in GPTQ and BiLLM, selecting 128 samples from the C 4 dataset, with a sequence length of 2048 . During quantization, we set the block size to 128 in BiLLM, PB-LLM, ARB-LLM, and HBLLM. Activations are kept in full precision (FP 16 ).

Baselines. We compare HBLLM against several state-of-the-art LLM binarization methods, including BiLLM, ARB-LLM and PB-LLM, ensuring that all implementations adhere to the details provided in their respective papers. BiLLM, ARB-LLM and PB-LLM all utilize the PTQ approach for model calibration through OBQ based method of GPTQ. For ARB-LLM, we evaluate two of its bestperforming variants, ARB-LLMX and ARB-LLMRC. Both ARB-LLMx and ARB-LLMRC employ the salient column bitmap and group bitmap (CGB) for better performance. For PB-LLM, which allows variable ratios of salient weights to enhance accuracy, we have set the ratio of salient weights to 10% to ensure the average bit width of weight parameters remains below 2 bits. Given the significant accuracy improvements demonstrated by HBLLM over traditional binarization techniques, we also include a comparison with a leading method using orthogonal transforms: FrameQuant. For FrameQuant, quantization is performed not in the original weight space but in the structured orthogonal basis constructed through Fusion Frames. We evaluate two configurations: FrameQuant ( r = 1 . 0) and FrameQuant ( r = 1 . 1) , where the redundancy factor r controls the amount of redundancy introduced during the transformation.

4.2 Perplexity and Accuracy Results of 1-2 Bit Quantized Models

The perplexity and zero-shot accuracy results of previous 1 -2 bit quantization methods and the proposed HBLLM are presented in Table1. HBLLM consistently outperforms existing 1 -2 bit quantization techniques across all evaluation metrics.

Specifically, HBLLM reduces the language modeling perplexity by 33% -66% compared to previous methods, while achieving substantial improvements in QA task accuracy, with relative gains ranging from -0 . 73% to +11 . 3% . In our experiments, HBLLM slightly outperforms FrameQuant, a 2 . 2 -bit quantization method, and exhibits a particularly significant advantage on the LLaMA-3-8B model. Moreover, when compared with BiLLM and ARB-LLMX, HBLLM-col, demonstrates a clear advantage in both perplexity and accuracy, despite operating at comparable or lower bit-widths. These results indicate that HBLLM effectively narrows the performance gap between quantized models and their Float16 counterparts, achieving 1 . 22 × to 2 . 48 × of the original perplexity and retaining 73 . 8 %-88 . 8 %of the original QA accuracy.

4.3 Ablation Study

Salient Column Selection Criterion. To evaluate the impact of selection criteria in salient column screening on quantization effectiveness, we compare two strategies: the column ℓ 1 norm and the column ℓ 2 norm as significance indicators. Experimental results in Table 2a reveal that the column ℓ 2 norm consistently achieves lower quantization error and superior performance in downstream tasks,

(a) Study of salient column selection criterion

(c) Effectiveness of shared mean

(b) Study of grouping granularity

(d) Study of partitioning candidates number

indicating its greater effectiveness in capturing energy distribution across columns and enhancing quantization quality.

Granularity of Group Quantization. To explore the influence of grouping granularity on model performance, we compare global grouping with row-wise grouping strategies, evaluating both quantization error and perplexity, as shown in Table 2b. The results reveal that row-wise grouping significantly reduces quantization error and achieves lower perplexity compared to global grouping. This suggests that finer-grained row-wise partitioning better preserves local data fidelity, leading to improved quantized inference performance.

Shared Mean Strategy. Under the standard dual-partition quantization setting, we further explore a compression strategy that shares the quantization center across two partitions within each row. By unifying the mean for both partitions, the storage overhead of quantization coefficients can be significantly reduced. Experimental results in Table 2c demonstrate that the shared mean strategy even slightly reduces quantization error without degrading perplexity, verifying its effectiveness.

Choice of Partitioning Number. We investigate the impact of varying the number of partition candidates on final quantization performance under the row-wise grouping setting. Specifically, for each row, we generate partition candidates based on absolute value percentiles ranging from 10% to 90% , and evaluate the corresponding quantization error and perplexity, as shown in Table 2d. Experimental results indicate that moderately increasing the number of partition candidates can effectively reduce quantization error and further lower perplexity, while excessive partitioning yields diminishing returns and increases computational cost. Consequently, we adopt 40 partition candidates as the default setting to balance performance and efficiency.

4.4 Time and Memory Analysis

Time Comparison. As a binary PTQ framework, HBLLM eliminates the need for finetuning. The introduction of Haar wavelet transforms requires additional computation during quantization, yet this overhead remains fully acceptable. As shown in Table 3, HBLLM increases the quantization time by approximately 20% -30% compared to BiLLM across different model sizes. It is worth noting that ARB-LLMX and FrameQuant fail to complete quantization for LLaMA1 -13B and LLaMA1 -30B under the single-GPU-24 GB setting, while HBLLM

successfully completes the process, demonstrating better scalability.

Memory Comparison. As shown in Table 4, HBLLM-col achieves better performance while occupying a storage size comparable to ARBLLM. By employing a grouped shared-mean strategy, HBLLM improves compression efficiency without sacrificing performance. Specifically, HBLLM-col applies Haar transforms along the column dimension, such that only one grouped quantization operation is required per row on the transformed coefficients. Compared to HBLLM-row, this leads to reduced data fidelity but provides clear advantages in storage cost. Notably, the reported memory usage is measured at runtime in our setup and may be

influenced by model variants and implementation choices, leaving room for further engineering optimizations. The detailed storage calculation formulas can be found in the appendix D.

4.5 Inference Latency Estimation

To evaluate the inference latency of HBLLM, we conduct an experiment that combines direct measurement with estimation. Due to there is no existing inference framework that fully supports the dequantization algorithm used in HBLLM, we test GEMV on layers from the OPT-175B model instead. The tests are run on an NVIDIA P100 GPU following the GPTQ benchmark setup [1]. Our estimation results show that the inference latency of HBLLM is approximately 31 . 8% of the FP16 baseline inference time. For more details, please refer to the appendix G.

5 Conclusion

We introduce a 1 -bit weight only quantization HBLLM, which applies Haar transform to BILLM pipeline. Besides quantifying the coefficients on frequence domain, HBLLM integrates two innovative structure-aware grouping strategies to enhance fidelity. Furthermore, HBLLM optimize storage efficiency. As a results, HBLLM outperforms SOTA QAT quantization methods of LLM at 1 -bit across different LLM families and tests. The current HBLLM supports only quantized dense models. Next, we will focus on the MoE PTQ algorithm.

References

Adepu H, Zeng Z, Zhang L, et al. FrameQuant: flexible low-bit quantization for transformers. Proceedings of the 41st International Conference on Machine Learning . 2024: 203-227.

Ashkboos S, Croci M L, do Nascimento M G, et al. SliceGPT: Compress Large Language Models by Deleting Rows and Columns. The Twelfth International Conference on Learning Representations . 2024.

Lin J, Tang J, Tang H, et al. AWQ: Activation-aware weight quantization for on-device LLM compression and acceleration. Proceedings of Machine Learning and Systems , 2024, 6: 87-100.

Bisk Y, Zellers R, Gao J, et al. PIQA: Reasoning about physical commonsense in natural language. Proceedings of the AAAI Conference on Artificial Intelligence . 2020: 7432-7439.

Chee J, Cai Y, Kuleshov V, et al. QuIP: 2-bit quantization of large language models with guarantees. Advances in Neural Information Processing Systems , 2023, 36: 4396-4429.

Chen H, Lv C, Ding L, et al. DB-LLM: Accurate dual-binarization for efficient LLMs. Findings of the Association for Computational Linguistics: ACL 2024 . 2024: 8719-8730.

Clark C, Lee K, Chang M W, et al. BoolQ: Exploring the surprising difficulty of natural yes/no questions. Proceedings of NAACL-HLT . 2019: 2924-2936.

Clark P, Cowhey I, Etzioni O, et al. Think you have solved question answering? Try ARC, the AI2 reasoning challenge. arXiv preprint arXiv:1803.05457 , 2018.

Duan Y, Liu F, Jiao L, et al. SAR image segmentation based on convolutional-wavelet neural network and Markov random field. Pattern Recognition , 2017, 64: 255-267.

Edalati A, Ghaffari A, Nejad M G, et al. OAC: Output-adaptive calibration for accurate post-training quantization. Proceedings of the AAAI Conference on Artificial Intelligence . 2025, 39(16): 16453-16461.

Frantar E, Alistarh D. SparseGPT: Massive language models can be accurately pruned in one-shot. International Conference on Machine Learning . 2023: 10323-10337.

Frantar E, Ashkboos S, Hoefler T, et al. OPTQ: Accurate post-training quantization for generative pre-trained transformers. 11th International Conference on Learning Representations . 2023.

Gong R, Ding Y, Wang Z, et al. A survey of low-bit large language models: Basics, systems, and algorithms. arXiv preprint arXiv:2409.16694 , 2024.

Huang H, He R, Sun Z, et al. Wavelet-SRNet: A wavelet-based CNN for multi-scale face super resolution. Proceedings of the IEEE International Conference on Computer Vision . 2017: 1689-1697.

Huang W, Liu Y, Qin H, et al. BiLLM: Pushing the limit of post-training quantization for LLMs. Proceedings of the 41st International Conference on Machine Learning . 2024: 20023-20042.

Huang W, Zheng X, Ma X, et al. An empirical study of LLaMA3 quantization: From LLMs to MLLMs. Visual Intelligence , 2024, 2(1): 36.

Jo D, Kim T, Kim Y. Mixture of scales: Memory-efficient token-adaptive binarization for large language models. Advances in Neural Information Processing Systems , 2024, 37: 137474-137494.

Li Z, Yan X, Zhang T, et al. Arb-LLM: Alternating refined binarizations for large language models. arXiv preprint arXiv:2410.03129 , 2024.

Liang C, Zuo S, Zhang Q, et al. Less is more: Task-aware layer-wise distillation for language model compression. International Conference on Machine Learning . 2023: 20852-20867.

Mallat S. A Wavelet Tour of Signal Processing . Elsevier, 1999.

Marcus M, Kim G, Marcinkiewicz M A, et al. The Penn Treebank: Annotating predicate argument structure. Human Language Technology: Proceedings of a Workshop held at Plainsboro . 1994.

Merity S, Xiong C, Bradbury J, et al. Pointer Sentinel Mixture Models. International Conference on Learning Representations . 2017.

Mihaylov T, Clark P, Khot T, et al. Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing . 2018: 2381-2391.

Mukherjee S, Liu X, Zheng G, et al. Few-shot learning evaluation in natural language understanding. Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track . 2021.

Paperno D, Kruszewski G, Lazaridou A, et al. The LAMBADA dataset: Word prediction requiring a broad discourse context. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics . 2016: 1525-1534.

Raffel C, Shazeer N, Roberts A, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research , 2020, 21(140): 1-67.

Roemmele M, Bejan C A, Gordon A S. Choice of Plausible Alternatives: An Evaluation of Commonsense Causal Reasoning. AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning . 2011: 90-95.

Sakaguchi K, Bras R L, Bhagavatula C, et al. Winogrande: An adversarial Winograd schema challenge at scale. Communications of the ACM , 2021, 64(9): 99-106.

Shang Y, Yuan Z, Wu Q, et al. PB-LLM: Partially binarized large language models. The Twelfth International Conference on Learning Representations . 2024.

Shridhar K, Stolfo A, Sachan M. Distilling reasoning capabilities into smaller language models. Findings of the Association for Computational Linguistics: ACL 2023 , 2023: 7059-7073.

Sun M, Liu Z, Bair A, et al. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695 , 2023.

Touvron H, Lavril T, Izacard G, et al. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

Xiao G, Lin J, Seznec M, et al. SmoothQuant: Accurate and efficient post-training quantization for large language models. International Conference on Machine Learning . 2023: 38087-38099.

Xu Y, Han X, Yang Z, et al. OneBit: Towards extremely low-bit large language models. The Thirty-eighth Annual Conference on Neural Information Processing Systems . 2024: 66357-66382.

Yao Z, Yazdani Aminabadi R, Zhang M, et al. ZeroQuant: Efficient and affordable post-training quantization for large-scale transformers. Advances in Neural Information Processing Systems , 2022, 35: 27168-27183.

Zellers R, Holtzman A, Bisk Y, et al. HellaSwag: Can a machine really finish your sentence? Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics . 2019: 4791-4800.

Zhang S, Roller S, Goyal N, et al. OPT: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 , 2022.

A Introducing Haar Wavelet Transform into LLM Quantization

Haar wavelet transform converts an original vector into a new matrix containing low- and high-frequency information by computing averages and differences. This transformation can be extended to larger vectors and multidimensional data and is commonly used in image processing and signal analysis.

Applying the Haar wavelet transform to LLM quantization offers three advantages:

The inverse quantization set becomes richer in representation, which can be demonstrated via the CIQ metric.

The data distribution becomes more concentrated: with approximately 65% probability, the variance of the high- and low-frequency coefficient sets in each row is smaller than before the transformation.

The additional computational cost for inference is O ( d ) , where d is the input length, as the Haar transform can be implemented using local convolutional layers, resulting in lower cost than methods such as FrameQuant.

A.1 Definition of Haar Transform

Let the one-dimensional input signal be x . The output after wavelet transformation is:

where a i denotes low-frequency coefficients and b i denotes high-frequency coefficients.

This process can also be expressed in matrix form:

Because the Haar matrix is orthogonal, the inverse Haar transform is:

A.1.1 Row-wise and Column-wise Haar Transforms on Matrices

Define row-wise and column-wise Haar transforms of a matrix W as:

Below is a simple example showing how to apply Haar transforms to a matrix.

Given a 4 × 4 matrix defined as:

where

the result after applying row-wise Haar transform is denoted as

and after applying column-wise Haar transform is denoted as

where each element of matrices is multiplied by the coefficient √ 2 .

B Relationship Between Richness of Inverse Quantization Set and Model Fidelity

B.1 Definition and Application of CIQ Metric

The cardinality of an inverse quantization set (CIQ) measures the number of distinct values that can be recovered from quantized weights in each row. For linear quantization without partitioning, CIQ equals the bit-width of quantized weights, since linear quantization evenly distributes several points across the original data range. When partitioning strategies are applied, it characterize expressiveness of the quantization algorithm.

In HBLLM, which follows the GPTQ quantization scheme, quantization is done per row of each matrix block. We define CIQ in terms of a single row of a matrix block in the following discussion without loss of generality. Furthermore,

CIQ = min { row length , maximum recovery ability under given quantization parameters } .

holds.

To study the composition of IQ, we introduce a mapping from the set of quantized weights to the set of dequantized weights. Several mappings from quantized weights to inverse quantized values exist:

Identity mapping

Group merging mapping

Residual merging mapping

Inverse transformation mapping

For example, an inverse quantization set brought by residual merging mapping is defined as:

Furthermore, where | · | is the cardinality of a set.

An inverse quantization set brought by Haar inverse transformation is defined as:

And then,

Let an inverse quantization set produced by BiLLM algorithm be denoted as IQ BiLLM . According to BiLLM algorithm, IQ BiLLM can be expressed in the following form:

where IQ sal residual and IQ non-sal represent inverse quantization sets of the salient and non-salient parts, respectively. Lemma 1. BiLLM has at most 8 different dequantized values per row.

Proof. According to (17) and the definition of CIQ, we can infer that

Since the salient part adopts a residual approximation strategy, it follows that

Given that | X sal 1 | = | X sal 2 | = | X non-sal 1 | = | X non-sal 2 | = 2 , substituting (19) into (18) yields:

Next, we analyze theoretical upper bounds of CIQ for HBLLM algorithms. We first consider the case of HBLLM-col. Let an inverse quantization set produced by HBLLM-col be denoted as IQ HBLLM-col . According to HBLLM-col algorithm, IQ HBLLM-col can be expressed in the following form:

where IQ sal Haar and IQ non-sal Haar represent the dequantized sets of the salient and non-salient parts, respectively. Both parts employ a group quantization strategy under Haar transform, so the upper bound for each part is the product of the cardinality of the two Haar sub-band quantization sets. Additionally, within HBLLM-col algorithm, each sub-band has two groups, resulting in a total of four quantized values. Based on the above analysis, we arrive at the second conclusion.

Lemma 2. HBLLM-col has at most 64 different dequantized values per row.

Proof.

The CIQ upper bound of HBLLM-row algorithm is significantly larger than that of HBLLM-col algorithm. Let an inverse quantization set produced by HBLLM-row algorithm be denoted as IQ HBLLM-row .By HBLLM-row algorithm, IQ HBLLM-row can be shown in the following form:

Unlike HBLLM-col, the non-salient part of HBLLM-row encompasses the entire matrix area, resulting in overlap with the salient part. Therefore, HBLLM-row incorporates a residual approximation on the salient part, further increasing the CIQ upper bound. Additionally, within HBLLM-row algorithm, each sub-band has two groups, resulting in a total of four quantized values. Based on the above analysis, we arrive at the third conclusion.

Lemma 3. HBiLLM-row can have over 1024 different dequantized values per row.

Proof.

Lemma 3 is a result that holds under the assumption that there are sufficiently many columns in the salient part. In practical algorithms, due to the dual constraints of the quantized matrix size and the total bitrate, the upper bound of CIQ HBLLM-row is much less than 1024. The specific upper bound can be described in Lemma 4.

Lemma 4. Let the size of a quantized matrix block be d × d , where d ≤ 256 , and let the proportion of the number of columns in the salient part to the total number of columns be p (where 0 < p < 1 ). Then, we have

Proof. It is easy to get by Lemma 2 and Lemma 3.

Theorem 1. Under the same proportion p of the salient part, CIQ HBLLM-row ≥ CIQ HBLLM-col holds.

Proof. This follows from Lemma 4.

where IQ sal HBLLM-row is defined as and

(a) HBLLM-row

1.8741.7361.599 1.4611.3241.1861.0480.9110.773 0.6360.4980.3610.2230.085 0.052 0.190 0.327 0.465 0.602 0.740 0.878 1.015 1.153 1.290 1.428

x10-2

DistributionofinversequantizationValues(Unique:23)

1.903 1.7571.6101.4641.3171.1711.0240.8780.7310.5850.4380.292 0.145 0.001 0.147 0.294 0.440 0.587 0.733 0.880 1.026 1.173 1.319 1.466 1.612

x10-2

(b) HBLLM-col

B.2 Measured Values of the CIQ Metric for the HBLLM Algorithm

To validate the theoretical analysis of CIQ, we present two representative examples under the HBLLM-row and HBLLM-col schemes, respectively. These examples illustrate how the CIQ values measured in practice align with the theoretical bounds derived earlier.

Example 1: HBLLM-row. As shown in B.1a, consider a row of length 128 where 14 elements are marked as salient. After quantization and reconstruction, the total number of distinct values observed is 42. Among these, 30 values originate from the non-salient part (i.e., | IQ non-sal Haar | = 30 ) and 7 values from the salient part (i.e., | IQ sal Haar | = 7 ).

Although the salient part involves residual merging, the final reconstructed set still satisfies:

which confirms that the practical CIQ value stays within the theoretical upper bound in Lemma 4.

Example 2: HBLLM-col. As shown in B.1b, In a second example under the HBLLM-col scheme, a row consists of 119 non-salient and 9 salient elements. The measured CIQ is 23, with 16 values from the non-salient part and 7 values from the salient part. This result again satisfies:

These observations demonstrate that in practice, the effective size of the inverse quantization set is significantly below the worst-case bounds, especially when some quantized values are shared or overlap. They also confirm the effectiveness of the HBLLM decomposition strategies in maintaining a compact and expressive representation of quantized weights.

B.3 Limitations of CIQ Metric Analysis and the Necessity of Introducing a Structure-Aware Grouping Strategy

Although HBLLM significantly outperforms BiLLM in the CIQ metric, it lacks a clear advantage in quantization performance without the structure-aware grouping strategy introduced in this paper.

To assess the performance differences before and after implementing this strategy, we conduct experiments evaluating perplexity and QA accuracy. Experiment data can be found in Table B.1:

BiLLM+ ℓ † 2 : Employing ℓ 2 -based saliency-driven column selection together with multi-parameter intra-row grouping.

Haar+BiLLM: This refers to a method obtained by removing the ℓ 2 -norm-based saliency-driven column selection and multi-parameter intra-row grouping strategies from HBLLM. This method integrates the one-dimensional discrete Haar wavelet transform applied row-wise or column-wise, thereby deriving two approaches: Row-Haar+BiLLM and Col-Haar+BiLLM, respectively.

DCT+BiLLM: This approach applies the BiLLM algorithm to coefficient matrices obtained from the onedimensional Discrete Cosine Transform (DCT) applied row-wise or column-wise on weight matrices, resulting in Row-DCT+BiLLM and Col-DCT+BiLLM. Unlike the Haar+BiLLM method, DCT+BiLLM uses a global transformation strategy by first mapping the entire matrix to the Fourier domain before quantization. In contrast, Haar+BiLLM applies the Haar transform to matrix blocks, with quantization following the BillM process, making it a local orthogonal transformation.

HBiLLM+: These method, refer to those derived from Haar+BiLLM, utilizing the strategies proposed in our paper. They include HBLLM-row+ ℓ 2 , HBLLM-col+ ℓ 2 , HBLLM-col+ ℓ † 2 and HBLLM-col+ ℓ † 2 .

Note: ℓ 2 denotes activation of ℓ 2 -norm-based saliency-driven column selection; † denotes activation of frequency-aware multi-parameter intra-row grouping.

The main experimental results are summarized as follows:

Directly applying Haar transform into BiLLM pipeline does not significantly improve 1 -bit quantization performance.

-Row-Haar + BiLLM shows slight improvement.

-Col-Haar + BiLLM decreases performance.

Implementing the 'saliency-driven column selection via ℓ 2 norm' strategy leads to:

-Significant improvements for HBLLM-row + ℓ 2 and HBLLM-col + ℓ 2 compared to their predecessors.

-Perplexity tests show:

32 -64% reductions on C4 and Wiki2 test sets.

HBLLM-col + ℓ 2 shows notable improvements, but still lags behind BiLLM on the PTB test set.

-QA testing accuracy improves by 3 -10% .

Further introducing 'frequency-aware multi-parameter intra-row grouping' results in:

-HBLLM-row + ℓ 2 + Row-wise-grouping is the best.

-26 -45% reductions in perplexity on C4 and Wiki2.

-Significant improvements on the PTB test set, surpassing BiLLM.

-Cumulative accuracy in QA testing increases by 2 -8% .

This experimental result demonstrates that introducing a structure-aware grouping strategy is essential for effectively combining the Haar transform with the BiLLM algorithm.

B.4 Effectiveness of Haar Transform and the Importance of Local Orthogonality

As shown in Table B.1, although HBLLM integrates multiple strategies, it is important to disentangle the specific contribution of the Haar-based frequency decomposition from other components such as saliency selection and structure-aware grouping. To this end, we conduct dedicated ablation studies to quantify the standalone effectiveness of the Haar transform and contrast it with global orthogonal alternatives such as Discrete Cosine Transform (DCT).

We summarize our key observations below:

Effectiveness of Haar Transform: While incorporating either the ℓ 2 -based saliency selection or the structureaware grouping alone yields only modest improvements to BiLLM, introducing the Haar transform leads to consistently more substantial gains in both perplexity and QA accuracy. Notably, even under partial activation (e.g., HBLLM-col+ ℓ 2 ), the models outperform their BiLLM counterparts.

-Both HBLLM-row+ ℓ 2 † and HBLLM-col+ ℓ 2 † significantly outperform BiLLM and BiLLM+ ℓ † 2 , underscoring the crucial role of Haar in preserving the frequency-domain structure of weights.

-Row-Haar+BiLLM, even without ℓ 2 -norm-based saliency-driven column selection or grouping, shows consistent performance gains, confirming that Haar decomposition independently enhances quantization representation capacity.

Local vs. Global Orthogonal Transforms: We further analyze the impact of replacing Haar transform with global DCT.

-Applying the global row-wise DCT results in severe degradation across all benchmarks, with RowDCT+BiLLM performing significantly worse than BiLLM.

-Applying the global column-wise DCT offers moderate improvement on LLaMA2-7B; however, ColDCT+BiLLM still lags behind BiLLM in all OPT-1.3B tests.

-These results highlight that global transforms struggle to capture local variations in weight distributions, which are effectively preserved by block-wise Haar decomposition.

As shown in Table B.2, global transforms such as DCT also incur substantial computational overhead compared to local Haar transforms:

In conclusion: Haar transform independently and robustly contributes to quantization fidelity, even in the absence of auxiliary strategies such as ℓ 2 -norm-based saliency-driven column selection or grouping; local orthogonal transforms like Haar are consistently more effective than global ones like DCT in preserving localized frequency-domain structures-an essential property for stable and expressive 1-bit quantization.

C Analysis of the Correlation Between Data Concentration and Model Fidelity

In this section, we explores the positive correlation between improved data concentration and enhanced model fidelity after quantization, following the application of the Haar transform and structure-aware grouping strategy proposed by HBLLM. This correlation provides a theoretical foundation for the effectiveness of the HBLLM quantization method. We first observe that the concentration of coefficient distribution improves with a 65% probability after applying the Haar row transform. Based on this observation, we mathematically model the probability of variance improvement in data concentration. We then apply this variance improvement probability to the HBLLM-row and HBLLM-col methods to validate the correlation between enhanced data concentration and improved model fidelity after quantization.

C.1 Improvement of Data Concentration by Haar Transform

We discusses the improvement in data concentration resulting from the Haar transform, which is typically described by variance-lower variance indicates higher data concentration. We examine the impact of the Haar row transform on the distribution characteristics of the weight matrix. Specifically, we select a matrix block from the OPT-1.3B model and compare the variance of each original weight vector with the variances of the low-frequency and high-frequency subbands obtained after Haar decomposition. The variances of each row from different methods are arranged in ascending order, as shown in Figure C.1. Notably, approximately 65% of the rows exhibit a variance in at least one subband that is lower than the original value, indicating that the Haar row transform generally enhances data concentration.

5.0×10-4

4.5

4.0

3.5

3.0

2.5

二

Original

二

Haar Low

Haar High

2.0

Therefore, we anticipate that the application of HBLLM's structure-aware grouping strategy may further improve data concentration.

C.2 Mathematical Modeling of Variance Improvement Probability

To quantify the improvement in data concentration achieved by the Haar transform combined with the structureaware grouping strategy, we introduce a random variable p to describe the probability of improvement in data concentration for each row after applying the strategy, as well as the expected probability of improvement E ( p ) for each matrix block. Let the matrix block size be d × d , and the j -th row, after the Haar row transform, be divided into M subgroups, each containing n j i coefficients, with an intra-group variance of V j i . The variance of the entire row before transformation is denoted as ˜ V 0 . The sample value p j for the j -th row is calculated as follows:

where, the sign function sign( · ) takes the value of 1 only when the subgroup variance is less than the original row variance; otherwise, it is 0 . The value p j represents the proportion of coefficients in the j -th row that belong to subgroups with improved concentration. Based on this, we define the expected value E ( p ) for the entire matrix block to characterize the average level of overall data concentration improvement:

C.3 Analysis of the Relationship Between Data Concentration and Quantization Fidelity

To further investigate the impact of improved data concentration on model fidelity, we collected data on the probability of variance improvement, the relative ℓ 2 error of matrix blocks before and after quantization, and the corresponding model fidelity, as shown in Figure C.2. The relative ℓ 2 error serves as the optimization criterion for HBLLM quantization, while model fidelity is measured by perplexity-lower perplexity indicates higher fidelity after quantization.

We analyzed the distribution changes of relative ℓ 2 errors for all matrix blocks across different models before and after quantization, and we plotted the perplexity performance under various grouping strategies. The experimental results demonstrate that HBLLM not only significantly enhances data concentration but also effectively mitigates the growth of quantization error, thereby better preserving the model's original performance.

Figure C.2a displays the proportion E ( p ) of Haar coefficients across all matrix blocks that meet the variance improvement criterion after combining the row wavelet transform and grouping strategy. The points on the graph represent E ( p ) for a matrix block in a quantized model, with each graph showing the values of E ( p ) arranged in ascending order. As illustrated, after applying the strategy, over 65% of Haar coefficients in all matrix blocks achieved an improvement in data concentration, with median improvements of 67% and 68% , respectively.

The results from Figure C.2 lead to the following conclusion: after applying Haar transform combined with the structure-aware grouping strategy proposed by HBLLM, there is a positive correlation between the improvement in data concentration and the enhancement of model fidelity after quantization.

C.4 Empirical Analysis of Saliency Ratio Distribution Across Layers and Blocks

To better understand the structure-aware quantization behavior of our method, we analyze how saliency ratio is distributed across different types of layers and transformer blocks. A saliency ratio is defined as the proportion of weights selected by our Hessian-based criterion during quantization.

We visualize the results using two plots: a histogram showing the saliency ratio distribution across different layer types, and a line plot depicting the evolution of block-wise saliency ratio across the network. As shown in Figure C.3a and Figure C.3b, query and key projections exhibit low saliency (mostly below 5% ), while value, gate, and up-projection layers tend to have significantly higher saliency. Additionally, we observe a gradual increase in saliency ratio from shallow to mid transformer blocks, followed by stabilization.

These results confirm that saliency ratio is not uniformly distributed, but highly dependent on layer type and block depth. This validates our strategy of adapting quantization granularity according to the structure of the model.

D Storage Cost Analysis and Inference Execution Process

To comprehensively assess compression effectiveness, we divide the stored data into three types: weight overhead , coefficient overhead, and flag overhead.

(a) Variance Improvement Score E ( p ) per block

S/0

Median:

●0.676

Median:

0.650

RelativeQuantizationError

Median:

●40.72

Median:

·38.48

●0.670

0.650

40.26

●37.10

0.674

6590

36.82

●34.92

02'0

●0.684

0.70

●0.663

50

●35.59

● 34.46

1

E(p)

0.65

0.65

0.60

OPT-1.3B

0.60

LLaMA2-7B

OPT-1.3B

LLaMA2-7B

HBLLM-col: global

HBLLM-col: row-wise

HBLLM-row: global

HBLLM-row: row-wise

HBLLM-col: global

HBLLM-col: row-wise

HBLLM-row: global

HBLLM-row: row-wise

40

103

Perplexity

→35

30

102

25

20

101

C4

Wiki2

PTB

C4

Wiki2

PTB

OPT-1.3B

LLaMA2-7B

HBLLM-col: global

HBLLM-col: row-wise

HBLLM-row: global

-—HBLLM-row: row-wise

(c) Perplexity Comparison

[0,5%]

(5%,10%]

(10%,15%]

0.8

(15%,20%]

Proportion

0.6

0.4

0.2

0.0

transformer q

block

k

out

gate

up

down

(a) Saliency Ratio Distribution Across Layers and Blocks

18

Mean

Median

16

Min

14

(%)

12

Saliency

10

8

6

4

transformerq

2

block

k

V

out

gate

up

down

mean

Figure C.3: The Distribution of Saliency between Different Types of Weight Layers and across Different Transformer Blocks.

Weight Overhead (W-bits) . This refers to the number of bits used to store binarized weights. In standard 1-bit schemes, each weight requires only 1 bit. However, in schemes that retain salient weights, these may be stored with higher precision (e.g., 2 -bit or 8 -bit), increasing the overall weight overhead.

To increase fidelity, some weights would use more than 1 bit. For example, the salient part employs two 1-bit values with residual approximation. As a result, the average weight overhead per weight (denoted as W-bits) of Billm results would become fractional, such as 1 . 08 bits.

Coefficient Overhead (C-bits) . This refers to the additional bit-width required to store scaling factors and means. For example, OneBit introduces two scaling vectors per row or column, while ARB-LLMRC further computes scaling factors for both rows and columns. Although these parameters are smaller in size than the weight matrix, they must be tightly controlled in precision-critical applications.

Flag Overhead (F-bits) . These bits are used to store indicators for salient/non-salient weights (such as the "salient column" tag in PB-LLM or bitmap/group masks in BiLLM), or group affiliation information (e.g., group IDs).

To objectively evaluate the compression efficiency of various methods, we use the 'average bit-width per weight' ( Average-Bit ) as a unified metric. This metric characterizes not only storage cost but also the bandwidth required from memory to GPU registers. The average bit-width is computed as:

here, the total number of parameters refers to the product of the matrix dimensions. The structure expansion factor accounts for mismatches in the number of stored units and original parameters (e.g., 1 for non-restructuring methods, > 1 for methods like FrameQuant).

F-bits

W-bits

C-bits

High-Precision (e.g. INT8) Weight

Non-Salient

Non-Salient

Input

ndundno

Mask

Salient

Mask

Salient

242

59

(a) PB-LLM

(b) BiLLM

Non-Salient

Input

+

ndano

Mask

Salient

(c) ARB-LLM

Mask

Mask

Non-Salient

Non-Salient

里

nduj

Mask

里

日

+

ndund

Mask

里

Salient

Salient

(d) Row-HBLLM

(e) Col-HBLLM

D.1 Data Distributions of Various Binarization Methods

Taking LLaMA 1 -7 B as an example, Figure D.1 and Table D.1 illustrate how different LLM binarization methods distribute their storage data. PB-LLM uses a hybrid-precision quantization strategy that encodes 10% of weights with 8-bit asymmetric linear quantization and binarizes the remaining 90% . A 1-bit flag differentiates salient from non-salient weights. Non-salient parts are recovered using shared scaling and mean vectors, while salient weights use separate parameters and zero-points. All coefficients are stored in FP16 and shared per output channel, resulting in 0 . 5 bits of coefficient overhead per weight. PB-LLM's AvgBit is 3 . 2 bits.

BiLLM extends basic binarization with residual approximation for salient weights. Columns are divided into salient and non-salient parts. 90% of weights use standard 1 -bit encoding, while the salient portion is enhanced with residual binarization. This yields a weight overhead of 1 . 09 bits per weight. Additional structure information (bitmap, grouping) accounts for 1 . 008 bits per weight. With FP 16 -stored coefficients and two sets of residual parameters for salient parts, total coefficient overhead is 0 . 875 bit per weight. Hence, BiLLM's AvgBit is 2 . 973 bits.

ARB-LLM introduces group modeling for salient columns based on BiLLM to better capture complex distributions. weight overhead remains 1 . 09 bits per weight. Structure metadata (CGB) takes 1 . 008 bits per weight. All

coefficients are stored in FP 16 . Salient columns are grouped and assigned two sets of second-order coefficients, bringing coefficient overhead to 1 . 25 bits per weight. ARB-LLM's AvgBit reaches 3 . 348 bits.

HBLLM extends BiLLM by introducing structure-aware grouping strategies to improve quantization fidelity in the Haar domain. The weight matrix is split into salient and non-salient parts. Salient columns undergo column-wise Haar transforms, while non-salient parts are transformed along either row or column directions, followed by grouped binarization. All weights are quantized using standard 1 -bit encoding. HBLLM-col has a weight overhead of 1 . 00 bit per weight; HBLLM-row employs a neighborhood averaging strategy (FillAvg) to reconstruct missing values, increasing the weight overhead to 1 . 08 bits/weight.Saliency bitmaps and frequency-aware grouping metadata add 1 . 008 bits per weight. All reconstruction coefficients are stored in FP 16 . HBLLM-row forms four subgroups per row with independent scale and mean values, resulting in a coefficient overhead of 1 . 25 bits per weight. HBLLM-col shares four subgroups across two rows, averaging two groups per row, and applies intra-band mean sharing to reduce coefficient overhead to 0 . 875 bit per weight. The final average bit-widths of HBLLM-row and HBLLM-col are 3 . 418 and 2 . 883 bits, respectively.

D.2 Details of Average Bit-Width Calculation

The average bit-width of a quantized matrix ̂ W ∈ R n × m is defined as the total memory cost (in bits) divided by the number of elements in the original matrix:

For W ∈ R n × m , block size k , the memory of ̂ W after standard row-wise binarization is

Moreover, second-order row-wise binarization can be represented as

since row-wise µ 1 and µ 2 can be combined together as µ = µ 1 + µ 2 . Thus, the memory required by BiLLM can be formulated as

where c is the number of salient columns for W .

Similarly, we can formulate the memory occupation of first-order row-column-wise binarization and ARB-RC as

In addition, since CGB is used in the experiments, the total memory of ARC-RC + CGB is

Furthermore, we formulate the memory cost of PBLLM by considering both unsalient weights and salient weights as

where r binary denotes the ratio of the binarized weights.

HBLLM-row adopts four subgroups per row with independent α and µ , intra-band mean sharing, and a neighborhood-based reconstruction strategy (FillAvg), which increases the group bitmap cost.

HBLLM-col shares four subgroups across two rows and applies intra-band mean sharing.

Example: Average Bit-width of HBLLM-row and HBLLM-col

Assume W ∈ R n × m , block size k = 128 and the number of salient columns is c = 0 . 08 m . The total memory cost for HBLLM-row can be expressed as:

Thus, the average bit-width is:

Similarly, for HBLLM-col:

Then, the average bit-width becomes:

E HBLLM Implementation

The implementation details of the HBLLM quantization pipeline are provided in Algorithm E.1.

Algorithm E.1 HBLLM: Detailed functions process

func SALIENT ( W , H c ) 1: S ← W 2 / [ H c b : b + β,b : b + β ] 2 // salient matrix 2: rows {·} ← topk ( ∥ S ∥ 2 , dim = 0) 3: e ←∞ // searching error 4: K ← 0 // optimal number of salient columns 5: for i = 1 , 2 , . . . , len ( rows ) do 6: B 1 ← BINARY ( W : ,j ∈ rows [: i ] ) 7: B 2 ← BINARY ( W : ,j / ∈ rows [: i ] ) 8: if ∥ W - ( B 1 ∪ B 2 ) ∥ 2 < e then 9: e ←∥ W - ( B 1 ∪ B 2 ) ∥ 2 10: K ← i 11: end if 12: end for 13: return rows [: K ] func HAARQUANT ( W , mode ∈ { COL , ROW } ) 1: W low ←H low ( W ) 2: p ∗ 1 ← SEG_ROW_SEARCH ( W low ) 3: B low ← BINARY ( W ( b ) | w i,j |≤ p ∗ 1 , low ) + BINARY ( W ( b ) | w i,j | >p ∗ 1 , low ) 4: W diff ← W -H - 1 ( B low , 0 ) 5: W high ←H high ( W diff ) 6: p ∗ 2 ← SEG_ROW_SEARCH ( W high ) 7: B high ← BINARY ( W ( b ) | w i,j |≤ p ∗ 1 , high ) + BINARY ( W ( b ) | w i,j | >p ∗ 1 , high ) 8: B ← B low + B high 9: return B

func BINARY ( W )

1:

µ

1

m

∑

m

j

=1

W

.j

// row-wise mean

2: ˜ W ← W - µ // centered matrix

3: α = √ ∥ ˜ W ∥ 2 2 m // row-wise scale

4: B ← α · sign ( ˜ W ) + µ

5: return B

func SEG_ROW_SEARCH ( W )

1: n ← number of rows in W

2: e ← + ∞× 1 n × 1 // row-wise error

3: p ∗ ← 0 n × 1 // optimal break-point of each row

4: for τ = 0 . 1 , 0 . 2 , . . . , 0 . 9 do

5: p ← τ × max( abs ( W ) . ( dim = 1))

6:

B

1

BINARY

(

W

|

w

i

∈

[

n

]

,

:

|

p

)

7:

B

2

BINARY

(

W

|

w

i

∈

[

n

]

,

:

|

>

p

)

8:

if

∥

W

-

(

B

1

+

B

2

)

∥

2

< e

then

9:

e

←∥

W

-

(

B

1

+

B

2

)

∥

2

10:

p

∗

p

11: end if

12: end for

13: return p ∗

F Additional Experimental Results

F.1 Experimental Results for DeepSeek-R1-Distill-Llama-8B

Table F.1 provides further experimental results on DeepSeek-R1-Distill-Llama-8B model. In line with the trends observed in Table 1, HBLLM consistently outperforms existing 1 -2 bit quantization techniques across all these evaluation metrics. Remarkably, even when applied to the more complex and modern DeepSeek-R1Distill-Llama-8B architecture, HBLLM preserves the same performance advantages previously demonstrated on LLaMA3-8B. This assessment highlights the effectiveness of HBLLM for LLMs.

F.2 Comparison on 9 zero-shot QA datasets

In the main paper, we report the average accuracy ( AvgQA ) across 9 zero-shot QA datasets to provide a high-level comparison of different quantization methods. In this appendix, we present the detailed accuracy on each individual dataset. As shown in Tables F.2-F.5 our method consistently delivers strong performance across all datasets, further validating its effectiveness and robustness in diverse zero-shot question answering tasks.

F.3 Comparison of Avg. Relative Perplexity and Avg. Relative QA Accuracy across Models

To provide a unified view of model performance across both language modeling and common sense reasoning tasks, we compute two metrics: Avg. Relative Perplexity and Avg. Relative QA . Avg. Relative Perplexity is calculated as the average over three language modeling datasets: C4, Wiki2 and PTB. Avg. Relative QA is computed as the average across 9 zero-shot QA datasets.

Relative Perplexity is defined as:

where S Perplexity and S FP Perplexity are the perplexities of the models after and before quantization, respectively.

Relative QA is defined as:

where S QA and S FP QA are the QA scores of the models after and before quantization, respectively.

Figure F.1 presents the comparison of these two metrics across the LLaMA-1, LLaMA-2, LLaMA-3, and OPT family models. HBLLM consistently shows lower relative perplexity and higher relative QA accuracy compared to prior 1 -2 bit quantization methods (PB-LLM, BiLLM, and ARB-LLMRC), demonstrating both effectiveness and scalability across diverse LLMs.

Avg.RelativePerplexity

101

101

101

101

10°

￥

10°

10°

100

7B

13B

30B

65B

7B

13B

70B

88

70B

1.3B

2.7B

6.7B

13B

30B

1.0

1.0

1.0

1.0

0.9

0.9

0.9

0.8

0.8

0.8 -

0.8

0.7

0.7

0.6

0.6 -

0.7

0.6

0.5

0.5

0.6

7B

13B

30B

65B

7B

13B

70B

8B

70B

1.3B

2.7B

6.7B

13B

LLaMA-1

LLaMA-2

LLaMA-3

OPT

*1

FP16

I1

PB-LLM

1

BiLLM

T

ARB-LLMRC

HBLLM-row(ours)

—HBLLM-col(ours)

(a) Study of salient column selection criterion

(b) Study of grouping granularity

（%)

Median:

Median:

(%)

Median:

Median:

Relative Quantization Error

35.53

●34.26

Error

●40.26

●37.10

OS'SE·

●34.23

36.82

34.92

●34.71

●33.57

●40.72

38.48

38

●34.59

36

● 33.54

Quantization

50

●35.59

●34.46

34

Relative

J0

OPT-1.3B

LLaMA2-7B

OPT-1.3B

LLaMA2-7B

HBLLM-col - f1

HBLLM-col - 2

HBLLM-roW - 1

HBLLM-row - f2

HBLLM-col: global

HBLLM-col: row-wise

-

HBLLM-row: row-wise

Error(%)

Median:

Median:

Relative Quantization Error (%)

Median:

Median:

36.82

34.92

36.62

●35.32

●35.50

34.23

36.01

●34.81

●35.59

●34.46

35.59

●34.46

40

34.59

36

●33.54

●35.47

34.36

36

34

85

32

OPT-1.3B

LLaMA2-7B

OPT-1.3B

LLaMA2-7B

HBLLM-col

HBLLM-col: shared mean

HBLLM-row

HBLLM-row: shared mean

num=10

num=40

num=80

(c) Effectiveness of shared mean

(d) Study of partitioning candidates number

F.4 Ablation Study from the Perspective of ℓ 2 Relative Error

To complement the ablation studies presented in the main paper, we provide an additional analysis from the perspective of relative quantization error in terms of the Frobenius norm. Specifically, we compute the relative ℓ 2 error for each quantized weight block, defined as the Frobenius norm of the quantization residual normalized by the original weight norm. We then examine the distribution of these errors across different layers and quantization methods to assess fidelity at the matrix level.

The relative error of a quantized weight matrix is defined as:

where W FP and W B denote the full-precision and quantized weight matrices, respectively.

Figure F.2 visualizes the sorted relative quantization errors under different ablation settings, including (a) salient column selection criterion, (b) grouping granularity, (c) shared mean strategy, and (d) number of partitioning candidates. Across all cases, our proposed HBLLM consistently achieves lower median relative error compared to prior approaches, further confirming the effectiveness of each component.

In Figure F.2a, we observe that using the ℓ 2 norm as the column selection criterion leads to significantly lower quantization errors than ℓ 1 , validating its stronger ability to capture the energy distribution of weights. In Figure F.2b, row-wise grouping substantially outperforms global grouping, indicating the importance of fine-grained adaptivity. In Figure F.2c, the shared mean strategy slightly reduces quantization error while offering better compression efficiency. Finally, in Figure F.2d, increasing the number of partition candidates reduces error up to a point, with 40 candidates offering the best trade-off.

These observations align well with perplexity-based trends, and jointly confirm the robustness and precision of HBLLM under various quantization design choices.

G Inference Latency Estimation and Efficiency Analysis

To evaluate the computational efficiency of HBLLM in practical inference scenarios, we design an estimationbased experiment to analyze its inference latency. This is necessary because no current inference framework fully supports the dequantization algorithm used in HBLLM, and constructing such a backend involves significant engineering effort beyond the current scope.

Our findings suggest that the inference latency of HBLLM is approximately 31 . 8% of the FP16 baseline, indicating strong computational benefits despite the algorithm's structural complexity.

Latency Estimation Methodology. We define the relative inference time of HBLLM compared to FP16 as:

where p is the portion of time spent on matrix-vector multiplication (GEMV) during inference, and l is the acceleration factor of GEMV after quantization. Following prior works such as GPTQ [ ? ], we use p = 0 . 78 .

To compute l , we measure:

where T torch denotes the runtime of FP16 GEMV using PyTorch, and T hqmv refers to our quantized matrix-vector multiplication kernel (HQMV).

Since the current hqmv implementation does not support Intra-Frequency Grouping (IFG), we use the runtime of HBLLM-col without IFG (denoted HBLLM-col w/o IFG) as a proxy to estimate the runtime of HBLLM-col with full IFG. Notably, applying IFG doubles the number of groups, potentially leading to: roughly 2 × more CUDA warp divergence, a 90% increase in average data loading (Avgbit increases to 2 . 88 bit).

However, since data loading and computation can overlap on GPU, we conservatively estimate that the runtime with IFG should not exceed twice the runtime without IFG.

Setup: We benchmark a single linear layer of OPT-175B under the same GEMV input/output shape used in GPTQ [12]. All timing experiments are conducted on an NVIDIA P100 GPU.

-Measured: T torch = 1 . 35 × 10 -3 s

-Measured: T hqmv for HBLLM-col w/o IFG = 8 . 54 × 10 -5 s

-Estimated: T hqmv for full HBLLM-col (with IFG) ≈ 1 . 70 × 10 -4 s

This yields:

Discussion: This result indicates that HBLLM inference, despite involving additional processing (e.g., dequantization and grouping), maintains high efficiency due to:

-Lightweight binary matrix operations;

-Reduced memory bandwidth consumption;

-Effective overlap of memory access and computation in GPU execution.

Overall, HBLLM achieves strong acceleration over FP16 without relying on highly specialized hardware or handcrafted fusion kernels, making it a viable candidate for practical deployment in low-bit LLM inference systems.

H Licenses for Existing Assets

[32], llama, llama2, llama3

https://huggingface.co/TheBloke, llama, llama2, llama3

[37], https://github.com/facebookresearch/metaseq/tree/main/projects/OPT, OPT-175B LICENSE AGREEMENT

[15], https://github.com/Aaronhuang-778/BiLLM, MIT License

[18], https://github.com/ZHITENGLI/ARB-LLM, Apache License 2.0

[1], https://github.com/vsingh-group/FrameQuant

[29], https://github.com/scotfree/PbLLM, Creative Commons Zero v1.0 Universal

[4], https://leaderboard.allenai.org/physicaliqa/submissions/get-started

[7], https://github.com/google-research-datasets/boolean-questions

[8], https://huggingface.co/datasets/allenai/ai2_arc, Creative Commons Attribution Share Alike 4.0

[21], https://huggingface.co/datasets/ptb-text-only/ptb_text_only, Dataset provided for research purposes only

[22], https://huggingface.co/datasets/mindchain/wikitext2, Creative Commons Attribution-ShareAlike License.

[23], https://github.com/allenai/OpenBookQA, Apache License 2.0

[24], https://github.com/EleutherAI/lm-evaluation-harness, MIT License

[25], https://zenodo.org/records/2630551, Creative Commons Attribution 4.0 International

[26], https://github.com/google-research/text-to-text-transfer-transformer#datasets, Apache License 2.0

[27], https://asgordon.github.io/copa.html, BSD 2-Clause License

[28], https://github.com/allenai/winogrande, Apache License 2.0

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS Paper Checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction clearly state our main contributions, which are supported by the theoretical and empirical results presented in the Section 3 and Section 4.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Please refer to Section 5.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Please refer to Appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Please refer to Section 4.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code will be made publicly available upon acceptance of this paper.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Please refer to Section 4.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Error bars are not reported because it would be too computationally expensive. Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Quantization for models <30B was run on 4 × RTX 3090 (24GB), and for models ≥ 30B on A800-80GB. See Section 4.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We carefully read and follow the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [No]

Justification: The paper is purely fundamental research and does not involve social impact. Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [No]

Justification: This paper does not release new models or datasets.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Please refer to Appendix.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: : The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: LLMs were not involved as core components in the development of the method or results in this paper.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes HBLLM, a 1-bit post-training quantization method for large language models that uses Haar wavelet transforms to improve expressiveness and accuracy. It introduces two structure-aware grouping strategies: one based on frequency and the other on saliency (L2-norm). For less important weights, it shares group means to save space. Experiments on OPT and LLaMA models show its good results, with only 1.08 bits per weight and strong performance.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**\n\n1. The primary contribution is that this method is the first to integrate \"localized\" orthogonal transformations (i.e., Haar wavelets) into a BiLLM-style quantization process. With other 3 auxillary strategies, Frequency-aware multi-parameter intra-row grouping/L2-norm-based saliency-driven column selection/Intra-frequency-band mean sharing, is achieves comprehensive good results.\n\n2. Experiments on comprehensive models, baselines, and evaluation benchmarks demonstrate that the proposed approach is effective.\n\n**Weakness**\n\n1. The main difference and contribution of this paper compared to BiLLM lies in the introduction of local orthogonal transforms, corresponding to lines 7–10 of the HBLLM algorithm. However, the actual impact of these local transforms on performance (e.g., PPL and accuracy) is not clearly demonstrated. It would be helpful to show in ablation studies what happens when this component is removed or replaced by a global orthogonal transform.\n\n2. The authors mention in the introduction that global orthogonal transforms are computationally expensive, thus motivating the use of local transforms. However, only the overall runtime is reported. A more detailed analysis comparing the computational cost of local vs. global transforms would provide useful insight.\n\n3. There is no evaluation or discussion of computational efficiency, such as matrix multiplication speed or end-to-end inference latency. This analysis would be important for understanding the practical benefits of the proposed method.\n\nQUESTIONS:\nSee weakness.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes the HBLLM approach to enhance 1-bit post-training quantization for LLMs. HBLLM uses Haar wavelet transforms to decompose weights by frequency, enabling more effective quantization. The proposed method introduces structure-aware grouping and saliency-based selection to improve accuracy and storage efficiency. Results on the OPT and LLaMA models show that HBLLM achieves state-of-the-art performance for 1-bit quantization.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- Good empirical improvements.\n- The paper is easy to follow.\n- The paper provides extensive experiments across multiple LLMs and tasks.\n\nWeaknesses:\n\n- The paper detects the saliency parameters in an element-wise manner, which could result in more overhead compared to a channel-wise approach. Did the authors consider this?\n- The motivation for using the Haar Transform to capture the structural differences of weights is not clear.\n- The paper does not number the equations.\n\nQUESTIONS:\n1. What is the difference between the proposed saliency-driven selection via L2 norm and AWQ [1] ?\n\n2. The objective function in the paper does not consider calibration data. However, the authors use calibration data to compute the parameter importance metric. What would happen if the reconstruction loss also considered the calibration data?\n\n3. As far as I know, the frequency domain is often used to capture the characteristics of data in the feature space rather than the weight space. Please correct me if I am wrong. Why does transforming weights to the frequency domain using the Haar Transform help capture the structural differences of weights?\n\n4. In Table 1, could you provide the equation used to compute the average bit-width in detail? Since element-wise saliency detection creates overhead, why is the bit-width using HBLLM-col 1.00?\n\nRef: Lin, Ji, et al. \"AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration\". CoRR abs/2306.00978 (2023). 2023.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
The submission proposes a new post-training LLM quantization technique based on frequency-domain lossy weight compression. It shares the main idea with other lossy frequency-domain compression schemes that were applied successfully to media data types in the past. Instead of exploiting different perception sensitivity levels of humans depending on frequency bands (works for media types), this method uses the Haar-transform, a saliency selection method of matrix columns and a new metric called CIQ that is used for the optimization.

STRENGTHS AND WEAKNESSES:
The strength of the submitted work is the idea. It seems to deliver consistently better performance at 1 bit-per-weight or close to that compared to >2 bits-per-weight of the next best performing technique which is FrameQuant.

Minor weaknesses are missing experimental results showing inference speed (latency / throughput) and a few details that are left unclear (see Questions).

QUESTIONS:
* Please add measurements showing inference speed (latency / throughput) especially as the technique requires a sequence of more complex math operations.

* Line 177: is K a single global parameter? Would multiple K work as well and if yes, how (maybe based on the energy of the Haar-spectrum)?

* Table 3 (FrameQuant), close to line 291: is it possible to get at least an estimate for the total amount of memory FrameQuant would have allocated?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 4
ORIGINALITY: 4
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes HBLLM, a 1 bit PTQ method for LLM. The key idea is to apply a Haar wavelet transform to decompose weight matrices into frequency components, enabling structure-aware quantization. The method further introduces two grouping strategies: frequency-aware intra-row grouping and saliency-driven column selection, alongside a shared-mean quantization scheme for non-salient components. Experiments show improved perplexity and QA performance compared to prior 1-bit quantization methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n(1) Introduces the Haar wavelet transforms as a localized orthogonal transformation to increase binary expressiveness.\n(2) Introduces two innovative grouping strategies, frequency-aware multi-parameter intra-row grouping and l2-norm saliency-driven column selection, and also employs a shared mean for non-salient weights, to better preserve important weight structures.\n\nWeaknesses\n\n(1) While the Haar transform is well-motivated theoretically, its contribution is not isolated. The authors do not compare their method with a baseline where grouping and saliency selection are applied directly in the original weight space without the Haar transform. This leaves unclear whether the Haar decomposition meaningfully improves quantization results beyond what grouping and saliency alone achieve.\n\n(2) Fig.2 is not clear enough, maybe adding more description in the caption would help understanding. Also, if could add a figure to illustrate the weight decomposetion (before and after) would greatly enhance the reader's understanding of the core idea.\n\nQUESTIONS:\n(1) I encourage the authors to disentangle the contribution of Haar-based frequency decomposition from other components (e.g., grouping, saliency selection) to provide a clearer understanding of where the benefits originate.\n\nLIMITATIONS:\nThe authors did not discuss the limitations of the work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}