CBQ: Cross-Block Quantization for Large Language Models
Authors: Xin Ding
Xiaoyu Liu
Zhijun Tu
Yun Zhang
Wei Li
Jie Hu
Hanting Chen
Yehui Tang
Zhiwei Xiong
Baoqun Yin
Yunhe Wang
Venue: ICLR 2025 Spotlight (2025)
Keywords: Large Language Model Compression
ultra-low bits precision
Primary area: other topics in machine learning (i.e., none of the above)

Abstract: Post-training quantization (PTQ) has played a pivotal role in compressing large language models (LLMs) at ultra-low costs. Although current PTQ methods have achieved promising results by addressing outliers and employing layer- or block-wise loss optimization techniques, they still suffer from significant performance degradation at ultra-low bits precision. To dissect this issue, we conducted an in-depth analysis of quantization errors specific to LLMs and surprisingly discovered that, unlike traditional sources of quantization errors, the growing number of model parameters, combined with the reduction in quantization bits, intensifies inter-layer and intra-layer dependencies, which severely impact quantization accuracy. This finding highlights a critical challenge in quantizing LLMs. To address this, we propose CBQ, a cross-block reconstruction-based PTQ method for LLMs. CBQ leverages a cross-block dependency to establish long-range dependencies across multiple blocks and integrates an adaptive LoRA-Rounding technique to manage intra-layer dependencies. To further enhance performance, CBQ incorporates a coarse-to-fine pre-processing mechanism for processing weights and activations. Extensive experiments show that CBQ achieves superior low-bit quantization (W4A4, W4A8, W2A16) and outperforms existing state-of-the-art methods across various LLMs and datasets. Notably, CBQ only takes 4.3 hours to quantize a weight-only quantization of a 4-bit LLAMA1-65B model, achieving a commendable trade off between performance and efficiency.

Published as a conference paper at ICLR 2025
CBQ: CROSS-BLOCK QUANTIZATION FOR LARGE LAN- GUAGE MODELS
Xin Ding Xiaoyu Liu
Zhijun Tu
Yun Zhang'
Wei Li
Jie Hu²
Hanting Chen² Yehui Tang² Zhiwei Xiong' Baoqun Yin
Yunhe Wang
'Univeriity of Science and Technology of China Huawei Noah's Ark Lab Hong Kong University of Science and Technology (GZ)
ABSTRACT
Post-training quantization (PTQ) has played a pivotal role in compressing large language models (LLMs) at ultra-low costs. Although current PTQ methods have achieved promising results by addressing outliers and employing layer- or block- wise loss optimization techniques, they still suffer from significant performance degradation at ultra-low bits precision. To dissect this issue, we conducted an in- depth analysis of quantization errors specific to LLMs and surprisingly discovered that, unlike traditional sources of quantization errors, the growing number of model parameters, combined with the reduction in quantization bits, intensifies inter-layer and intra-layer dependencies, which severely impact quantization accuracy. This finding highlights a critical challenge in quantizing LLMs. To address this, we propose CBQ, a cross-block reconstruction-bent based PTQ method for LLMs. CBQ leverages a cross-bles bbockk dependency to establish long-range dependencies across multiple blocks and integrates an adaptive LoRA-Roundee Rounding technique to manage intra-layer dependencies. To further enhance performance, CBQ incorporates a coarse-to-fine pre-processing mechanism for processing weights and activations. Extensive experiments show that CBQ achieves superior low-bit quantization (W4A4, W4A8, W2A16) and outperforms existing ofte--of-thearr methods across various LLMs and datasets. Notably, CBQ only takes 4.3 hours to quantize a weight- only quantization of a 4-bit LLAMA1-65B model, achieving commendable trade off between performance and efficiency.
INTRODUCTION
Large language models (LLMs) (Wei et al. (2022a); Radford et al.; Zhang et al.; Brown et al. (2020b); Dettmers et al. (2022)), have sparked immense academic and industrial interest owing to their remarkable performance in handling complex natural languages tasks (Hendrycks et al. (2020b); Bisk et al. (2020b); He et al. (2017); Ainslic et al. (2023); Liu et al. (2024b)). During to significant computational resources for inference and deployment, the post- te-trrriiing quantization (PTQ) technique (Choi et al. (2018); Frantar et al. (2022a); Nagel et al. (2019); Wei et al. (2023); Li al. (2025)) operating with limited calibration data and computational resources is more in demand for compressing LLMs.
Existing PTQ methods typically optimize models on layer block basis, addressing outliers (Wei et al. (2022b; 2023); Chee et al. (2024); Liu et al. (2024a)) and employing first- or second-order optimization techniques (predominantly optimizing models on a layer-by-layer or block-by-bloce basis) (Shao et al. (2023); Frantar et al. (2022b); Liu et al. (2023a)). However, these approaches often suffer from significant performance degradation, particularly in low-bit settings such as W2A16 and W4A4, as illustrated in Table 1, due to inherent limitations. Previous work, like AdaRound (Nagel et al. (2020)), analyzed rounding errors and showed that simple rounding is not always the optimal quantization strategy, greatly improving quantization for CNNs. This inspired us to analyze quantiza- tion loss for LLMs, comparing high-bit and low-bit scenarios. We found that in low-bit quantization,
'Equal Contribution
This work was done during Xin Ding and Xiaoyu Liu's internship at Huawei Noah's Ark Lab 1 *Corrsponndigg author:bqyin eust..ddu.en, yunhe. huawei.com

---

Published as a conference paper at ICLR 2025
intra-layer and inter-layer dependencies within models become pronounced, especially model size increases. This indicates that previous methods, whether focused on optimizing quantization parameters within a layer or block through first- or second-order techniques, or refining rounding errors, fall short of achieving optimal outcomes. Instead, is essential to fully account for the inter-layer and intra-layer relationships.
To address this, we propose CBQ, a cross-block reconstruction-bat based PTQ method tailored for LLMs, surpassing traditional layer-wise and block-wise reconstruction techniques. CBQ introduces cross- block dependency (CBD) into block-wise reconstruction, maintaining the integrity of the model's internal dependencies during quantization. Our approach optimizes multiple transformer blocks within a sliding window with overlapping. allowing for more effective and non-local optimization of quantization parameters. Using the CBD method, CBQ incorporates a LoRA- Aooonding technique, employing two low-rank matrices to learn adaptive compensation values for quantized weights. Notably, we jointly optimize the compensation matrices and the step sizes of weights and activations within the overlapping window, which helps manage intra-layer dependencies to rectify weight quantization errors while preserving training efficiency. Furthermore, CBQ introduces novel unified coarse-to to-finett pre-processing (CFP) strategy from statistical perspective to evaluate outliers in weights and activations, precisely handling outliers while minimizing damage to normal channels. CFP employs quartile criterion initially estimate the range of outliers and then assesses the intra-class and inter-class distances between outliers and normal values to precisely identify their locations. This approach facilitates the truncation of weight outliess and the application of equivalent scaling to activation outliers.
The contributions of this paper are summarized follows:
We performed a comprehensive analysis of the error sources in low-bit quantization scenarios for LLMs, and theoretically demonstrated the significant impact of intra-layer and inter-layer dependencies on the effectiveness of model quantization.
We propose CBQ, a unified PTQ method designed for LLMs, incorporating a cross-block reconstruction strategy that introduces a Cross-Block Dependency (CBD) mechanism to preserve the model's internal dependencies during quantization, and LoRA-Rounding to utilize intra-layer dependencies for optimizing adaptive compensation matrices.
We design a coarse-to- pre- rrepprcessiin strategy (CFP) that can simultaneously detect and manage outliers in both weights and activations, effectively preventing disruption normal activation channels and weights.
Extensive experiments demonstrate the effectiveness of our method in ultra-low bit quanti- zation settings such as W4A4, W4A8, and W2A16. 16. Notably, it outperforms state-of-the-arr methods across diverse models and benchmark datasets.
MOTIVATION
To analyze the sources of quantization errors in large models when quantizing weights or activations, we assume a matrix M representing a set of weights or activations as the current quantization target, and L denotes the quantization loss of the model under this matrix. Let e denote a small perturbation introduced by quantization and L(M) represent the task loss that we aim to minimize. Then,we can derive the following equation within the Taylor expansion:
EL + [(M) ~ B
5m + 1 1. JCC
OC
0M2 O(1I=11)) = = (M) ) + 2 H (M) (1)
1
2
As discussed in previous work (Frantar et al. (2022b)), The error introduced by quantization is sufficiently small, the higher-order terms in the Taylor expansion can be neglected. Therefore, we analyze the first- and second-order terms, (M) and H (M) which can be defined as follows.
K
ac
M) EIV M [(M)
(2)
Đm,
K 82C
( - -EV Σ
Σ
(3)
aM,a
2

---

Published as a conference paper at ICLR 2025
Wile
(a)
0.1
Lagert Soaee 2nt
4bK
Lagh
0.
Layer
(b)
(c)
Figure 1: (a) Visualization of the absolute values of the Hessian matrix for weights within a single layer of LLAMA-7B, (b) Hessian matrix visualization of the loss with respect to the scale across 32 layers of LLAMA-7B, and (c) the relationship between the average scale the first two transformer blocks in LLAMA-7B and the corresponding loss.
Let K denote the number of elements in the LLM involved in the quantization. Using Equation and 3, the influence of any two elements i and j the final quantization loss be calculated. From equations 1, 2, 3, can be observed that when the quantization perturbation E small, ||=||2 is also small, allowing to disregard the implications of the equation 3. In this case, the quantization error is primarily related to the current quantization target M, analogous to high-bit quantization. However, when performing low-bit quantization, ||e|| increases, necessitating consideration of the impact described by Equation 3. This indicates that when i * j. relationships between different M are introduced. This relationship manifests in two aspects: when quantizing a single layer, reflects intra-layer dependencies among parameters, and when quantizing the entire model, inter-layer dependencies must also be considered. Furthermore, given that the complexity of the Hessian matrix H is proportional to On+²), where represents the number of parameters, the growth in model size, both in terms parameters and layers, leads to a marked intensification of intra-layer and inter-layer dependencies.
To better illustrate intra-layer and inter-layer dependencies, we visualize Equation for both indi- vidual layers and the entire model using LLAMA-7B. Additionally, present visualizations of the dependencies between adjacent blocks, as referenced in Figure
By analyzing Figure 1, observe notable increase in the values of off-diagonal elements during lower-bie bit quantization. This increase indicates strengthening both inter-layer and intra-layer dependencies, with closer elements exhibiting stronger correlations. Furthermore, comparisons of the scales between adjacent layers provide a clearer understanding of the substantial impact that inter-layer dependencies have final quantization outcomes in low-bit scenarios.
Therefore, taking into account both intra-layer and inter-layer dependencies, we present the quan- tization framework for LLMs under low-bit settings, which be expressed by the following equation:
arg min Σ (OTTW,,**),T(CCCC QT&((()) + Aiv. Q(x*)) H. keh
(4)
where T and QT represent the floating-por ppint and quantized transformer blocks, respectively. Q.(-) represents the quantization process. E(-) represents the metric evaluate the reconstruction errors be-
tween outputs of quantized block and full full-precision block. We jointly optimize all transformer blocks with inter-layer dependencies while compensating for intra-layer relationships using { ||: H.).
3

---

Published as a conference paper at ICLR 2025
Full-precto preci-oeciioon model
Truncate weight
Scale activatoon
Cross-ble dependency
Conreto-Fiiee Prepr (CFP)
min(mna)
SW (0.1)
11 ex
binstiedd
LoRA-Rouxdingglee Weight Quantization
Quantized model with ( ..W.
.i. 14 _42 "istsskkjjk overlape1
Figure 2: Workflow of the proposed CBQ. CBQ firstly utilizes coarse-to-fine preprocessing to handle the outliers of weights and activations, and then employs cross-block optimization strategy learn quantization step sizes and weight adaptive rounding matrices with supervision from the corresponding full- full-precision model. This sequential block-wise method minimizes aggregate error propagation through cross-block dependency modeling.
METHOD
In this section, we introduce the proposed cross-block quantization framework tailored to LLMs. As illustrated in Fig. 2, CBQ firstly handles the outliers of weights and activations, and then jointly learns step sizes of weights and activations and weight-compensatiin manner. CBQ reconstructs the output feature of the last block each sliding window based on the corresponding supervision of the full-precision model.
matrices in cross-block
3.1 CROSS-BLOCK RECONSTRUCTION
To maintain inter-layer dependencies, is necessary to optimize the layers with significant depen- dencies together. As shown in Figure 1, the strongest dependencies are typically observed between adjacent layers. Therefore, we introduce a cross-block dependency (CBD) scheme using a sliding window approach. This scheme enables the simultaneous optimization of multiple blocks within the window. Furthermore, the two adjacent sliding windows have overlapping blocks, ensuring that the blocks between the windows also interconnected. The CBD scheme enhances the connectivity and cooperation between blocks, enabling them to jointly contribute to the quantization process. This holistic optimization strategy leads to better overall performance and addresses the limitations of block-wise reconstruction in capturing cross-blocd dependencies. We formulate the optimization with the CBD scheme
arg min SX W AW B (Wikk
(5)
where sisk K, Ti, represents the blocks from block to block & within one sliding window, and the same applies to the symbols S and i.k The optimization object Zyee is as follow:
Cree = (W X(i)), (,
For the distance metric, we incorporate L2 and Kullback-Leinn Leibler divergence (KLD) loss (Kullback & Leibler (1951)) to measure reconstruction error. KLD computes the likelihood distribution between output features that undergo the softmax function. It tends to suppress outliers in the feature space and enhance the robustness the optimization process. By incorporating both terms, our method captures both the spatial distance and the distribution discrepancy, leading to more comprehensive and robust optimization process. Then the distance metric is formulated as:
(h,,, h2) = |4 h22 + DxL. ((v)))
where h and fy are hidden states from the outputs of full-precision blocks and quantized blocks, respectively. a is the sofumax function. ||-| represents the C, distance and D Dx (-) represents the KL.D distance. We provide the ablation study on the loss functions in Appendix B. B.Table 5.
4

---

Published as a conference paper at ICLR 2025
3.2 LORA-ROUNDING FOR WEIGHT QUANTIZATION
AdaRound (Nagel et al. (2020)) introduces to learn a better weight-rounding matrix for post-training quantization that adapts the data and the task loss. As shown in Eq. 8, we can obtain the weight- rounding matrix Aw E Rdxk with a learnable matrix V E IRdxk with a rectified sigmoid function:
Aw Clip Sigmoid (S +7,0,111
where { and stretch parameters and fixed 1.1 and -0.1, and Clip(-) clamps the inputs into a given range. The size the weight-rounding matrix Áw is the same as the original weights.
When the transformer blocks are within the overlap of the CBD sliding window mechanism, the rounding matrix can serve as an effective representation of intra-layer dependencies. We utilize compensation matrix and jointly optimize it with the quantization step sizes for weights and activations, which can be expressed as follows:
arg min ( xix), Tuu(Q(()) ((xix)) S S A'
(9)
s.t. - +1 overlap
(10)
However, as shown in Experiment Table 3b, we found that LLMs with billion-level parameters result in an exceptionally large W. which can lead to significant computational overhead and substantially impact the convergence of training. (Shao applied to models with billions of parameters due to the vast solution space, which aligns with our experimental findings. Thus, we employ low-rank adaptive learning on the compensation matrices, decomposing V with much smaller low-rank matrices, and only optimize them in post-training quantization, the decomposition defined as:
(2023)) has also mentioned that AdaRound cannot be
Aw = A x A2, A E Rddr A2 E R
(11)
Where the rank << min(d, k). we utilize a random Gaussian initialization for A and zero for Ag, thus Aw is set to zero at the beginning of post-trann traintran quantization. During training, each element of Aw encouraged into with regularizer loss:
Leom 1-112A
1
(12)
Where 3 is annealing factor. Following (Nagel al. (2020)), is set to higher in the initial phase and set to lower in the later phase of the optimization to encourage it to converge to 0 or 1. We also conduct Aw |Aw] in the later phase of the optimization force each element into (0. 1} exactly.
Compared with vanilla AdaRound for LLMs. The proposed MaRA-Rounding reduces the number of learnable parameters from d x & to (d + k) x r and changes the training strategy, significantly accelerating the optimization process, we conduct ablation experiments in the next section 5.3.
3.3 OVERALL LOSS
In summary, by leveraging CBD and low-rank decomposition of the We slide the window to the last block with an interval and update all the quantization parameters Sw, Sx AI, A2 within the window, ensuring the preservation of both intra-layer and inter-layer relationships of the model, thereby achieving optimal performance. The total loss for optimizing the 7th block to the th block within a sliding window is formulated as
matrix,
Listal Lree + Yooms
(13)
where the is the
to balance the reconstruction error and compensation error.
3.4 COARSE-TO-FINE PRE-PROCESSING
Outlier handling is crucial in quantizing LLMs. Figure in Appendix F illustrates the prevalent outliers in weights and activations, which pose significant challenges to the quantization process. Although there are many existing studies based on outliers problem, these studies typically focus on outliers in either weights activations individually, such in (Chee al. (2024); Wei et al. (2022b);
5

---

Published as a conference paper at ICLR 2025
Xiao et al. (2022)). However, there is no precise strategy that simultaneously detect and handle outliers in both weights and activations. This single- mode approach can potentially damage normal activation channels and weights due to incorrect outlier detection. To address this issue, discard the previous assumption of normal distributions for weights and activations (Wu et al. (2023)), and based on statistical principles (Massart al. (2005)), propose coarse-to-fine pre-processing strategy to decouple the outlier handling in activations and weights. Relevant theoretical details can be found in Appendix
The comprehensive algorithm of the outlier detection is illustrated in Algorithm 1 in Appendix K, which is divided into two stages.
Coarse-grained detection. In the first stage, we perform coarse-grained detection by calculating the lower and upper quartile values (Q) and Q3) and the interquartile range (!QR) (Massart et al. (2005)) in the numerical distribution (either activations or weights). Based on these calculations, we obtain a coarse outlier set O = {i x > T, x E X). where = Qu + XIIQR and x is set to 1.5. This stage greatly reduces the search space for outlier detection.
Fine-graines detection. In the second stage, we perform fine-grained detection by searching for threshold that splits the coarse outlier set into an outlier subset Ouutleer and reserved subset Oreserved. The goal is to minimize the intra-set variance M = Var (eesrveed) while maximizing the distance between the two subsets Manter = (Min(Ooutlien) Oontlier) - Max Max(Orresca (reserwed)) To balance these objectives, we define a metric M = Minter Mintra where X2 = 1.0. By minimizing this metric, we can effectively identify outliers and distinguish them from the remaining data.
Removing outliers in weights has minimal impact on performance, whereas outliers in activations, particularly in specific channels, can greatly affect performance if directly removed. Consequently, our approach involves truncating weight outliers and scaling outliers in activations based on the detected outliers in both weights and activations. Figure 3 in Appendix F provides visual evidence of weight outliers being truncated within the outlier group.
The scaling factor 8; for the activation tensor in 7th channel (represented as X2) is determined by the maximum absolute value of the truncated outlier set O*:
8; MMax(IX:)/MMax(())
(14)
This scaling factor is then applied to update weights and activations following prior work (Wei et al. (2023)) to counteract destabilizing fluctuations from remaining outliers.
RELATED WORK
Post-training quantization. The post-training quantization (PTQ) algorithm (Nagel al. (2021) I); Wu et al. (2020; 2023); Zhang et al. (2018)) converts the pre-trained full- full-precision network into fixed-point network with a few unlabeled calibration data and computational overhead, which enables fast deployment on various devices. Recent post-training quantization methods have been widely explored in vision models (Liu et al. (2021); Hubara et al. (2021); Frantar & Alistarh (2022); Cai et al. (2020); Li et al. (2022)). Some techniques like AdaQuant (Hubara et al. (2020)), AdaRound (Nagel et al. (2020)), and BRECQ (Li et al. (2021)) minimize the distance between floating point and quantized model outputs to optimize quantization parameters. While BRECQ incorporates Fisher information and jointly optimizes layers within each residual block, it still obtains sub-optimal performance for not capturing interactions across neighbouring residual blocks. The proposed CBQ improves quantization accuracy that accounts for dependencies between adjacent blocks.
Quantization for large language models. Existing large language models such as BLOOM (Laurençon et al. (2022)), OPT (Zhang et al. (2022)), and LLAMA (Touvron et al.; 2023)) contain tens of billions of parameters, and require massive memory footprint and computa- tion requirements in the inference (Ashkboos et al. (2023); Wang et al. (2010); Bolya & Hoffman (2023); Brown et al. (2020a); Jacob et al. (2018)). Recent works have been proposed to compress LLMs with post-training quantization methods that do not require complete training procedure and access to a full training dataset. LLM.int8) (Dettmers et al.), ZeroQuant (Yao et al. (2022)) and nuQmm (Park al. (2022)) focus on quantizing the parameters with mixed-precision decomposition

---

Published as a conference paper at ICLR 2025
scheme, representing the outliers with 16-bit and others with 8-bit. These methods can not truly accelerate the inference of LLMs for that is hard to implement on hardware. Other methods like GPTQ (Frantar et al. (2022b)) and AWQ (Lin et al. (2023)) can efficiently quantize LLMs but they focus on FP16 activations and INT4 weights, which can not benefit from the integer matrix multiplica- tion of existing Al accelerators. Additionally, Some methods like SmoothQuant (Xiao al. (2022)), Outlier Suppression (Wei et al. (2022b)), Outlier Suppression+ (Wei et al. (2023)) and QLLM (Liu al. (2023a)) aim at processing activation outliers (Zhao al. (2019)) and lack optimization for the weight quantization. Moreover, these methods rely on hand-craft quantization strategies which are tuned based on extensive experimentation for optimization. Recent block reconstruction-basee PTQ method OmniQuant (Shao et al. (023)),QLLM (Liu et al. (2023a)), have experienced significant accuracy degradation in low-bit settings. In contrast, CBQ introduces a more precise outlier detection strategy and optimizes the reconstruction process through CBD and LoRA-Rounding mechanisms by maintaining both intra-layer and inter-layer dependencies.
5
EXPERIMENTS
5.1 SETUP
Models and datasets. We conduct experiments on large language models with different sizes, including OPT (Zhang et al. (2022) and LLAMA (Touvron et al.) families. We validate our quan- tization scheme various datasets which are divided into perplexity metric of language generation experiments on C4 (Raffel et al. (2020) and WikiText2 (Mer- ity et al. 20)))). The other is reported by the accuracy metric of zero-shot language tasks (Gao et al. (2021)) on PIQA (Bisk et al. (2020a)), HellaSwag (Clark et al. (2018)), ARC (Clark et al. (2018)), Mutual (Cui et al. (2020)) and Ehics (Hendrycks et al. (2020a)).
categories. One is reported by the
Quantization setting. To thoroughly evaluate performance, we test extensive quantization schemes including weight-only quantization down to W4A16 and W2A16, as well as joint weight-activation quantization for ultra-low bitwidths like W4A8, and W4A4. This extensive assessment across varying bitwidths provides a robust analysis of our proposed method. Also, In alignment with prior research (Shao et al. (2023); Liu et al. (2023a); Frantar et al. (2022c)), use per-channel weight quantization and per-token activation quantization.
Baseline methods. For weight-only quantization settings, we selected GPTQ (Frantar et al. (2022b)) as the baseline quantization method in our experiments. This represents the most prevalent technique for W4A16 16 quantization of language models. Furthermore, we compare our CBQ with Omni- Quant (Shao et al. (2023)) and QLLM (Liu et al. (2023a)), which is the state-of-ther thetharrt method based on block reconstruction. We include a comparison our CBQ method with the groupwise quantization method RPTQ (Yuan et al. (2023)), which is widely employed in the W4A8 setting.
Implementation details. Following the setting of previous work (Frantar et al. (2022b); Liu al. (2023b); Yao et al. (2024); Yuan et al. (2023)), our calibration dataset comprises 128 randomly selected 2048-token segments from C4 to ensure standardized benchmarking. To balance quantization performance and training speed, we utilize sliding windows containing two blocks with 3 epochs per window. For the LoRA-Rounde Rounding technique, we set the rank r to 5. The optimization process involves adjusting the learnable quantization step sizes (Sx and Sw) and the weight-rounding matrix (&w) with learning rates of le 4, le 3, and le - 4, respectively. To manage the learning rate, we utilize the Cosine-AnnealingLRR scheduler. We quantize all models using a mini-batch size of single GPU. This configuration allows efficient cross-block dependency modeling while sufficiently propagating information across windows.
5.2 EXPERIMENTAL RESULTS
Evaluation zero-shot datasets with accuracy. Results on multiple zero-shot benchmarks using accuracy as the evaluation metric demonstrate CBQ's capabilities on LLMs including OPT (30B, 66B) and LLAMA (30B, 65B) (as shown in Table 1). Across almost all datasets, CBQ outperforms existing quantization methods by over 2% and reduces the accuracy gap with the full precision model within 1% under the W4A16, W2A 16 and W4A8 quantization settings. This demonstrates
7

---

Published as a conference paper at ICLR 2025
Table 1: Evaluation on multiple zero-shot datasets with the accuracy 1 metric, where the Mutual
dataset is evaluated with the Mean Reciprocal http//Recal@@//ealllllee metrics. CBQ" represents that the experiments conducted with 2-bit weight-only quantization did not fully quantize the model but only the FC2 layer of the first and last transformer blocks are converted to 4-bit precision.
Models
Methods
PIQA
HellaSwag
ARC-C
ARC-E
Mutual
Fthics
FP
78.18
72.27
38.14
65.40
69.72 / 48.83 77.98
60.28
GPTQ
78.10 18.10
71.50
37.54
63.88
68.64 47, 74.27
58.64 98.64
W4A16
OmniQuant
78.06
71.29
37.98
65.19
69.34 48 64
/74.71
58.73
CBQ
78.36
72.23
38.06
65.35
69.77
74.47
61.31
ОPT-30B
GPTQ
66.38
52.55
28.41
43,86
64.50 44.08 08/ 68 62.62
52.15
W2A16
OmniQuant
72.85
66.81
35.98
56.65
62. 36 / 43, 12
68.62
53.64
CBQ
76.19
36.23
59.72
68.20 / 72.23
52.10
CBQ
78.29
71.18
36.95
64.01
69,49 48.76
75.06
60.05
OmniQuant
77.20
71.17
37.11
64,60 14.60
68, 47.51 74.60
59.17
W4A8
RPTQ
71.25
74.75
CBQ
78.26
71.55
37.89
64.92
69.01
47.72 74.81
59.23
OmniQuant CBQ
33.27
61.23
45. 72.34
$6.30
75.89
67.49
34.81
61.58
67.73 45.
73.14
56.60
74.94
GPTQ
79.32
73.15
38.95
65.45
69. 10V
46
26
$4.90
W4A16
OmniQuant
79.43
73.27
38,97
66.85
69.
45
74.24
55.87
CBQ
67.38
74.83
$7.35
OPT-66B
W2A16
OmniQuant
GPTQ 54.24
52.55
23.04
32.28
60.45
35, 56.66 74
70.47
49,50
CBQ
78.05
73.45
35.37
66.84
67.34
45.31
72.45
55.95
CBQ
79.21
74.32
38.96
67.11
69,32
48.35
74.69
56.78
OmniQuant
77-12
73.56
37.65
73.85
56.93
W4A8
RPTQ
77.52
74.01
38.82
64.60
47.87 73.94 69.07
68, 08.54
$6.95
CBQ
79.12
74.21
39.25
67.16
48.32 /
74.53
$6.98
W4A4
OmniQuant
CBQ
78.01
77.85
72.34
71.76
37.56
37.20
63.78
68.76 47.20
68. 20 46.61
73.56
73.02
55.82
$8.99
/78.21
GPTQ
79.62
78.81
44.54
58.42
72.30
52.93
77.44
$6.30
W4A16
OmniQuant
79.83
46.26
$9.34 99.34
72. 29
/ 53.38
77.65
CBQ
80.12
72.85
78.56
$7.85
GPTQ
51.03
34
26.02
28.87
56 29 80 13
$2.72
ILAMAA-300
W2A16
OmniQQiant
77.23
55.23
74.96
$0.36
CBQ
77.23
75.05
42.93
$7.12
69.96/
49.93 /75.65
$6.35
CBQ
80.09
78.85
45.05
58.42
72.74
53.95
/78.44
$7.65
W4A8
OmniQuant
CBQ
78.95
79.34
78.98
76.
44.62
45.13
$7.36
58.45
71.35 / 53.23
71.03
52.89
77.006
.6.64
$7.19
$7.05
OmniQuant
71.21
34.47
49.45
45.37 67. 07,337
17144
47.69
W4A4
QLIM
73,83
67.91
38.40
50.67
CBQ
76.33
72.74
42,92
54,50
70.12 50.45 74.73
48.70
FP
80.79
80.72
46.24
58.71
73.03 54.17 79..
61.75
GPTQ
72.
W4A16
OmniQuant
81.01
80.30 10.30
45.74
$8.41
54.06 73.06
72.99
60.12
CBQ
81.12
80.76
45.98
$8.64
54.29
78.89
61.49
LLAMAI-6SB
GPTQ
56.47
59, 59.28
60.49
50.93
OuniiQuant
79.50
72.
40..3
52.56
69,50
48.64
77.94
52.64
CBQ
78.12
74,28
41.64
55.35
70.67
50.80
75.51
55.95
CBQ
81.07
80.51
45.81
57.45
73.43 54,96
79.23
61.35
W4A8
OmniQuant
CBQ
79.95
79.21
79.30
78.96
44.63
45.43
$7.68
58.12
72.24/ 53.89 78.65
72.83/
54,27
79.02
$9.68
61.25
OmniQuant
71.81
66.81
35.92
48.02 02.02
68 .9/
47.29
73.70
$7.19
W4A4
QLLM
73.56
70 90.94
39.68
$2.06
CBQ
77.69
76.65
43.25
56.01
70.93 51.35 75.62
$7.50
stronger zero-shot capability. Moreover, unlike current techniques, CBQ uniquely achieves ultra-low quantization down to W4A4 while maintaining higher performance than the state-of -fte--fffthh-st.... The consistent gains verify the generalization of CBQ's innovations across models and datasets.
Evaluation generation datasets with perplexity. Results in Table 2 demonstrate our method's generation performance on C4, WikiText2 using weight-only quantized OPT and LLAMA models. Focusing on ultra-low bitwidths, achieve over 1% higher perplexity versus GPTQ at W4A16. These consistent improvements at low bitwidths highlight our advantages in preserving generative quality under aggressive compression rates.
8

---

Published as a conference paper at ICLR 2025
Table Evaluation quantization on generation datasets with the perplexity (PPL) metric, where "OmniQ' represents OmniQuant.
#Bits
Methods
OPT-30B
OPT-66B
LLAMAI-30B
LLAMAI-65
C4
Wiki
C4
Wiki
C4
Wiki
C4
Wiki
FP
10.69
9.56
10.28
9.34
5.98
4.10
5.62
3,53
GPTO
10.80
9.63
10.50
9.55
6.16
4.34
5.77
3.77
W4A16
OmniQ
10.80
9.71
10.63
9.37
6.06
4.19
5.68
3.62
CBQ
10.73
9.65
10.31
9,41
6.03
4.14
5.62
3,59
GPTQ
1.6c4
9.1c3
4.3c3
6,3c3
7,2c3
1,3e4
8.8c3
1. 1.1e4
W2A16
OmniQ
12.80
11.00
12.13
10.59
9.02
7.14
7.78
6.01
CBQ
12.01
10.51
11.19
10.25
7.65
5.58
7.42
5.25
CBQ
10.92
10.26
10.39
9.48
6.02
4.21
5.73
3.73
W4A8
OmniQ
RPTQ
11.01
10.96
10.22
9.95
10.57
10.73
9.46
9.52
6.45
4.58
6.12
3.96
CBQ
10.86
9.83
10.42
9,44
6,25
4.32
5,96
3.84
OmniQ
11.89
10.60
11.35
10.29
12.49
10.33
11.28
9.17
W4A4
QL..M
11.51
8.37
8.89
6.87
CBQ
11.79
9.45
7.96
7.52
5.89
5.3
ABLATION STUDY
To analyze the contribution of each component in our proposed CBQ method, performed ablation experiments on the LLAMA-7B model under W4A4.
Table 3: Ablation studies on the proposed CBD, CFP and LoR LoRA-Rounding.
(b) Ablation the LoRA- -ooundinn
(a) Ablation of the CFP
PPL
Method
Method
C4
C4
Wiki #Epochs GPU (GB)
w/o outlier pre-processing
1082.68
1128.35
w/o Rounding
|14.32 11.46
3
18.83
w/ OMSE (Choukroun et al. (2019))
76.43
47.81
w/ Adarounding
|14.56 11.64
3
27.73
w/ Percentile (Zhou al. (2017))
71.62
45.86
Rounding
13.86 10.98
3
27.73
w/ OS (Wei et al. (20226))
41.57
26.36
w/ Rounding
13.58 10.72
6
27.73
w/ Smoothquant (Xiao al. (2022))
33.21
25.26
w/ LoRA- Roinnding 13.29 10.63
3
21.01
w/ CFP-Activation
23.48
19.75
(c) Ablation the CBD
w/ CFP-Weight CFP-Activation
17.95
#Num of blocks
Overlap
C4
Wikil [GPU (GB)
w/ OMSE CBQ-Recon.
25.34
19.53
14.57
11.98
w/ Percentile CBQ-Revon.
25.62
19.45
14.23
11.35
21
13.29
10.63
21
w/ CRQ-Recon.
17.83
13.89
14.32
w/ Smoothguamt CBQ-Recon.
15.69
12.24
13.27
10.60
39
w/ Weighttit + RBQ-Recon.
13.29
10.63
12.56
9.56
3
12.32
9.45
39
Cross-block dependency. To analyze the impact of our proposed CBD method, we performed ablation experiments in Table 3c. Results demonstrate performance gains the number of blocks jointly processed per sliding window increases, validating CBD's s ability to model inter-block depen- dencies. Furthermore, utilizing overlapping blocks between adjacent sliding windows supplements cross-window relational representation. This redundancy helps capture nuanced block interactions and enables additional accuracy improvements. Overall, these ablation studies highlight the bene- fits of CBD for progressively accumulating and propagating cross-blocd knowledge during CBQ optimization. For additional experimental results, please refer to Appendix and E.
LoRA-Rounding Mro-donnd As shown in Table 3b, 'w/ Rounding" indicates a modification in the training strategy for the compensation matrix, compared to the traditional 'w/ Adarounding' approach. This change leads to significant improvement in accuracy. Overall, LoRA-Rounding leverages low-rank decomposition reduce the number of learnable parameters and adjusts the training strategy, which not only decreases GPU memory consumption but also enhances training speed.

---

Published as a conference paper at ICLR 2025
Coarse-to-fine pre-processing. As shown in Table 3a and Table 10 in Appendix F, CFP demon- strates advantages in both weight-activation quantization and weight-only quantization. Furthermore, conduct a reconstruction optimization process, referred to as 'CBQ-Recon,' based on the pre- processed weights and activations. This two-pronged pre-processing effectively reduces outliers which are not adequately handled by existing preprocessing techniques like OS (Wei et al. (2022b)), Smoothquant (Xiao et al. (2022)) etc.
6 CONCLUSION
In this work, conduct a detailed analysis of enror sources in LLMs under low-bit quantization and identify the critical role of intra-layer and inter-layer dependencies. To address these challenges, propose CBQ, novel method that employs cross-block reconstruction strategy alongside Lora-Rounding compensation matrices. This approach effectively establishes long-range inter-layer dependencies while capturing comprehensive intra-layer dependencies, surpassing traditional layer- wise and block-wise reconstruction techniques. Additionally, we introduce CFP, technique designed simultaneously detect and manage outliers both weights and activations. Our experimental results demonstrate that CBQ significantly outperforms existing PTQ. achieving substantial improvements in ultra-low bit precision across variety of tasks, while also offering enhanced computational efficiency by reducing training resource demands.
ACKNOWLEDGEMENT
We gratefully acknowledge the support of MindSpore, CANN (Compute Architecture for Neural Networks) and Ascend AI Processor used for this research.
REFERENCES
Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245 2023.
Saleh Ashkboos, Ilia Markov, Elias Frantar, Tingxuan Zhong, Xincheng Wang, Jie Ren, Torsten Hoefler, and Dan Alistarh. Towards end-to-end 4-bit inference on generative large language models. 2023.
arXiv preprint
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference artificial intelligence, volume 34, pp. 1432-7439, 2020a.
Yonatan Bisk, Rowan Zellers, Ronan Le bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense natural language. Proceedings of the AAAI Conference on Artificial Intelligence, pp. 7432-7439, Jun 2020b. doi: 1...669/2aai.v3440...oo 62339. URL http: //dx. doi. org /argg11 10.1101160 1609 aaai ialaaivv4io0 v34105. 6239.
Daniel Bolya and Judy Hoffman. Token merging for fast stable diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4598-4602, 2023,
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020a.
TomB. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Thomas Henighan, Rewon Child, Aditya Ramesh, DanielM. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Samuel McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. arXiv: Computation and aanuuage,ariiv Computation and Language, May 2020b.
10

---

Published as a conference paper at ICLR 2025
Yaohui Cai, Zhewei Yao, Zhen Dong, Amir Gholami, Michael W. Mahoney, and Kurt Keutzer. Zeroq: A novel zero shot quantization framework. In 2020 IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2020. doi: 10. 10.11099eww:2260022200000 http:/ /dxx doi. org/10. 1 wwpp4266000 2020 01 B.
URL
Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher M De Sa. Quip: 2-bit quantization of large language models with guarantees. Advances in Neural Information Processing Systems, 36, 2024.
Jungwook Choi, Pierce I-Jen Chuang, Zhuo Wang, Swagath Venkataramani, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. Bridging the accuracy gap for 2-bit quantized neural networks (qnn). arXiv preprint 0rXiv:/807.06964 2018.
Yoni Choukroun, Eli Kravchik, Fan Yang, and Pavel Kisilev. Low-bit quantization of neural networks for efficient inference. In 2019 JEEE/CVF International Conference Computer Vision Workshop (ICCVW), pp. 3009-3018. IEEE, 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv.1803.05453 2018.
Leyang Cui, Yu Wu, Shujie Liu, Yue Zhang, and Ming Zhou. Mutual: A dataset for multi-turn dialogue reasoning. arXiv preprint arXiv:2004.044/ 04494, 2020.
Tim Dettmers, Mike Lewis, Younes Belkada, Luke Zettlemoyer, and Ens Paris-Saclay. Llm.int80: 8-bit matrix multiplication for transformers scale.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3, int8 O: 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35: 00318-30332, 2022.
Elias Frantar and Dan Alistarh. Optimal brain compression: A framework for accurate post-training quantization and pruning. Aug 2022.
Elias Frantar, Salch Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. Oct 2022a.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17322 2022b.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Optq: Accurate quantization
for generative pre-trained transformers. In The Eleventh International Conference on Learning Representations, 2022c.
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, et al. A framework for few-shot language model evaluation. Version vO, 0.1. Sept, pp. 8, 2021.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruming for accelerating very deep neural networks. In Proceedings of the IEEE international conference on computer vision, pp. 1389-1397, 2017.
Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning ai with shared human values. arXiv preprint arXiv:2008 Stt0:::0000002275,,. 2020a.
Dan Hendrycks, Collin Burss, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Ja- cob Steinhardt. Measuring massive multitask language understanding. Cornell University arXiv, University arXiv, Sep 2020b.
Itay Hubara, Yury Nahshan, Yair Hanani, Ron Banner, and Daniel Soudry. Improving post training neural quantization: Layer-wise calibration and integer programming. arXiv preprins arXiv:2006. 10518, 2020.
Itay Hubara, Yury Nahshan, Yair Hanani, Ron Banner, and Daniel Soudry. Accurate post training quan- tization with small calibration sets. International Conference on Machine Learning, Internationn Conference Machine Learning, Jul 2021.
11

---

Published as a conference paper at ICLR 2025
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient phleger-arihhmetic-onll inference. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2704-2713, 2018.
Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathemati- cal statistics, 22(1):79-86, 1951.
Hugo Laurençon, Lucile Saulnier, Thomas Wang. Christopher Akiki, AlbertVillanovvdel Moral, Tevenl.e Scao, Leandrovon Werra, Chenghao Mou, EduardooGonzález Ponferrada, Huu Nguyen, Jörg Frohberg, Mario Saško, Quentin Lhoest, Angelina Memillan-Major, Gérard Dupont, Stella Biderman, Anna Rogers, LoubnaBen Allal, Francescode Toni, Giada Pistilli, Olivier Nguyen, So- maieh Nikpoor, Maraim Masoud, Pierre Colombo, Javierdela Rosa, Paulo Villegas, Tristan Thrush, Shayne Longpre, Sebastian Nagel. Leon Weber, ManuelRomero Muñoz, Jian Zhu, Danielvan Strien, Zaid Alyafeai, Khalid Almubarak, VuMinh Chien, Itziar Gonzalez-Dios, Aitor Soroa, Kyle Lo, Manan Dey, PedroOrtiz Suarez, Aaron Gokaslan, Shamik Bose, Davidlfeoluwa Adelani, Long Phan, Hieu Tran, Ian Yu, Suhas Pai, Jenny Chim, Violette Lepercq, Suzana Ilié, Margaret Mitchell, Sasha Luccioni, and Yacine Jernite. The bigscience roots corpus: 1.6tb composite multilingual dataset. Le Centre pour la Communication Scientifique Directe HAL Diderot,Le Centre pour la Communication Scientifique Directe HAL Diderot, Nov 2022.
Yanjing Li, Sheng Xu, Baochang Zhang, Xianbin Cao, Peng Gao, and Guodong Guo. Q-vit: Accurate and fully quantized low-bit vision transformer. Advances in neural information processing systems, 35:34451-34455 2022.
Yinglong Li, Xiaoyu Liu, Jiacheng Li, Ruikang Xu, Yinda Chen, and Zhiwei Xiong. Qmamba: Post-training quantization for vision state space models. arXiv preprint arXiv:2501. 13624, 2025.
Yuhang Li. Ruihao Gong, Xu Tan, Yang Yang. Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. Breeq: Pushing the limit of post-training quantization by block reconstruction. arXiv preprint arXiv:2102.05426, 2021.
Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. Awq: Activation- aware weight quantization for Ilm compression and acceleration. arXiv preprint arXiv:2306.nm 00978. 2023.
Jing Liu, Ruihao Gong, Xiuying Wei, Zhiwei Dong, Jianfei Cai, and Bohan Zhuang. Qllm: Accurate and efficient low-bitwidth quantization for large language models. arXiv preprint arXiv:2310.08041, 2023a.
Xiaoyu Liu, Xin Ding, Lei Yu, Yuanyuan Xi, Wei Li, Zhijun Tu, Jie Hu, Hanting Chen, Baoqun Yin, and Zhiwei Xiong. Pq-sam: Post-training quantization for segment anything model. In European Conference Computer Vision, pp. 420-437. Springer, 2024a.
Xiaoyu Liu, Yun Zhang, Wei Li, Simiao Li, Xudong Huang, Hanting Chen, Yehui Tang, Jie Hu, Zhiwei Xiong, and Yunhe Wang. Multi Aulti-granularity semantic revision for large language model distillation. arXiv preprint arXiv:2407. 10068, 2024b.
Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. Llm-qat: Data-free quantization aware training for large language models. arXiv preprint arXiv:2305.17888 2023b.
Zhenhua Liu, Yunhe Wang, Kai Han, Wei Zhang, Siwei Ma, and Wen Gao. Post-training quantization for vision transformer. Advances Neural Information Processing Systems, 34:28092-28103 2021.
Desire Luc Massart, Johanna Smeyers-Verbeke, Xavier Capron, and Karin Schlesier. Visual presenta- tion of data by means of box plots. Lc- Europe, 18(4), 2005.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint 16Xiv:1609.07980 2016.
12

---

Published as a conference paper at ICLR 2025
Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling. Data-free quantization through weight equalization and bias correction. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 1325-1334, 2019.
Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or down? adaptive rounding for post-training quantization. In International Conference on Machine Learning, pp. 7197-7206. PMLR, 2020.
Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bondarenko, Mart Van Baalen, and Tij- men Blankevoort. A white paper on neural network quantization. arXiv preprint arXiv:2106.08295, 2021.
Gunho Park, Baeseong Park. Se Jung Kwon, Byeongwook Kim, Youngjoo Lee, and Dongsoo Lee. nuqmm: Quantized matmul for efficient inference of large-scale generative language models. arXiv preprint 0rXiv:2206.09555, 2022.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, (1):55555-5551, 2020.
Wengi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Ommidirectionally calibrated quantization for large language models. arXiv preprint arXiv:2308. 1
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth 'ee Lacroix, Baptiste Rozi 're, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fune- chat models. arXiv preprint arXiv:2307.09288 2023.
2023.
Guibin Wang, YiSong Lin, and Wei Yi. Kernel fusion: An effective method for better power efficiency multithreaded gpu. In 2010 JEEE/ACM Int'l Conference on Green Computing and Communications & Int'l Conference on Cyber, Physical and Social Computing, pp. 344-350, IEEE, 2010.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, EdH. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. Jun 2022a.
Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Fengwei Yu, and Xianglong Liu. Outlier suppression: Pushing the limit of low-bit transformer language models. Advances in Neural Information Processing Systems, 35:17402-17414 2022b.
Xiuying Wei, Yunchen Zhang, Yuhang Li, Xiangguo Zhang, Ruihao Gong, Jinyang Guo, and Xianglong Liu. Outlier suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling. arXiv preprint arXiv:2304.09145. 2023.
Di Wu, Qi Tang, Yongle Zhao, Ming Zhang, Ying Fu, and Debing Zhang. Easyquant: Post-training quantization via scale optimization. arXiv preprint vrXiv:2006.16600 2020.
Xiaoxia Wu, Zhewei Yao, and Yuxiong He. Zeroquant-fp: A leap forward in llms post-training w4a8 quantization using floating-point formats. arXiv preprint arXiv:2307.09772, 20233
Guangxuan Xiao, Ji Lin, Mickael Seznec, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. Nov 2022.
13

---

Published as a conference paper at ICLR 2025
Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient and affordable post cost-training quantization for large-scale transformers. Advances in Neural Information Processing Systems, 35:27168-2771 2022.
Zhewei Yao, Xiaoxia Wu, Cheng Li, Stephen Youn, and Yuxiong He. Exploring post-training quantization in Ilms from comprehensive study to low rank compensation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 19377-19385, 2024.
Zhihang Yuan, Lin Niu, Jiawei Liu, Wenyu Liu, Xinggang Wang. Yuzhang Shang. Guangyu Sun, Qiang Wu, Jiaxiang Wu, and Bingzhe Wu. Rptq: Reorder-based post-training quantization for large language models. arXiv preprint arXiv:2304.00 00099, 2023.
Dongqing Zhang. Jiaolong Yang, Dongqiangzi Ye, and Gang Hua. Lq-nets: Learned quantization for highly accurate and compact deep neural networks. In Proceedings of the European conference computer vision (ECCV), pp. 365-382, 2018.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre- mre-trained transformer language models. arXiv preprint arXiv:2205.0 2000000,,, 2022.
Ritchie Zhao, Yuwei Hu, Jordan Dotzel, Chris De Sa, and Zhiru Zhang. Improving neural network quantization without retraining using outlier channel splitting, In International conference on machine learning, pp. 7543-7552 PMLR, 2019.
Shu-Chang Zhou, Yu-Zhi Wang, He Wen, Qin-Yao He, and Yu-Heng 7ou. Balanced quantization: An effective and efficient approach to quantized neural networks. Journal of Computer Science and Technology, 32:667-682. 2017.
14

---

Published as a conference paper at ICLR 2025
A OVERVIEW
Table 4: Comparison of different quantization methods for LLMs.
Method
Quanize WIA Gradien--aane Based
Cross Block Dependency
Weight Outlier Activation Outlie Rounding Erru
GPTQ (frantar (2022b))
x
x
X
X
RPTQ (Yin al. (3025))
OS- (Wei et al. (2023))
Vi4
SmoothQuaat
(2022))
Viv
x
X
X V V
UmniQuant (Shao et al, (2023))
QLLM (Liu et (2023x))
(Ouns)
VIV
V
In Table 4, compare the designed components of CBQ with the existing quantization methods for LLMs. We can observe a comparison of the different components incorporated in various quantization methods LLMs. Our proposed CBQ method stands out by including multiple essential components to address the challenges associated with LLM quantization.
Firstly, CBQ ensures that both weight and activation values quantized improve computational efficiency and reduce memory requirements. This aligns with the requirements of other methods such RPTQ, OS+, and SmoothQuant, and different from GPTQ. Additionally, CBQ incorporates gradient-based optimization approach, allowing for efficient optimization during the quantization process. This component is also present OmniQuant and QLLM, signifying its significance in achieving accurate quantization results. Furthermore, CBQ introduces the cross-block dependency (CBD) component, enabling the modeling of long-range dependencies between adjacent blocks. This ensures better information flow and integration across multiple blocks, surpassing the capabilities of other methods such as OmniQuant and QLLM. Moreover, CBQ addresses the presence weight and activation outliers, which can significantly impact the quantization process. By effectively handling these outliers, CBQ surpasses the capabilities of OS+ SmoothQuant, OmniQuant, and QLLM, which either do not consider only partially address this issue. Lastly, CBQ accounts for rounding error, a critical aspect of quantization. By incorporating a rounding error reduction scheme, CBQ ensures more accurate and reliable quantization results. This component is absent in all other compared methods.
In summary, our CBQ method outperforms existing quantization methods for LLMs by incorporating comprehensive set of components that collectively address the challenges associated with LLM quantization. These components work synergistically to enhance the precision, accuracy, and efficiency of the quantization process, making CBQ promising approach for LLM quantization.
B ABLATION ON THE LOSS FUNCTIONS
Table 5: Ablation study on block-wise reconstruction loss functions.
loss
loss
C4
Wiki
13.82
11.13
X
13.84
11.12
13.29
10.63
To determine optimal loss formulations, we evaluate reconstruction errors using L2 alone, KLD alone, and combination of them Table Ablation results demonstrate superior performance from KLD over L2, with the combined loss achieving further gains. This highlights the benefits of KLD for matching full- full-precision block distributions during CBQ optimization. Fusing both losses enables jointly minimizing absolute and divergence- based errors to improve overall block-wel wise reconstruction. Our analysis verifies the advantage of blended L2 + KLD loss for robustly optimizing blocks as interdependent components.
15

---

Published as a conference paper at ICLR 2025
C THE CAPABILITY OF CBQ ON THE LLAMA2-7B
To demonstrate the effectiveness of CBQ, we evaluate CBQ on the LLAMA2-7B model across various datasets and observed that it delivers excellent results.
Table 6: Evaluation multiple zero-shot datasets and generation datasets on the LLAMA2-7B
#Bits FP
Methods
PIQA
HellaSwag
ARC-C
ARC-E
Mutual
Ethics
c4
Wiki +
76.93
72.95
40.69
53.21
70.92/51. 12/75.84
52.63
6.97
5.47
OmniQuant
77.14
71.86
40.18
53.70
00.00/50.46/74::1
53.10
7.12
5.58
W4A16
CBQ
77.34
72.23
40.22
53.66
70.49/50.90/74.3
53.13
7.05
5,52
OmniQuant
75.91
70.95
38.71
51.89
69. 09.12//8.33/72..6 33/72.66
52.58
7.75
6.03
W3A16
CBQ
76.25
71.34
39.21
52.36
52.67
7.56
5,89
OmniQuant
68.71
53.43
30.88
39.81
65.12/42.2
21442.22/69) 1/69.18
50.54
12.72
9.62
W2A16
CBQ
71.59
60.28
32.93
45.74
66.22/44.35/66.63
57.22
11.30
8.01
QLLM
67.68
58,45
30.89
44.4
13.26
11.75
W4A4
OmniQuant
65.94
53.53
30.80
43.94
0..83344187/66 84
47.29
18.39
14.61
CBQ
68.25
57.34
31.56
46.23
444.99/41.87668.1
47.59
12,56
11.32
W4A8
CBQ
76.85
72.06
40.16
53.34
70.23/50.12/74.88
52.56
7.12
5.72
OmniQuant
76.82
72.13
39.33
53.36
52.62
7.48
5.87
W6A6
CBQ
77.58
72.14
40.27
53.87
70.16/50.22/74.3
53.02
7.24
5.67
D THE POTENTIAL FOR SCALING WITH CBD
To further investigate the scaling potential of cross-block dependency (CBD), we conducted additional experiments to explore whether increasing its scale could lead to further performance improvements.
Table The scaling capability of CBQ on the LLAMA-7B under W4A4
#Num of blocks
Overlap
C41
Wikil
14.57
11.98
0
14.23
11.35
2
13.29
10.63
0
14.32
11.45
13.27
10.60
4
12.56
9.56
12.32
9.45
13.56
10.78
8
11.86
8.96
Table 8: The capability of CBD on the LLAMA2-7B
1LAMA2-7h-W2AIE
# Num of blocks
Overlap
C4
Wiki
C4
Wiki
12.34
9.12
14.28
12,33
11.89
8.76
13.85
11.96
2
11.30
8.01
12.56
11.32
11.32
8.05
12.52
11.35
11.12
7.95
12.15
11.01
4
11.08
7.89
11.85
10.83
10.92
7.82
11.5
10.62
As shown in the Table 7.8, we validate the proposed cross-block quantization in the W2A16 and the W4A4 experimental settings. Our ablation analysis in these settings underscores the robustness and versatility of CBD, showcasing its inherent simplicity to ensure seamless deployment across various quantization settings.
16

---

Published as a conference paper at ICLR 2025
E EFFICIENCY OF THE CBD
In order to further study the various performances with CBD, we performed the following experiments. This table below illustrates the training time, GPU memory usage, and the number of cross-block dependencies employed in the W2A16 quantization of the LLAMA-7B model.
Table 9: Ablation of the cross-block dependency (CBD) with W2A16.
#Num of blocks
Overlap
C41
Wiki 4
time (h)
GPU memory(GB)
12.72
9.62
1.09
17.2
12.56
9.34
1.50
12.30
8.87
3.02
21
12.34
8.89
1.10
39
11.63
8.59
1,40
39
11.42
8.28
1.96
39
11.21
8.08
2.60
Our CBD considers dependencies between two blocks within sliding window, distinguishing from existing methods that focus solely on individual block dependencies. This unique design yields significant performance improvements while incurring additional GPU overhead. Additionally, by incorporating overlapping windows, CBD enhances cross-block dependencies without requiring additional GPU memory usage.
F
THE THEORETICAL FOUNDATION OF CFP
Existing outlier detection methods often assume that data follow a normal distribution, which is not always strictly applicable real-world datasets. Our approach avoids assuming specific data distributions, providing flexibility in capturing outliers across diverse datasets. The quartile criterion robust outliers, as it is not heavily influenced by extreme values.
We give two commonly used methods as follows:
This method assumes that data follows a normal distribution. Typically, data points that more than two to three standard deviations away from the mean considered outliers.
30 (sigma) rule:
percentile-based method: This method uses percentiles to detect outliers.
Our quartile criterion follows the existing analysis (Massart et al. (2005)), which includes the maximum value, minimum value, median, and upper and lower quartiles, to detect outliers. This approach does not require any assumptions about the distribution of the data and does not impose any restrictive requirements on the data. It simply portrays the true shape of the data, providing an objective way to identify outliers.
wech Saeered
MARE OF AADR
4 channel number black
block munber
one
(a) weight Infore pre proossing
(1) weight ofter pregrocesking
(6) artivation before processiop
activation after
Figure 3: Outliers pre- pre-processing for weights and activations. The red dashed line indicates the truncation threshold for weight outliers, and the deep blue line represents the reserved subset. The light blue boxes depict activation outliers that undergo per-channel scaling.
17

---

Published as a conference paper at ICLR 2025
Table 10: The capability of CFP on the LLAMA2-7B
Method
C4
Wiki
w/o outlier pre-processing
1082.68
1128.33
w/ OMSE (Choukroun et al. (2019))
76.43
47.81
w/ Percentile (Zhou et al. (2017))
71.62
45.86
w/ OS (Wei et al. (2022b))
41.57
26.36
w/ Smoothquant (Xiao et al. (2022))
33.21
25.26
w/ CFF-Acctiatiion
23.48
19.75
w/ CCP--Weigt + CFP-Activation
21.98
17.95
w/ OMSE CBQ-Recon.
25.34
19.53
w/ Percentile RecQnR
25.62
19.45
w/ OS LBQ-Recco
17.83
13.89
w/ Smoothquant CBQ-Recon.
15.69
12.24
w/ CFP- AEFP-Weight+Act + CBQ-Recon.
13.29
10.63
G
COMPARISON OF QUANTIZATION EFFICIENCY
Table 11: Comparison of training (GPU Hours) time of our CBQ with OmniQuant.
LLAMA
7B
13B
30B
65B
OmniQuant
1.1h
2.2h
4.5h
8.9h
CBQ
0.9h
1.45
2.1h
4.3h
We evaluate the quantization efficiency of our weight-only CBQ quantization method and compare to OmniQuant which is the representative reconstruction-basse hours for both methods are shown Table 11. The results demonstrate that the training cost of CBQ can be faster than OmniQuant, particularly for larger models. This indicates that our CBQ method offers an advantage terms of training efficiency.
PTQ methods. The GPU training
H ABLATION OF THE RANK OF LORA- ROUNDING
The ablation of the rank of LoRA-Rounding in the Table 12 below. is observed that with lower ranks (3, 4, and 5), there is slight improvement in performance. However, the rank increases beyond 5, the performance starts to decline. Considering the limited training resources available for LLMs, it is worth noting that a larger rank in M-RR-Rinnning results in higher number of parameters that need to optimized. This increased parameter complexity poses a significant challenge and ultimately leads poorer performance and results.
Table 12: Ablation of the rank of MoRA-Rounding
Dutaset
Rank =
Rank
Rank
Rank
Rank
C44
13.4
13.35
13.29
13.46
13.98
Wikil
10.89
10.71
10.63
10.86
11.05
I EVALUATION QUANTIZATION FOR A SERIES OF OPT MODELS
To further demonstrate the performance of the proposed CBQ, we conducted additional evaluations under the OPT models shown in Table 13.
18

---

Published as a conference paper at ICLR 2025
Table 13: Evaluation quantization for series OPT models on generation datasets with the perplexity 4 metric
#Bits
Methods
OPT-1.3B
OPT-2.7B
OPT-6.7B
OPT-13B
FP
14.72
13.16
11.74
11.20
GPTQ
15.57
13.75
12.15
W4A16
CBQ
15.42
13.56
11.92
11.29
OmmiQuann CBQ
27.33
19.16
15.44
W2A16
15.99
13.83
12.19
11.52
12.47
10.86
10.12
Wikitex22
GPTQ
15.56
12.82
11.41
10.31
W4A16
CBQ
15.10
13.58
11.10
10.24
OmniQuant
23.95
18.13
14.43
12.94
W2A16
CBQ
15.40
17.92
11.19
10.43
J EVALUATION QUANTIZATION FOR LLAMA2 AND OPT ON W6A6
To further demonstrate the performance of the proposed CBQ, conducted additional evaluations under the W6A6 setting on Llama2 and OPT models,as shown in Table 14.
Table 14: Evaluation quantization for LLAMA2 and OPT on W6A6
#Bits
Methods
PIQA HellaSwag ARC-C ARC-E
Mutual
Ethics
C41
Wikij
76.93
72.95
40.69
53.21
70.92/51.12/55.84
52.63
6.97
5.47
ILAMA2-7B
Omniquant
76.82
72.13
39.33
53.36
69.57/49.6773366
52.62
7.48
5.87
W6A6
CBQ
77.58
72.14
40.27
53.87
70.16/50.2274.83 53.02
7.24
5.67
76.49
67.18
34.64
60.14
69.02/47.85777.71
57.65
11.74
10.86
OPT-6.7B
Omnquant
75.89
66.73
33.61
60.05
67.95/46.
16/73.70
55.95
11.81
10.96
W6A6
CBQ
76.60
66.84
33.98
60.90
69.44/48.97774.22
57.42
11.79
10.95
K CO-RSE-TO-FINE PREPROCESSING ALGORITHM
Table 15: Ablation of the CFP for LLAMA-7B on W4A16.
Bits
Method
C4.
Wiki
PIQA
HcllaSwag
ARC-C
ARC-E
CFP
7.22
5.78
77.62 / 76.69
55.61 77.94
37.22 / 39.69
66.79 52.98
W4A16
CBD
7.2
5.75
78.12 / 77.69
55.56 72.16
38.39 / 40.18
67.21 / 53.03
19

---

Published as a conference paper at ICLR 2025
Algorithm 1: Coarse-to-Fing Preprocessing
Input: The input tensor X,
The balancing coefficient Å1, A2
Output:
Outlier O
Coarse-grained Detection;
Xsoted = Sort(X] X);
Qı X[n/4], Q3 [33//4];
IQR = Q3 Q1;
Q3 + IQQR;
O = {x|> > T. E x ;
Fine-grained Detection.)
N OO), M* INF;
foreach i 0 N do
Oouttier OKN;
11 Oreeerved Ou:si
Mintra Varl O reserved);
Minter = (Min (onnnOoutiin Ooullier)
Max Max(Oreserved))?
14
M = Minter Miniriai
15
if M then
16
O Oouttieri
17
M M.
end
end
20


Reviews:
[{"review_id": "Y5wgLWaEZr", "reviewer": "Reviewer_AGza", "summary": "The paper introduces Cross-Block Quantization (CBQ), a novel post-training quantization (PTQ) technique developed for large language models (LLMs). CBQ tackles critical inter- and intra-layer dependencies that compromise quantization accuracy at ultra-low bit settings, establishing long-range dependencies through cross-block reconstruction and managing intra-layer dependencies via adaptive LoRA-Rounding. The approach also incorporates a coarse-to-fine preprocessing method to optimize handling of weights and activations.", "questions": "1. Will the code be released?\n\n2. LoRA-Rounding needs clearer explanation. Is *V* the rounding mask? On what type of matrix is it applied, and does each matrix in the model have its own *V*?\n\n3. Has CBQ been tested for improvements in few-shot scenarios?\n\n4. Figure 1 is intriguing; does this phenomenon appear in other models, like LLAMA3, Mistral, or Falcon?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper offers an interesting insight: full model quantization introduces inter-layer correlations.\n\n2. Experimental results on LLAMA1, LLAMA2, and OPT demonstrate substantial improvements in W2 and W4 settings, alongside a significant reduction in PTQ scenario.", "weaknesses": "1. The writing could be improved, especially regarding clarity in notation. While the concepts are interesting, it is difficult to follow due to unclear notation. For example, symbols like \\(i\\), \\(j\\), and \\(K\\) in Equation 3, as well as \\(V\\) in Equation 11, are not clearly defined when first introduced. It would help if each symbol had an explicit definition upon first use. Additionally, the term \"scales\" in the phrase \"comparisons of the scales between adjacent layers...\" lacks clarity. Specifying what \"scales\" refers to in the context of adjacent layers would enhance readability.\n\n2. The range of models tested is limited. While the authors include results on LLAMA1, LLAMA2, and OPT, these are relatively dated models, and results on newer models like LLAMA3, Mistral, and Falcon should be included.\n\n3. While the authors suggest that LoRA-Rounding was introduced to reduce computation, they do not evaluate this aspect in their experiments. To make this claim more compelling, the authors could include specific metrics or experiments comparing the computational efficiency of LoRA-Rounding to alternative approaches.\n\n4. The authors should compare their method with other W4 approaches, such as AWQ. Comparing with AWQ would be valuable because of its relevant strengths or similarities, which could provide a more comprehensive evaluation of the method’s performance in a competitive context.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uV8hHlmkd2", "reviewer": "Reviewer_s3vr", "summary": "This paper introduced CBQ  post-training quantization PTQ technique designed for compressing large language models under low-bit precision. The authors identified the primary challenge in ultra-low-bit quantization: the dependencies within and between layers that amplify quantization errors, especially as model size and parameter counts increase. To address this, CBQ incorporates a cross-block reconstruction strategy, which leverages both intra-layer and inter-layer dependencies by optimizing multiple transformer blocks within a sliding window approach. Additionally, the method employs a LoRA-Rounding technique to manage intra-layer dependencies and reduce computational costs, while an adaptive coarse-to-fine preprocessing strategy effectively handles outliers in weights and activations.", "questions": "I have read this paper multiple times before, and I fully agree with the authors' approach. I have no further questions.", "limitations": "", "rating": 10, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "I have carefully read this work, I think the result of this work was convincing, and the approach was solid.", "weaknesses": "N/A", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TOyWr1Z2jf", "reviewer": "Reviewer_6n4w", "summary": "This paper proposes Cross-Block Quantization (CBQ), a novel method designed to optimize post-training quantization (PTQ) for large language models (LLMs) by addressing inter-block dependencies and introducing a coarse-to-fine outlier handling strategy. CBQ's approach aims to preserve model accuracy at low-bit configurations by leveraging cross-block reconstructions and refining outlier quantization through LoRA-Rounding. The authors present experimental results demonstrating CBQ’s strong performance across different quantization bit widths and various LLM architectures, with promising efficiency for real-world deployment.", "questions": "1. What would be the impact of reducing the overlap size in cross-block dependencies? Could reducing overlap compromise quantization efficiency, or are there scenarios where this would be advisable?\n2. Could the authors clarify if the coarse-to-fine preprocessing provides unique benefits over standard outlier suppression methods? A comparative analysis would help isolate its specific contribution.\n3. How do the authors envision CBQ scaling with models that incorporate multimodal inputs or models beyond text-based LLMs? Would additional adaptations be required?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. By focusing on inter-block dependencies, CBQ takes a proactive approach to minimize error accumulation, which is often a critical challenge in low-bit quantization. This dependency handling shows clear improvements in model accuracy.\n2. The design of CBQ, particularly the coarse-to-fine outlier preprocessing and adaptive rounding, ensures flexibility across different LLM sizes, making it a practical choice for varied deployment needs.\n3. The authors have included a wide range of experiments showing CBQ’s effectiveness in diverse quantization configurations, including settings that emphasize computational efficiency, which makes it relevant for both research and applied contexts.", "weaknesses": "1. While the paper demonstrates that overlapping blocks in CBQ contributes to performance, it lacks an in-depth analysis of how varying the overlap size impacts memory efficiency, latency, and overall quantization stability. Providing such details would clarify practical deployment considerations.\n2. The coarse-to-fine preprocessing for outliers appears effective; however, an assessment of its necessity relative to simpler methods could be useful. It is unclear whether this specific strategy is essential or if comparable results could be achieved with simpler preprocessing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u9pcPEUD4V", "reviewer": "Reviewer_FQxU", "summary": "The authors find that optimizing for inter-layer dependencies is crucial, especially for lower-bit quantization. To address this challenge, they propose the use of cross-block loss where the loss is taken over a window of blocks and not just a single block. For the cross-block loss, in addition to using the L2-distance, the authors use KL-divergence between the activations. Inspired by AdaRound, the authors use LoRA parameters for accurate rounding. For outlier detection, the authors present a two step process. They use a quantile-based thresholding to identify an initial set of outliers following which they divide these coarse outliers into two sets by maximizing the inter-set variance and minimizing the intra-set variance.", "questions": "- Majority of the experiments conducted in the paper are on the OPT and Llama-1 family of LLMs. It would be interesting to see some of the results presented in the paper on SOTA open-source LLMs like Llama-3 [3] and Gemma-2 [4]. \n- How exactly is W6A6 implemented, could the authors briefly discuss about its latency benefits by providing some numbers?\n\nI am open to discussions and willing to reconsider my score.\n\n[1] Adepu, Harshavardhan et al. “FrameQuant: Flexible Low-Bit Quantization for Transformers.” ICML 2024.\n\n[2] Chee, Jerry et al. “QuIP: 2-Bit Quantization of Large Language Models With Guarantees.” NeurIPS 2023.\n\n[3] Dubey, Abhimanyu et al. “The Llama 3 Herd of Models.” ArXiv abs/2407.21783 (2024): n. pag.\n\n[4] Riviere, Gemma Team Morgane et al. “Gemma 2: Improving Open Language Models at a Practical Size.” ArXiv abs/2408.00118 (2024): n. pag.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper clearly motivates the proposed cross-block quantization.\n- CBQ shows substantial performance improvements over OmniQuant, especially in the 2-bit setting.\n- The authors provide a thorough ablation study on several aspects of the proposed objective.", "weaknesses": "- The addition of LoRA parameters should take some hit in latency. The authors should present the latency numbers, especially for W2A16 and W4A4.\n- Given that the paper focuses on two-bit quantization, comparison to several recent rotation-based methods such as FrameQuant [1] and QuIP [2] are omitted from the related works section and the results. The authors should compare against these methods for 2-bit quantization.\n- In practice, most quantization frameworks use sub channel quantization for weights, the authors should report weight-only quantization numbers with sub-channel quantization for all bit-widths.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "51kgvwcPmp", "reviewer": "Reviewer_mZis", "summary": "This work introduces Cross-Block Quantization (CBQ), a novel post-training quantization (PTQ) framework targeting large language models (LLMs). CBQ's core innovation is its cross-block dependency mechanism, which introduces dependencies across multiple blocks to mitigate error accumulation. Additionally, a coarse-to-fine preprocessing strategy and LoRA-Rounding are proposed to handle weight and activation outliers. The authors validate CBQ on various LLMs and demonstrate that it achieves state-of-the-art performance in low-bit settings, like W4A4, across a range of datasets.", "questions": "The paper introduces a coarse-to-fine preprocessing (CFP) strategy for handling weight and activation outliers, yet it is unclear how this approach compares in complexity and execution time against other established methods like OMSE or SmoothQuant under similar quantization settings. Could the authors clarify the computational trade-offs of CFP compared to these methods? Additionally, a broader comparison in terms of both quantization accuracy and processing efficiency would strengthen the evaluation of CFP's effectiveness.\n\nCan the authors elaborate on the expected quantization efficiency for models larger than 100 billion parameters?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Innovation: The idea of using cross-block dependencies to reduce quantization errors is both intuitive and impactful, as it tackles a recognized issue in PTQ for LLMs.\n\nExperimental Scope: CBQ is tested extensively across multiple LLM architectures and low-bit settings, showing clear performance gains over prior methods.\n\nSignificance: Reducing computational overhead for LLMs without retraining is valuable for real-world applications, and CBQ provides a viable solution for efficient model deployment.", "weaknesses": "Baseline Comparisons: Although the results on LLAMA models are promising, the exclusion of more recent PTQ methods like BiE[1] limits the impact of the results. Including such baselines could provide a more complete picture of CBQ’s effectiveness.\n\nBroader Applicability: The focus on language models limits the generalizability of CBQ. Expanding evaluations to other architectures, such as vision transformers and VLMs (vision language models), could strengthen the paper’s applicability.\n\n[1] BiE: Bi-Exponent Block Floating-Point for Large Language Models Quantization. ICML 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)