CBQ: Cross-Block Quantization for Large Language Models

## Abstract

Abstract Post-training quantization (PTQ) has played a key role in compressing large language models (LLMs) with ultra-low costs. However, existing PTQ methods only focus on handling the outliers within one layer or one block, which ignores the dependency of blocks and leads to severe performance degradation in low-bit settings. In this paper, we propose CBQ, a cross-block reconstruction-based PTQ method for LLMs. CBQ employs a cross-block dependency using a homologous reconstruction scheme, establishing long-range dependencies across multiple blocks to minimize error accumulation. Furthermore, CBQ incorporates a coarse-to-fine preprocessing (CFP) strategy for suppressing weight and activation outliers, coupled with an adaptive LoRA-Rounding technique for precise weight quantization. These innovations enable CBQ to not only handle extreme outliers effectively but also improve overall quantization accuracy.
Extensive experiments show that CBQ achieves superior low-bit quantization (W4A4, W4A8, W2A16) and outperforms existing state-of-the-art methods across various LLMs and datasets. Notably, CBQ quantizes the 4-bit LLAMA1-65B model within only 4.3 hours on a single GPU, achieving a commendable tradeoff between performance and quantization efficiency.

## 1 Introduction

Large language models (LLMs) (Wei et al., ; ; ; Brown et al., ), have sparked immense academic and industrial interest owing to their remarkable performance in handling complex natural languages tasks (Hendrycks et al., ; Bisk et al., ), like language generation, translation, question answering, and text summarization etc. However, the sheer size of these models, often comprising billion-level parameters, demands significant computational resources for inference and deployment. This problem necessitates the employment of model compression techniques such as network pruning (Han et al., ), knowledge distillation (Hinton et al., ), network quantization (Choi et al., ; Frantar et al., ), and neural architecture search (Zoph & Le, ) etc.
Compared with most compression methods that require huge training costs and access to complete training data, the post-training quantization (PTQ) technique  (Frantar et al., ; Lin et al., ; Xiao et al., ; Wei et al., ) operating with limited calibration data and computational resources is more in demand for compressing LLMs.

Previous PTQ methods (Wei et al., ; Xiao et al., ; Shao et al., ) for LLMs mainly focus on designing accurate outliers suppression schemes for floating-point activations, and then conduct quantization with vanilla policy. They locate the position and optimize the channel scale of activations with outliers in a layer-by-layer or block-by-block manner, as shown in Fig. 1. However, these methods lead to accumulated errors and sub-optimal quantization across the entire model for not taking the dependency across blocks into consideration. Moreover, these PTQ methods suffer from the absence of joint optimization for reducing quantization errors of weights and activations, resulting in significant performance degradation when compressing LLMs into low-bit settings, such as W2A16 and W4A4.

In this paper, we present CBQ, a comprehensive PTQ solution tailored for LLMs, which incorporates outlier suppression, learnable quantization step sizes, and adaptive rounding in a unified cross-block framework, as shown in Fig. 2. CBQ introduces a cross-block dependency (CBD) into block-wise reconstruction, helping mitigate the issue of accumulated errors across the entire model. Our approach ensures that multiple transformer blocks within a sliding window with overlapping are optimized simultaneously. This overlapping window mechanism, complemented by a novel homologous reconstruction scheme, allows for more effective and non-local optimization of quantization parameters. To reduce reconstruction difficulties posed by extreme weight and activation outliers, CBQ incorporates a novel coarse-to-fine preprocessing (CFP) strategy. CFP utilizes a quartile criterion to coarsely determine the range of outliers and then precisely pinpoint the locations of the outliers based on feature distributions. Then we truncate weight outliers and perform equivalent scaling on the activation outliers.
Furthermore, CBQ introduces a LoRA-Rounding technique, employing two low-rank matrices to learn the adaptive rounding values for quantized weight, which helps rectify weight quantization errors while preserving training efficiency. It is worth noting that the step sizes of weights and activations, and the rounding matrices are jointly optimized in our proposed quantization framework. CBQ’s innovative approach in cross-block reconstruction, advanced outlier processing, and adaptive rounding establish it as a groundbreaking solution in the field of PTQ for LLMs.

The contributions of this paper are summarized as follows:

- •
We propose a unified PTQ method tailored to LLMs based on cross-block reconstruction, which introduces a cross-block dependency mechanism to reduce accumulated errors in a block-wise manner.
- •
We design a coarse-to-fine pre-processing strategy to truncate weight outliers and equivalently scale activation outliers to reduce the reconstruction difficulty.
- •
We present LoRA-Rounding, an efficient adaptive rounding scheme that optimizes rounding errors of weight quantization via two low-rank matrices.
- •
Extensive experiments demonstrate that CBQ achieves low-bit quantization settings such as W4A4, W4A8, and W2A16, and outperforms state-of-the-art methods across various models and benchmark datasets.

## 2 Related works

#### Post-training quantization.

The post-training quantization (PTQ) algorithm (Nagel et al., ) converts the pre-trained full-precision network into a fixed-point network with a few unlabeled calibration data and computational overhead, which enables fast deployment on various devices. Recent post-training quantization methods have been widely explored in vision models (Liu et al., ; Hubara et al., ; Frantar & Alistarh, ; Cai et al., ).
Some techniques like AdaQuant (Hubara et al., ), AdaRound (Nagel et al., ), and BRECQ (Li et al., ) minimize the distance between floating point and quantized model outputs to optimize quantization parameters. While BRECQ incorporates Fisher information and jointly optimizes layers within each residual block, it still obtains sub-optimal performance for not capturing interactions across neighboring residual blocks. The proposed CBQ improves quantization accuracy that accounts for dependencies between adjacent blocks.

#### Quantization for large language models.

Existing large language models such as BLOOM (Laurençon et al., ), OPT (Zhang et al., ), and LLAMA () contain tens of billions of parameters, and require massive memory footprint and computation requirements in the inference. Recent works have been proposed to compress LLMs with post-training quantization methods that do not require a complete training procedure and access to a full training dataset. LLM.int8() (), ZeroQuant (Yao et al., ) and nuQmm (Park et al., ) focus on quantizing the parameters with mixed-precision decomposition scheme, representing the outliers with 16-bit and others with 8-bit. These methods can not truly accelerate the inference of LLMs for that is hard to implement on hardware. Other methods like GPTQ (Frantar et al., ) and AWQ (Lin et al., ) can efficiently quantize LLMs but they focus on FP16 activations and INT4 weights, which can not benefit from the integer matrix multiplication of existing AI accelerators. Additionally, Some methods like SmoothQuant (Xiao et al., ), Outlier Suppression (Wei et al., ), Outlier Suppression+ (Wei et al., ) and QLLM (Liu et al., ) aim at processing activation outliers and lack optimization for the weight quantization. Moreover, these methods rely on hand-craft quantization strategies which are tuned based on extensive experimentation for optimization. Recent block reconstruction-based PTQ method OmniQuant (Shao et al., ), suffers from error accumulation from the lack of cross-block dependency.
In contrast, CBQ incorporates outlier suppression, learnable step sizes, and adaptive rounding in a unified cross-block framework, which enables efficient and high-performance quantization for LLMs.

## 3 Methods

In this section, we introduce the proposed cross-block quantization framework tailored to LLMs. As illustrated in Fig. 2, CBQ firstly handles the outliers of weights and activations, and then jointly learns step sizes of weights and activations and weight-rounding matrices in a cross-block manner. CBQ reconstructs the output feature of the last block in each sliding window based on the corresponding supervision of the full-precision model.

We first give a brief introduction to quantization, which aims to represent weights and activations of float model with lower bit-width for reducing the memory and computational cost. Given floating-point weights $W$ and activations $X$, the $k$-bit quantized and de-quantized processing of weights and activations can be represented as

$$ $\displaystyle Q(X)=S_{X}\cdot\left\lfloor\frac{X}{S_{X}}\right\rceil;\ \ \ \ \ Q(W)=S_{W}\cdot(\left\lfloor\frac{W}{S_{W}}\right\rfloor+\Delta_{W}),$ (1) $$

where $Q_{\cdot}(\cdot)$ is an uniform quantizer, $S_{X}$ and $S_{W}$ are the quantization step sizes of activations and weights. $\left\lfloor\cdot\right\rceil$ and $\left\lfloor\cdot\right\rfloor$ represent rounding to the nearest integer and rounding towards negative infinity, respectively. $\Delta_{W}$ denotes the quantization rounding matrix for weight, where each element belongs to $\{0,1\}$. For simplicity, we omit the offset and clipping operation in Eq. 1.

Figure: Figure 3: Diagram of the homologous reconstruction scheme. $\mathcal{L}_{homo_rec}$ and $\mathcal{L}_{norm_rec}$ represent the homologous reconstruction loss and the normal reconstruction loss, respectively.

### 3.1 Cross-block reconstruction

Most existing post-training quantization methods (Xiao et al., ; Lin et al., ; Wei et al., ; Frantar et al., ) for LLMs conduct calibration in a layer-wise manner, optimizing the quantization step sizes of weights and inputs, respectively. OmniQuant (Shao et al., ) adopts a learnable method in a block-wise manner but ignores the issue of accumulated errors across the entire model. Besides, previous quantization methods get the rounding values of weights based on the policy of rounding to the nearest integer value, which is not the best for it not adapts to the data and the task loss (Nagel et al., ). In this paper, we propose to jointly optimize the step sizes and weight rounding values ($S_{X}$, $S_{W}$ and $\Delta_{W}$) in a unified cross-block framework. For a large language model with $K$ transformer blocks, the objective for joint optimization of the quantization parameters within $k^{th}$ block is formulated as

$$ $\mathop{\arg\min}\limits_{S^{k}_{X},S^{k}_{W},\Delta^{k}_{W}}\mathbb{E}(T_{k}( W^{k},X^{k}),T_{k}(Q(W^{k}),Q(X^{k})),$ (2) $$

where $1\leq k\leq K$, $T(\cdot)$ is a transformer block, $W^{k}$ and $X^{k}$ are weights and activations within the $k^{th}$ block, respectively. $S_{X}^{k}$ and $S_{W}^{k}$ are quantization step sizes of activation and weight quantization in the block, respectively, which are learnable during optimization. And $\mathbb{E}(\cdot)$ represents the metric to evaluate the reconstruction errors between outputs of quantized block and full-precision block.

Figure: Figure 4: Outliers pre-processing for weights and activations. The red dashed line indicates the truncation threshold for weight outliers, and the deep blue line represents the reserved subset. The light blue boxes depict activation outliers that undergo per-channel scaling.

#### Cross-block dependency.

While block-wise reconstruction is effective in saving computational memory, it only considers the local information within each block, neglecting the dependency across different blocks. The reconstruction error will accumulate as the stack of transformer blocks, incurring suboptimal solutions for quantization parameters To tackle this issue, we introduce a cross-block dependency (CBD) scheme using a sliding window approach. This scheme enables the simultaneous optimization of multiple blocks within the window. Furthermore, the two adjacent sliding windows have overlapping blocks, ensuring that the blocks between the windows are also interconnected. The CBD scheme enhances the connectivity and cooperation between blocks, enabling them to jointly contribute to the quantization process. This holistic optimization strategy leads to better overall performance and addresses the limitations of block-wise reconstruction in capturing cross-block dependencies.
We formulate the optimization with the CBD scheme as

$$ $\mathop{\arg\min}\limits_{S_{X}^{i,k},S_{W}^{i,k},\Delta_{W}^{i,k}}\mathbb{E}( T_{i,k}(W^{i,k},X^{i,k}),T_{i,k}(Q(W^{i,k}),Q(X^{i,k})),$ (3) $$

where $1\leq i\leq k\leq K$, $T_{i,k}$ represents the blocks from block $i$ to block $k$ within one sliding window, and the same applies to the symbols $S_{X}^{i,k}$, $S_{W}^{i,k}$ and $\Delta_{W}^{i,k}$.

Although the CBD technique helps reduce the reconstruction difficulty, it is important to note that it cannot completely eliminate the accumulated errors and the optimization within each window remains local. To address this limitation and further enhance the reconstruction process while ensuring stability, we introduce an additional homologous reconstruction scheme, as illustrated in Fig. 3.
In this scheme, we input the output of the previous block of the quantized model into the current block of the full-precision model. This generates an additional output that is homologous to the outputs of the quantized model. By minimizing the reconstruction errors between these two homologous outputs, CBD could help improve the overall accuracy and stability of the reconstruction process. Additionally, the optimization object is the average of the homologous reconstruction error $\mathcal{L}_{homo_rec}$ and the normal reconstruction error $\mathcal{L}_{norm_rec}$:

$$ $\displaystyle\mathcal{L}_{rec}=$ $\displaystyle\underbrace{\mathbb{E}(T_{i,k}(W^{i,k},X^{i,k}),T_{i,k}(Q(W^{i,k} ),Q(X^{i,k}))}_{\mathcal{L}_{norm_rec}}+$ (4) $\displaystyle\underbrace{\mathbb{E}(T_{i,k}(W^{i,k},Q(X^{i,k})),T_{i,k}(Q(W^{i ,k}),Q(X^{i,k}))}_{\mathcal{L}_{homo_rec}}.$ $$

For the distance metric, we incorporate $\mathcal{L}_{2}$ and Kullback-Leibler divergence (KLD) loss (Kullback & Leibler, ) to measure reconstruction error. KLD computes the likelihood distribution between output features that undergo the softmax function. It tends to suppress outliers in the feature space and enhance the robustness of the optimization process. By incorporating both terms, our method captures both the spatial distance and the distribution discrepancy, leading to a more comprehensive and robust optimization process. Then the distance metric is formulated as:

$$ $\displaystyle\mathbb{E}(h_{1},h_{2})=\left||h_{1}-h_{2}\right||_{2}+D_{KL}( \sigma(h_{1}),\sigma(h_{2})),$ (5) $$

where $h_{1}$ and $h_{2}$ are hidden states from the outputs of full-precision blocks and quantized blocks, respectively. $\sigma$ is the softmax function. $\left||\cdot\right||_{2}$ represents the $\mathcal{L}_{2}$ distance and $D_{KL}(\cdot)$ represents the KLD distance.
We provide the ablation study on the loss functions in Table 6.

Figure: Algorithm 1 Coarse-to-Fine Preprocessing

### 3.2 Coarse-to-fine pre-processing

**Table 1: Evaluation on multiple zero-shot datasets with the accuracy $\uparrow$ metric, where the Mutual dataset is evaluated with the Mean Reciprocal Rank/Recall$@1$/Recall$@2$ metrics. CBQ^∗ represents that the experiments conducted with 2-bit weight-only quantization did not involve fully quantizing the model but only the FC2 layera of the first and the last transformer blocks are converted to 4-bit precision.**
| Models | #Bits | Methods | PIQA | HellaSwag | ARC-C | ARC-E | Mutual | Ethics |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| OPT-30B | FP | - | 78.18 | 72.27 | 38.14 | 65.40 | 69.72 / 48.83 / 74.98 | 60.28 |
| W4A16 | GPTQ | 78.10 | 71.50 | 37.54 | 63.88 | 68.64 / 47.40 / 74.27 | 58.64 |  |
| OmniQuant | 78.06 | 71.29 | 37.98 | 65.19 | 69.34 / 48.64 /74.71 | 58.73 |  |  |
| CBQ | 78.36 | 72.23 | 38.06 | 65.35 | 69.77 / 49.32 / 74.47 | 61.31 |  |  |
| W2A16 | GPTQ | 66.38 | 52.55 | 28.41 | 43.86 | 64.50 / 41.08 / 68.62 | 52.15 |  |
| OmniQuant | 72.85 | 66.81 | 35.98 | 56.65 | 62.36 / 43.12 / 68.62 | 53.64 |  |  |
| CBQ | 76.19 | 66.90 | 36.23 | 59.72 | 68.20 / 47.29 / 72.23 | 52.10 |  |  |
|  | CBQ* | 78.29 | 71.18 | 36.95 | 64.01 | 69.49 / 48.76 / 75.06 | 60.05 |  |
| W4A8 | OmniQuant | 77.20 | 71.17 | 37.11 | 64.60 | 68.81 / 47.51 / 74.60 | 59.17 |  |
| RPTQ | 76.93 | 71.25 | 37.45 | 63.46 | 68.98 / 47.67 / 74.75 | 59.21 |  |  |
| CBQ | 78.26 | 71.55 | 37.89 | 64.92 | 69.01 / 47.72 / 74.81 | 59.23 |  |  |
| W4A4 | OmniQuant | 75.38 | 67.47 | 33.27 | 61.23 | 67.12 / 45.14 / 72.34 | 56.30 |  |
| CBQ | 75.89 | 67.49 | 34.81 | 61.58 | 67.73 / 45.94 / 73.14 | 56.60 |  |  |
| OPT-66B | FP | - | 79.81 | 74.86 | 40.01 | 67.26 | 69.84 / 48.87 / 74.94 | 58.14 |
| W4A16 | GPTQ | 79.32 | 73.15 | 38.95 | 65.45 | 69.10/ 48.46 / 74.26 | 54.90 |  |
| OmniQuant | 79.43 | 73.27 | 38.97 | 66.85 | 69.04 / 48.45 / 74.24 | 55.87 |  |  |
| CBQ | 79.71 | 74.69 | 39.18 | 67.38 | 69.50 / 48.65 / 74.83 | 57.35 |  |  |
| W2A16 | GPTQ | 54.24 | 52.55 | 23.04 | 32.28 | 60.45 / 35.56 / 61.74 | 49.50 |  |
| OmniQuant | 77.01 | 73.10 | 34.65 | 66.32 | 65.26 / 43.23 / 70.47 | 51.46 |  |  |
| CBQ | 78.05 | 73.45 | 35.37 | 66.84 | 67.34 / 45.31 / 72.45 | 55.95 |  |  |
|  | CBQ* | 79.21 | 74.32 | 38.96 | 67.11 | 69.32 / 48.35 / 74.69 | 56.78 |  |
| W4A8 | OmniQuant | 77.12 | 73.56 | 37.65 | 65.89 | 68.25 / 47.63 / 73.85 | 56.93 |  |
| RPTQ | 77.52 | 74.01 | 38.82 | 64.60 | 68.54 / 47.87 / 73.94 | 56.95 |  |  |
| CBQ | 79.12 | 74.21 | 39.25 | 67.16 | 69.07 / 48.32 / 74.53 | 56.98 |  |  |
| W4A4 | OmniQuant | 77.85 | 71.76 | 37.20 | 63.29 | 68.20 / 46.61 / 73.02 | 55.54 |  |
| CBQ | 78.01 | 72.34 | 37.56 | 63.78 | 68.76 / 47.20 / 73.56 | 55.82 |  |  |
| LLAMA1-30B | FP | - | 80.09 | 79.21 | 45.39 | 58.92 | 72.45 / 53.49 /78.21 | 57.42 |
| W4A16 | GPTQ | 79.62 | 78.81 | 44.54 | 58.42 | 72.30 / 52.93 / 77.44 | 56.30 |  |
| OmniQuant | 79.83 | 78.95 | 46.26 | 59.34 | 72.29 / 53.38 / 77.65 | 56.21 |  |  |
| CBQ | 80.12 | 79.11 | 46.65 | 59.89 | 72.85 / 53.95 / 78.56 | 57.85 |  |  |
| W2A16 | GPTQ | 51.03 | 26.34 | 26.02 | 28.87 | 56.53 / 29.80 / 58.13 | 52.72 |  |
| OmniQuant | 77.23 | 73.85 | 43.52 | 55.23 | 70.62 / 50.89 / 74.96 | 50.36 |  |  |
| CBQ | 77.23 | 75.05 | 42.93 | 57.12 | 69.96 / 49.93 /75.65 | 56.35 |  |  |
| CBQ^∗ | 80.09 | 78.85 | 45.05 | 58.42 | 72.74 / 53.95 /78.44 | 57.65 |  |  |
| W4A8 | OmniQuant | 78.95 | 76.34 | 44.62 | 57.36 | 71.03 / 52.89 / 77.06 | 57.05 |  |
| CBQ | 79.34 | 78.98 | 45.13 | 58.45 | 71.35 / 53.23 / 77.64 | 57.19 |  |  |
| W4A4 | OmniQuant | 71.21 | 64.65 | 34.47 | 49.45 | 67.10 / 45.37 / 71.44 | 47.69 |  |
| QLLM | 73.83 | 67.91 | 38.40 | 50.67 | - | - |  |  |
| CBQ | 76.33 | 72.74 | 42.92 | 54.50 | 70.12 / 50.45 / 74.73 | 48.70 |  |  |
| LLAMA1-65B | FP | - | 80.79 | 80.72 | 46.24 | 58.71 | 73.03 / 54.17 / 79.12 | 61.75 |
| W4A16 | GPTQ | 80.79 | 79.86 | 45.45 | 58.13 | 72.89 / 53.84 / 78.57 | 58.45 |  |
| OmniQuant | 81.01 | 80.30 | 45.74 | 58.41 | 72.99 / 54.06 / 79.11 | 60.12 |  |  |
| CBQ | 81.12 | 80.76 | 45.98 | 58.64 | 73.06 / 54.29 / 78.89 | 61.49 |  |  |
| W2A16 | GPTQ | 56.47 | 33.31 | 25.43 | 31.69 | 59.28 / 33.86 / 60.49 | 50.93 |  |
| OmniQuant | 79.50 | 72.38 | 40.35 | 52.56 | 69.50 / 48.64 / 74.94 | 52.64 |  |  |
| CBQ | 78.12 | 74.28 | 41.64 | 55.35 | 70.67 / 50.80 / 75.51 | 55.95 |  |  |
|  | CBQ* | 81.07 | 80.51 | 45.81 | 57.45 | 73.43 / 54.96 / 79.23 | 61.35 |  |
| W4A8 | OmniQuant | 79.21 | 78.96 | 44.63 | 57.68 | 72.24 / 53.89 / 78.65 | 59.68 |  |
| CBQ | 79.95 | 79.30 | 45.43 | 58.12 | 72.83 / 54.27 / 79.02 | 61.25 |  |  |
| W4A4 | OmniQuant | 71.81 | 66.81 | 35.92 | 48.02 | 68.49 / 47.29 / 73.70 | 57.19 |  |
| QLLM | 73.56 | 70.94 | 39.68 | 52.06 | - | - |  |  |
|  | CBQ | 77.69 | 76.65 | 43.25 | 56.01 | 70.93 / 51.35 / 75.62 | 57.50 |  |

**Table 2: Evaluation quantization on generation datasets with the perplexity (PPL) $\downarrow$ metric, where OmniQ represents OmniQuant.**
| #Bits | Methods | OPT-30B | OPT-66B | LLAMA1-30B | LLAMA1-65B |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| C4 | Wiki | C4 | Wiki | C4 | Wiki | C4 | Wiki |  |  |
| FP | - | 10.69 | 9.56 | 10.28 | 9.34 | 5.98 | 4.10 | 5.62 | 3.53 |
| W4A16 | GPTQ | 10.80 | 9.63 | 10.50 | 9.55 | 6.16 | 4.34 | 5.77 | 3.77 |
| OmniQ | 10.80 | 9.71 | 10.63 | 9.37 | 6.06 | 4.19 | 5.68 | 3.62 |  |
| CBQ | 10.73 | 9.65 | 10.31 | 9.41 | 6.03 | 4.14 | 5.62 | 3.59 |  |
| W2A16 | GPTQ | 1.6e4 | 9.1e3 | 4.3e3 | 6.3e3 | 7.2e3 | 1.3e4 | 8.8e3 | 1.1e4 |
| OmniQ | 12.80 | 11.00 | 12.13 | 10.59 | 9.02 | 7.14 | 7.78 | 6.01 |  |
| CBQ | 12.01 | 10.51 | 11.19 | 10.25 | 7.65 | 5.58 | 7.42 | 5.25 |  |
| CBQ* | 10.92 | 10.26 | 10.39 | 9.48 | 6.02 | 4.21 | 5.73 | 3.73 |  |
| W4A8 | OmniQ | 10.96 | 9.95 | 10.73 | 9.52 | 6.45 | 4.58 | 6.12 | 3.96 |
| RPTQ | 11.01 | 10.22 | 10.57 | 9.46 | - | - | - | - |  |
|  | CBQ | 10.86 | 9.83 | 10.42 | 9.44 | 6.25 | 4.32 | 5.96 | 3.84 |
| W4A4 | OmniQ | 11.89 | 10.60 | 11.35 | 10.29 | 12.49 | 10.33 | 11.28 | 9.17 |
| QLLM | - | - | - | - | 11.51 | 8.37 | 8.89 | 6.87 |  |
| CBQ | 11.79 | 10.34 | 11.02 | 9.45 | 9.73 | 7.96 | 7.52 | 5.89 |  |

To address the challenges in achieving optimal quantization parameters before cross-block reconstruction, we have conducted a thorough analysis of outliers in both weights and activations. Based on this analysis, we have developed an effective and unified design approach to handle extreme weight and activation outliers.

Figure 4 illustrates the block-by-block visualization of the weight distribution of the last fully connected layer, as well as the channel-by-channel visualization of the activation distribution within one block. It is observed that outliers in both weights and activations contribute to larger reconstruction errors between the quantized and full-precision models. To address this issue, we analyze the distribution of outliers and propose a coarse-to-fine pre-processing strategy to suppress outliers and reduce the difficulty of reconstruction.

#### Coarse-grained detection.

The comprehensive algorithm of the outlier detection is illustrated in Algorithm 1, which is divided into two stages. In the first stage, we perform coarse-grained detection by calculating the lower and upper quartile values ($Q_{1}$ and $Q_{3}$) and the interquartile range ($IQR$) in the numerical distribution (either activations or weights). Based on these calculations, we obtain a coarse outlier set $O=\{x|x>T,x\in X\}$, where $T=Q_{3}+\lambda_{1}IQR$ and $\lambda_{1}$ is set to 1.5. This stage greatly reduces the search space for outlier detection.

#### Fine-grained detection.

In the second stage, we perform fine-grained detection by searching for a threshold that splits the coarse outlier set into an outlier subset $O_{outlier}$ and a reserved subset $O_{reserved}$. The goal is to minimize the intra-set variance $M_{intra}=\text{Var}(O_{reserved})$ while maximizing the distance between the two subsets $M_{inter}=(\textrm{Min}(O_{outlier})-\textrm{Max}(O_{reserved}))^{2}$. To balance these objectives, we define a metric $M=M_{inter}-\lambda_{2}M_{intra}$, where $\lambda_{2}=1.0$. By minimizing this metric, we can effectively identify outliers and distinguish them from the remaining data.

Removing outliers in weights has minimal impact on performance, whereas outliers in activations, particularly in specific channels, can greatly affect performance if directly removed. Consequently, our approach involves truncating weight outliers and scaling outliers in activations based on the detected outliers in both weights and activations. This is different from existing outlier pre-processing methods (Xiao et al., ; Wei et al., ) that focus solely on activations.
Figure 4 provides visual evidence of weight outliers being truncated within the outlier group. Additionally, the figure demonstrates the presence of outliers in specific channels. Scaling these specific channels results in a significant concentration of the overall distribution.

The scaling factor $s_{i}$ for the activation tensor in $i^{th}$ channel (represented as $X_{i}$) is determined by the maximum absolute value of the truncated outlier set $O^{*}$:

$$ $\displaystyle s_{i}=\sqrt{\textrm{Max}(|X_{i}|)/\textrm{Max}(O^{*})}.$ (6) $$

This scaling factor is then applied to update weights and activations following prior work (Wei et al., ) to counteract destabilizing fluctuations from remaining outliers.

### 3.3 LoRA-Rounding for weight quantization

The performance degradation raised by weight quantization is non-negligible when compressing the model into low bit-width. The quantization error of weights comes from the rounding error (Nagel et al., ) and clipping error (Tu et al., ), but previous PTQ methods for LLMs only focus on the optimization of the latter one and do not take the rounding error into consideration. AdaRound (Nagel et al., ) introduces to learn a better weight-rounding matrix for post-training
quantization that adapts to the data and the task loss. As shown in Eq. 7, we can obtain the weight-rounding matrix $\Delta_{W}\in\mathbb{R}^{d\times k}$ with a learnable matrix $V\in\mathbb{R}^{d\times k}$ with a rectified sigmoid function:

$$ $\displaystyle\Delta_{W}=\textrm{Clip}(\textrm{Sigmoid}(V)(\zeta-\gamma)+\gamma ,0,1),$ (7) $$

where $\zeta$ and $\gamma$ are stretch parameters and are fixed to 1.1 and -0.1, and $\textrm{Clip}(\cdot)$ clamps the inputs into a given range. The size of the weight-rounding matrix $\Delta_{W}$ is the same as the original weights.

However, the technique of AdaRound is hard to apply directly to LLMs with billion-level parameters, which could lead to much computational overhead and significantly reduce the efficiency of post-training quantization. To further rectify weight quantization errors arising from rounding operations and prevent their accumulation in adjacent blocks, we introduce an efficient adaptive rounding method for LLMs, called LoRA-Rounding. Specifically, inspired by LoRA (Hu et al., ), we employ low-rank adaptive learning on the weight-rounding matrices, decomposing $V$ with much smaller low-rank matrices, and only optimize them in post-training quantization, the decomposition is defined as:

$$ $V=A_{1}\times A_{2},A_{1}\in\mathbb{R}^{d\times r},A_{2}\in\mathbb{R}^{r\times k},$ (8) $$

where the rank $r<<\textrm{min}(d,k)$. Following the setting of LoRA (Hu et al., ), we utilize a random Gaussian initialization for $A_{1}$ and zero for $A_{2}$, thus $\Delta_{W}$ is set to zero at the beginning of post-training quantization. During training, each element of $V$ is encouraged into 0 or 1 with a regularizer loss:

$$ $\mathcal{L}_{round}=\sum_{i,j}1-|2\Delta_{W}{(i,j)}-1|^{\beta},$ (9) $$

where $\beta$ is a annealing factor. Following AdaRound (Nagel et al., ), $\beta$ is set to higher in the initial phase and set to lower in the later phase of the optimization to encourage it to converge to 0 or 1. We also conduct $\Delta_{W}=\left\lfloor\Delta_{W}\right\rceil$ in the later phase of the optimization to force each element into $\{0,1\}$ exactly.

Compared with vanilla AdaRound for LLMs. The proposed LoRA-Rounding reduces the number of learnable parameters from $d\times k$ to $(d+k)\times r$ and accelerates the optimization process significantly, we conduct ablation experiments in the next section.

Compared with QLoRA (Dettmers et al., ). QLoRA is an efficient parameter finetuning method for quantized LLMs, freezes the quantized weight and optimizes the float low-rank matrices, which is much similar to the original LoRA but takes lower memory. However, LoRA-Rounding only optimizes the rounding error of weight quantization, which is a much different method that is designed for post-training quantization with ultra-low cost.

### 3.4 Overall loss

In summary, by utilizing a low-rank decomposition of the weight-compensated matrix, we effectively reduce the number of learnable parameters $S_{W},S_{X},A_{1},A_{2}$ while still achieving accurate correction of weight quantization errors within each block.
The total loss for optimizing the $i^{th}$ block to the $k^{th}$ block within a sliding window is formulated as

$$ $\mathcal{L}_{total}=\mathcal{L}_{rec}+\gamma\mathcal{L}_{round},$ (10) $$

where the $\gamma$ is the hyper-parameter to balance the reconstruction error and round error, which is set to 0.01 in our experiments. During optimization, We slide the window to the last block with an interval and update all the quantization parameters within the window, ensuring that the cumulative error in the quantized transformer block is minimized and achieves optimal performance.

## 4 Experiments

In this section, we conduct extensive experiments to validate the effectiveness and superiority of the proposed CBQ.

### 4.1 Setup

#### Models and datasets.

We conduct experiments on large language models with different sizes, including OPT (Zhang et al., ) and LLAMA () families. We validate our quantization scheme on various datasets which are divided into two categories. One is reported by the perplexity metric of language generation experiments on C4 (Raffel et al., ) and WikiText2 (Merity et al., ). The other is reported by the accuracy metric of zero-shot language tasks on PIQA (Bisk et al., ), HellaSwag (Clark et al., ), ARC (Clark et al., ), Mutual (Cui et al., ) and Ehics (Hendrycks et al., ).

#### Quantization setting.

To thoroughly evaluate performance, we test extensive quantization schemes including weight-only quantization down to W4A16 and W2A16, as well as joint weight-activation quantization for ultra-low bitwidths like W4A8, and W4A4. This extensive assessment across varying bitwidths provides a robust analysis of our proposed method. Also, with full consideration of the convenience of actual deployment of the quantized model, except for the settings with activation values below 4 bits that use per-token quantization, we perform per-tensor quantization on activations.

#### Baseline methods.

For weight-only quantization settings, we selected GPTQ (Frantar et al., ) as the baseline quantization method in our experiments. This represents the most prevalent technique for W4A16 quantization of language models, respectively.
Furthermore, we compare our CBQ with OmniQuant (Shao et al., ) and QLLM (Liu et al., ), which is the state-of-the-art method based on block reconstruction. We include a comparison of our CBQ method with the groupwise quantization method RPTQ (Yuan et al., ), which is widely employed in the W4A8 setting.

#### Implementation details.

Following the setting of previous work (Frantar et al., ), our calibration dataset comprises 128 randomly selected 2048-token segments from C4 to ensure standardized benchmarking. To balance quantization performance and training speed, we utilize sliding windows containing two blocks with 3 epochs per window.
For the LoRA-Rounding technique, we set the rank $r$ to 5. The optimization process involves adjusting the learnable quantization step sizes ($S_{X}$ and $S_{W}$) and the weight-rounding matrix ($\delta_{W}$) with learning rates of $1e-6$, $1e-7$, and $1e-4$, respectively. To manage the learning rate, we utilize the CosineAnnealingLR scheduler. We quantize all models using a mini-batch size of 1 on a single GPU. This configuration allows efficient cross-block dependency modeling while sufficiently propagating information across windows.

### 4.2 Experimental results

#### Evaluation on zero-shot datasets with accuracy.

Results on multiple zero-shot benchmarks using accuracy as the evaluation metric demonstrate CBQ’s capabilities on large models including OPT (30B, 66B) and LLAMA (30B, 65B) (as shown in Table 1). Across almost all public datasets, CBQ outperforms existing quantization methods by over 2% and reduces the accuracy gap with the full precision model to within 1% under the W4A16, W2A16 and W4A8 quantization settings. This demonstrates stronger zero-shot capability. Moreover, unlike current techniques, CBQ uniquely achieves ultra-low quantization down to W4A4 bitwidths while maintaining a higher performance than the state-of-the-arts. The consistent gains verify the generalization of CBQ’s innovations across models and datasets.

#### Evaluation on generation datasets with perplexity.

Results in Table 2 demonstrate our method’s generation performance on C4, WikiText2 using weight-only quantized OPT and LLAMA models. Focusing on ultra-low bitwidths, we achieve over 1% higher perplexity versus GPTQ at W4A16. These consistent improvements at low bitwidths highlight our advantages in preserving generative quality under aggressive compression rates.

**Table 3: Ablation of the cross-block dependency (CBD).**
| #Num of blocks | w/ Overlap | C4$\downarrow$ | Wiki $\downarrow$ |
| --- | --- | --- | --- |
| 1 | ✗ | 14.57 | 11.98 |
| 2 | ✗ | 14.23 | 11.35 |
| ✓ | 13.29 | 10.63 |  |
| 4 | ✗ | 14.32 | 11.45 |
| ✓ | 13.27 | 10.60 |  |

**Table 4: Ablation on the coarse-to-fine pre-processing. ‘CBQ-Recon.’ represents the proposed cross block reconstruction.**
| Method | C4 $\downarrow$ | Wiki $\downarrow$ |
| --- | --- | --- |
| w/o outlier pre-processing | 1082.68 | 1128.33 |
| w/ Smoothquant (Xiao et al., ) | 33.21 | 25.26 |
| w/ CFP-Activation | 23.48 | 19.75 |
| w/ CFP-Weight + CFP-Activation | 21.98 | 17.95 |
| w/ Smoothquant + CBQ-Recon. | 15.69 | 12.24 |
| w/ CFP-Weight + CFP-Activation + CBQ-Recon. | 13.29 | 10.63 |

#### Comparison of block reconstruction-based methods.

We provide a comprehensive validation of the superiority of our proposed CBQ method compared to the state-of-the-art block reconstruction-based methods, OmniQuant and QLLM. To evaluate the performance improvement, we conducted experiments that encompassed multiple zero-shot tasks and generation tasks. The results, as presented in Table 1 and Table 2, consistently demonstrate the significant performance gains achieved by our CBQ method.

### 4.3 Ablation Study

To analyze the contribution of each component in our proposed CBQ method, we performed ablation experiments on the LLAMA-7B model.

#### Cross-block dependency.

To analyze the impact of our proposed cross-block dependency (CBD) method, we performed ablation experiments in Table 3. Results demonstrate performance gains as the number of blocks jointly processed per sliding window increases, validating CBD’s ability to model inter-block dependencies. Furthermore, utilizing overlapping blocks between adjacent sliding windows supplements cross-window relational representation. This redundancy helps capture nuanced block interactions and enables additional accuracy improvements. Overall, these ablation studies highlight the benefits of CBD for progressively accumulating and propagating cross-block knowledge during CBQ optimization.

#### LoRA-Rounding.

As shown in Table 5, the incorporation of the LoRA-Rounding technique into the learning process effectively mitigates the rounding errors that typically arise from weight quantization. This improvement leads to a significant enhancement in quantization accuracy. In comparison to the conventional ’Rounding’ technique, our LoRA-Rounding approach utilizes low-rank decomposition to reduce the number of learnable parameters. This reduction not only reduces GPU memory consumption during training but also improves training speed. Despite the additional training costs, LoRA-Rounding significantly outperforms ’Rounding’ in terms of performance.

**Table 5: Ablation of the LoRA-Rounding.**
| Method | PPL $\downarrow$ |  |  |  |
| --- | --- | --- | --- | --- |
| C4 | Wiki | #Epochs | GPU (GB) |  |
| w/o Rounding | 14.32 | 11.46 | 3 | 18.83 |
| w/ Rounding | 13.86 | 10.98 | 3 | 27.73 |
| w/ Rounding | 13.58 | 10.72 | 6 | 27.73 |
| w/ LoRA-Rounding | 13.29 | 10.63 | 3 | 21.01 |

#### Coarse-to-fine pre-processing.

Ablation of the proposed coarse-to-fine pre-processing verifies performance improvements from suppressing outlier values before quantization, as shown in Table 4. Detecting and dynamic scaling extreme activation outliers reduces distortions, while additional truncating of weight outliers further stabilizes training. This tailored preprocessing stabilizes the data distribution for robust optimization at ultra-low bitwidths. It is important to note that this two-pronged pre-processing method is specifically designed to address the abundance of outliers in language models, which are not adequately addressed by the existing pre-processing technique of Smoothquant (Xiao et al., ). Furthermore, we conduct a reconstruction optimization process, referred to as ’CBQ-Recon.’, based on the pre-processed weights and activations. This approach, compared to the optimization based on the processing of Smoothquant, yields better performance.

#### Loss functions.

**Table 6: Ablation study on block-wise reconstruction loss functions. ‘homologous’ represents the homologous reconstruction.**
| KL loss | L2 loss | Homologous | C4 | Wiki |
| --- | --- | --- | --- | --- |
| ✗ | ✓ | ✗ | 13.82 | 11.13 |
| ✓ | ✗ | ✗ | 13.84 | 11.12 |
| ✓ | ✓ | ✗ | 13.55 | 10.88 |
| ✓ | ✓ | ✓ | 13.29 | 10.63 |

To determine optimal loss formulations, we evaluate reconstruction errors using L2 alone, KLD alone, and a combination of them in Table 6. Ablation results demonstrate superior performance from KLD over L2, with the combined loss achieving further gains. This highlights the benefits of KLD for matching full-precision block distributions during CBQ optimization. Fusing both losses enables jointly minimizing absolute and divergence-based errors to improve overall block-wise reconstruction. Our analysis verifies the advantage of blended L2+KLD loss for robustly optimizing blocks as interdependent components.
Moreover, the introduced homologous reconstruction scheme further reduces the reconstruction difficulty and improves the quantization performance.

#### Comparison of quantization efficiency.

**Table 7: Comparison of training (GPU Hours) time of our CBQ with OmniQuant. * represents that the training time of QLLM is implemented on LLAMA2 models.**
| LLAMA | 7B | 13B | 30B | 65B |
| --- | --- | --- | --- | --- |
| OmniQuant | 1.6h | 3.3h | 7.3h | 14.4h |
| QLLM* | 1.05 | 1.79 | - | 9.05 |
| CBQ | 0.9h | 1.45 | 2.1h | 4.3h |

We evaluate the quantization efficiency of our CBQ method and compare it to OmniQuant and QLLM, both of which are reconstruction-based PTQ methods. The GPU training hours for both methods are shown in Table 7. The results demonstrate that the training cost of CBQ can be faster than OmniQuant, particularly for larger models. This indicates that our CBQ method offers an advantage in terms of training efficiency.

## 5 Conclusion

In this study, we introduce CBQ, a unified post-training quantization method specifically designed for large language models. By jointly optimizing weight and activation quantization parameters through block-wise reconstruction, CBQ achieves efficient quantization without hand-craft searching. The introduced cross-block dependency scheme, low-rank decomposition scheme, and outlier detection strategy further improve the quantization performance. Experimental results demonstrate the state-of-the-art performance of CBQ, particularly in ultra-low bit settings like W4A4 and W2A16, and better maintain the capability of the full-precision model in W4A8 and W4A16 settings. The proposed CBQ method contributes to advancing PTQ techniques for LLMs, paving the way for efficient and effective deployment of large language models in resource-constrained devices.

## 6 Acknowledgement

We gratefully acknowledge the support of MindSpore, CANN (Compute Architecture for Neural Networks) and Ascend AI Processor used for this research.

## Appendix A Appendix

### A.1 Overview

**Table 8: Comparison of different quantization methods for LLMs.**
| Method | Quantize W/A | Gradient-Based | Cross-Block Dependency | Weight Outlier | Activation Outlier | Rounding Error |
| --- | --- | --- | --- | --- | --- | --- |
| GPTQ (Frantar et al., ) | ✓/ ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| RPTQ (Yuan et al., ) | ✓/ ✓ | ✗ | ✗ | ✗ | ✓ | ✗ |
| OS+ (Wei et al., ) | ✓/ ✓ | ✗ | ✗ | ✗ | ✓ | ✗ |
| SmoothQuant (Xiao et al., ) | ✓/ ✓ | ✗ | ✗ | ✗ | ✓ | ✗ |
| OmniQuant (Shao et al., ) | ✓/ ✓ | ✓ | ✗ | ✓ | ✓ | ✗ |
| QLLM (Liu et al., ) | ✓/ ✓ | ✓ | ✗ | ✗ | ✓ | ✗ |
| CBQ (Ours) | ✓/ ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |

In Table 8, we compare the designed components of our CBQ with the existing quantization methods for LLMs. We can observe a comparison of the different components incorporated in various quantization methods LLMs. Our proposed CBQ method stands out by including multiple essential components to address the challenges associated with LLM quantization.

Firstly, CBQ ensures that both weight and activation values are quantized to improve computational efficiency and reduce memory requirements. This aligns with the requirements of other methods such as RPTQ, OS+, and SmoothQuant, and is different from GPTQ.
Additionally, CBQ incorporates a gradient-based optimization approach, allowing for efficient optimization during the quantization process. This component is also present in OmniQuant and QLLM, signifying its significance in achieving accurate quantization results.
Furthermore, CBQ introduces the cross-block dependency (CBD) component, enabling the modeling of long-range dependencies between adjacent blocks. This ensures better information flow and integration across multiple blocks, surpassing the capabilities of other methods such as OmniQuant and QLLM.
Moreover, CBQ addresses the presence of weight and activation outliers, which can significantly impact the quantization process. By effectively handling these outliers, CBQ surpasses the capabilities of OS+, SmoothQuant, OmniQuant, and QLLM, which either do not consider or only partially address this issue.
Lastly, CBQ accounts for rounding errors, a critical aspect of quantization. By incorporating a rounding error reduction scheme, CBQ ensures more accurate and reliable quantization results. This component is absent in all other compared methods.

In summary, our CBQ method outperforms existing quantization methods for LLMs by incorporating a comprehensive set of components that collectively address the challenges associated with LLM quantization. These components work synergistically to enhance the precision, accuracy, and efficiency of the quantization process, making CBQ a promising approach for LLM quantization.

### A.2 Evaluation quantization for a series of OPT models

**Table 9: Evaluation quantization for a series of OPT models on generation datasets with the perplexity $\downarrow$ metric**
|  | #Bits | Methods | OPT-1.3B | OPT-2.7B | OPT-6.7B | OPT-13B |
| --- | --- | --- | --- | --- | --- | --- |
| C4 | FP | - | 14.72 | 13.16 | 11.74 | 11.20 |
| W4A16 | GPTQ | 15.57 | 13.75 | 12.15 | 11.36 |  |
| CBQ | 15.42 | 13.56 | 11.92 | 11.29 |  |  |
| W2A16 | OmniQuant | 27.33 | 19.16 | 15.44 | 14.16 |  |
| CBQ | 15.99 | 13.83 | 12.19 | 11.52 |  |  |
| Wikitext2 | FP | - | 14.62 | 12.47 | 10.86 | 10.12 |
| W4A16 | GPTQ | 15.56 | 12.82 | 11.41 | 10.31 |  |
| CBQ | 15.10 | 13.58 | 11.10 | 10.24 |  |  |
| W2A16 | OmniQuant | 23.95 | 18.13 | 14.43 | 12.94 |  |
| CBQ | 15.40 | 17.92 | 11.19 | 10.43 |  |  |

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Will the code be released?

2. LoRA-Rounding needs clearer explanation. Is *V* the rounding mask? On what type of matrix is it applied, and does each matrix in the model have its own *V*?

3. Has CBQ been tested for improvements in few-shot scenarios?

4. Figure 1 is intriguing; does this phenomenon appear in other models, like LLAMA3, Mistral, or Falcon?
soundness: 3
strengths: 1. The paper offers an interesting insight: full model quantization introduces inter-layer correlations.

2. Experimental results on LLAMA1, LLAMA2, and OPT demonstrate substantial improvements in W2 and W4 settings, alongside a significant reduction in PTQ scenario.
summary: The paper introduces Cross-Block Quantization (CBQ), a novel post-training quantization (PTQ) technique developed for large language models (LLMs). CBQ tackles critical inter- and intra-layer dependencies that compromise quantization accuracy at ultra-low bit settings, establishing long-range dependencies through cross-block reconstruction and managing intra-layer dependencies via adaptive LoRA-Rounding. The approach also incorporates a coarse-to-fine preprocessing method to optimize handling of weights and activations.
weaknesses: 1. The writing could be improved, especially regarding clarity in notation. While the concepts are interesting, it is difficult to follow due to unclear notation. For example, symbols like \(i\), \(j\), and \(K\) in Equation 3, as well as \(V\) in Equation 11, are not clearly defined when first introduced. It would help if each symbol had an explicit definition upon first use. Additionally, the term "scales" in the phrase "comparisons of the scales between adjacent layers..." lacks clarity. Specifying what "scales" refers to in the context of adjacent layers would enhance readability.

2. The range of models tested is limited. While the authors include results on LLAMA1, LLAMA2, and OPT, these are relatively dated models, and results on newer models like LLAMA3, Mistral, and Falcon should be included.

3. While the authors suggest that LoRA-Rounding was introduced to reduce computation, they do not evaluate this aspect in their experiments. To make this claim more compelling, the authors could include specific metrics or experiments comparing the computational efficiency of LoRA-Rounding to alternative approaches.

4. The authors should compare their method with other W4 approaches, such as AWQ. Comparing with AWQ would be valuable because of its relevant strengths or similarities, which could provide a more comprehensive evaluation of the method’s performance in a competitive context.

---

confidence: 5
contribution: 4
initial_rating: 10
presentation: 4
questions: I have read this paper multiple times before, and I fully agree with the authors' approach. I have no further questions.
soundness: 4
strengths: I have carefully read this work, I think the result of this work was convincing, and the approach was solid.
summary: This paper introduced CBQ  post-training quantization PTQ technique designed for compressing large language models under low-bit precision. The authors identified the primary challenge in ultra-low-bit quantization: the dependencies within and between layers that amplify quantization errors, especially as model size and parameter counts increase. To address this, CBQ incorporates a cross-block reconstruction strategy, which leverages both intra-layer and inter-layer dependencies by optimizing multiple transformer blocks within a sliding window approach. Additionally, the method employs a LoRA-Rounding technique to manage intra-layer dependencies and reduce computational costs, while an adaptive coarse-to-fine preprocessing strategy effectively handles outliers in weights and activations.
weaknesses: N/A

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. What would be the impact of reducing the overlap size in cross-block dependencies? Could reducing overlap compromise quantization efficiency, or are there scenarios where this would be advisable?
2. Could the authors clarify if the coarse-to-fine preprocessing provides unique benefits over standard outlier suppression methods? A comparative analysis would help isolate its specific contribution.
3. How do the authors envision CBQ scaling with models that incorporate multimodal inputs or models beyond text-based LLMs? Would additional adaptations be required?
soundness: 3
strengths: 1. By focusing on inter-block dependencies, CBQ takes a proactive approach to minimize error accumulation, which is often a critical challenge in low-bit quantization. This dependency handling shows clear improvements in model accuracy.
2. The design of CBQ, particularly the coarse-to-fine outlier preprocessing and adaptive rounding, ensures flexibility across different LLM sizes, making it a practical choice for varied deployment needs.
3. The authors have included a wide range of experiments showing CBQ’s effectiveness in diverse quantization configurations, including settings that emphasize computational efficiency, which makes it relevant for both research and applied contexts.
summary: This paper proposes Cross-Block Quantization (CBQ), a novel method designed to optimize post-training quantization (PTQ) for large language models (LLMs) by addressing inter-block dependencies and introducing a coarse-to-fine outlier handling strategy. CBQ's approach aims to preserve model accuracy at low-bit configurations by leveraging cross-block reconstructions and refining outlier quantization through LoRA-Rounding. The authors present experimental results demonstrating CBQ’s strong performance across different quantization bit widths and various LLM architectures, with promising efficiency for real-world deployment.
weaknesses: 1. While the paper demonstrates that overlapping blocks in CBQ contributes to performance, it lacks an in-depth analysis of how varying the overlap size impacts memory efficiency, latency, and overall quantization stability. Providing such details would clarify practical deployment considerations.
2. The coarse-to-fine preprocessing for outliers appears effective; however, an assessment of its necessity relative to simpler methods could be useful. It is unclear whether this specific strategy is essential or if comparable results could be achieved with simpler preprocessing.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - Majority of the experiments conducted in the paper are on the OPT and Llama-1 family of LLMs. It would be interesting to see some of the results presented in the paper on SOTA open-source LLMs like Llama-3 [3] and Gemma-2 [4]. 
- How exactly is W6A6 implemented, could the authors briefly discuss about its latency benefits by providing some numbers?

I am open to discussions and willing to reconsider my score.

[1] Adepu, Harshavardhan et al. “FrameQuant: Flexible Low-Bit Quantization for Transformers.” ICML 2024.

[2] Chee, Jerry et al. “QuIP: 2-Bit Quantization of Large Language Models With Guarantees.” NeurIPS 2023.

[3] Dubey, Abhimanyu et al. “The Llama 3 Herd of Models.” ArXiv abs/2407.21783 (2024): n. pag.

[4] Riviere, Gemma Team Morgane et al. “Gemma 2: Improving Open Language Models at a Practical Size.” ArXiv abs/2408.00118 (2024): n. pag.
soundness: 3
strengths: - The paper clearly motivates the proposed cross-block quantization.
- CBQ shows substantial performance improvements over OmniQuant, especially in the 2-bit setting.
- The authors provide a thorough ablation study on several aspects of the proposed objective.
summary: The authors find that optimizing for inter-layer dependencies is crucial, especially for lower-bit quantization. To address this challenge, they propose the use of cross-block loss where the loss is taken over a window of blocks and not just a single block. For the cross-block loss, in addition to using the L2-distance, the authors use KL-divergence between the activations. Inspired by AdaRound, the authors use LoRA parameters for accurate rounding. For outlier detection, the authors present a two step process. They use a quantile-based thresholding to identify an initial set of outliers following which they divide these coarse outliers into two sets by maximizing the inter-set variance and minimizing the intra-set variance.
weaknesses: - The addition of LoRA parameters should take some hit in latency. The authors should present the latency numbers, especially for W2A16 and W4A4.
- Given that the paper focuses on two-bit quantization, comparison to several recent rotation-based methods such as FrameQuant [1] and QuIP [2] are omitted from the related works section and the results. The authors should compare against these methods for 2-bit quantization.
- In practice, most quantization frameworks use sub channel quantization for weights, the authors should report weight-only quantization numbers with sub-channel quantization for all bit-widths.

---

confidence: 5
contribution: 2
initial_rating: 6
presentation: 3
questions: The paper introduces a coarse-to-fine preprocessing (CFP) strategy for handling weight and activation outliers, yet it is unclear how this approach compares in complexity and execution time against other established methods like OMSE or SmoothQuant under similar quantization settings. Could the authors clarify the computational trade-offs of CFP compared to these methods? Additionally, a broader comparison in terms of both quantization accuracy and processing efficiency would strengthen the evaluation of CFP's effectiveness.

Can the authors elaborate on the expected quantization efficiency for models larger than 100 billion parameters?
soundness: 3
strengths: Innovation: The idea of using cross-block dependencies to reduce quantization errors is both intuitive and impactful, as it tackles a recognized issue in PTQ for LLMs.

Experimental Scope: CBQ is tested extensively across multiple LLM architectures and low-bit settings, showing clear performance gains over prior methods.

Significance: Reducing computational overhead for LLMs without retraining is valuable for real-world applications, and CBQ provides a viable solution for efficient model deployment.
summary: This work introduces Cross-Block Quantization (CBQ), a novel post-training quantization (PTQ) framework targeting large language models (LLMs). CBQ's core innovation is its cross-block dependency mechanism, which introduces dependencies across multiple blocks to mitigate error accumulation. Additionally, a coarse-to-fine preprocessing strategy and LoRA-Rounding are proposed to handle weight and activation outliers. The authors validate CBQ on various LLMs and demonstrate that it achieves state-of-the-art performance in low-bit settings, like W4A4, across a range of datasets.
weaknesses: Baseline Comparisons: Although the results on LLAMA models are promising, the exclusion of more recent PTQ methods like BiE[1] limits the impact of the results. Including such baselines could provide a more complete picture of CBQ’s effectiveness.

Broader Applicability: The focus on language models limits the generalizability of CBQ. Expanding evaluations to other architectures, such as vision transformers and VLMs (vision language models), could strengthen the paper’s applicability.

[1] BiE: Bi-Exponent Block Floating-Point for Large Language Models Quantization. ICML 2024

Meta-review: The reviewers listed some weaknesses: The evaluation is limited to older models like LLAMA1, LLAMA2, and OPT, with no comparison to newer models like LLAMA3, Mistral, or Falcon. Despite claims that LoRA-Rounding reduces computation, no experiments are provided to support this. The comparison with other methods, such as AWQ, is missing, and the impact of varying overlap size in CBQ is not analyzed in depth, especially regarding memory efficiency and latency. Additionally, the necessity of the coarse-to-fine preprocessing strategy is unclear, and latency measurements for certain bit-widths are absent. Comparisons to recent rotation-based methods like FrameQuant and QuIP, as well as weight-only quantization with sub-channel quantization, are also omitted. Finally, the focus on LLAMA models limits the generalizability of the approach, and including evaluations on other architectures like vision transformers or VLMs would broaden its applicability.
And the authors addressed most of them in the rebuttal period and reviewers raised their rating. One of the reviews that was strong accept was very short, so it is not heavily considered in this decision. There is no negative review, so will be accepted.

Decision: Accept (Spotlight)