BCQ: Block Clustered Quantization for 4-bit (W4A4) LLM inference

Authors: Reena Elangovan, Charbel Sakr, Anand Raghunathan, Brucek Khailany
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: Post-training Quantization, Large Language Models, Codebooks, Clustering, Block Clustered Quantization

Abstract:
Post-training quantization (PTQ) is a promising approach to reducing the storage and computational requirements of large language models (LLMs) without additional training cost. Recent PTQ studies have primarily focused on quantizing only weights to sub-8-bits while maintaining activations at 8-bits or higher. Accurate sub-8-bit quantization for both weights and activations without relying on quantization-aware training remains a significant challenge. In this work, we introduce a novel quantization method called block clustered quantization (BCQ) wherein each operand tensor is decomposed into blocks (a block is a group of contiguous scalars), blocks are clustered based on their statistics, and a dedicated optimal quantization codebook is designed for each cluster. We propose a PTQ algorithm called Locally-Optimal BCQ (LO-BCQ) that iterates between the steps of block clustering and codebook design to greedily minimize the quantization mean squared error. When weight and activation scalars are encoded to W4A4 format (with 0.5-bits of overhead for storing scaling factors and codebook selectors), we advance the current state-of-the-art by demonstrating <1% loss in inference accuracy across several LLMs and downstream tasks.

Decision: reject
Reviewer scores: 8, 3, 5, 6

Reviews and discussion:
<thread>
creation date:
1730125946141

thread_type:
official_review

<thread_object>
creator: reviewer_zf8x

<content>
summary:
This paper introduce a novel block quantization method called LO-BCQ, which achieves state-of-the-art level while avoiding additional memory overheads by iterative block clustering and quantization through locally optimization pipeline.

strengths:
1. This paper iteratively optimizes block clustering and per-cluster quantization to minimize MSE loss for both weight and activation and achieves advanced level. 
2. LO-BCQ avoids memory overheads caused by per-block specific codebook and scaling factors in previous methods by sharing scaling factors in a group of blocks.

weaknesses:
1. The paper ignores to describe the distinctions between block quantization and group-wise quantization. because from the perspective of form and definition, there seems to be no difference between them.
2. This paper solely conduct comparison experiments with block quantization based methods. However, there are many other widely used PTQ methods for LLMs such as OmniQuant, GPTQ, QuIP, etc. 
3. In addition to accuracy, efficiency is an important criterion for measuring the effectiveness of a proposed method. Compared to general PTQ methods, LO-BCQ involves numerous optimization processes, which to some extent reduce the method's efficiency.
4. The logical flow of the writing needs to be improved, especially for Introduction.

questions:
1. For weakness1, if feasible, please involve detailed analysis and experiments for block quantization and group-wise quantization, such as AWQ-g128.
2. For weakness2, to demonstrate the advanced nature of your method, please include comparisons with the mentioned PTQ methods in your experimental tables. 
3. For weakness3, Please provide the running time consumption of LO-BCQ. If the proposed method is faster than OmniQuant, which is currently known for a relatively low-efficient but effective PTQ method, it would still be within an acceptable range.
4. The author should check for some spelling errors in the manuscript, such as "biwidth" in line 054.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730290249302

thread_type:
official_review

<thread_object>
creator: reviewer_ojwf

<content>
summary:
This paper introduces a method named block clustered quantization (BCQ), which decomposes operand tensors into blocks, clustering them based on statistics and designing codebooks for each cluster. When weight and activation scalars are encoded to W4A4 format, BCQ achieves the current state-of-the-art by demonstrating < 1% loss in inference accuracy across several LLMs.

strengths:
1. This paper is well written, clearly described, and presents an effective methodology that makes group quantization a significant enhancement across a wide range of tasks and models;
2. Combining a quantization problem with a clustering problem is novel enough, I think, and the author's algorithm is simple and effective enough.

weaknesses:
1. Although the authors' group quantization achieved a significant improvement in accuracy, the paper does not report results on inference speed. From QServe we can learn that group quantization is very inefficient in terms of speed of reasoning. Does the authors' proposed quantization scheme have any performance advantage over group quantization, except for the advantage of saving some storage? Can the authors provide the corresponding latency metrics?
2. The authors' experiments should include other model types such as LLaMA3 and Mistral, or even LLaMA3.1 and LLaMA3.2, and I am very much looking forward to the performance of the authors' method on MOE LLM model.
3. Figure 2, Figure 3 and Figure 4 can be modified to better illustrate and/or an algorithm description of the steps.

questions:
1. Has the author's algorithm been compared to GPTQ, which is a very effective way to quantify large models in PTQ quantization, and how does the author's method compare to GPTQ in terms of effectiveness and efficiency?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730557220518

thread_type:
official_review

<thread_object>
creator: reviewer_lf8w

<content>
summary:
The Paper introduces Block Clustered Quantization (BCQ), a novel method aimed at achieving accurate sub-8-bit quantization for weights and activations in large language models without relying on quantization-aware training (QAT). BCQ decomposes operand tensors into blocks, clusters these blocks and designs dedicated optimal quantization codebooks for each cluster. This approach allows for tailored quantization that minimizes mean squared error (MSE) and maintains high inference accuracy.

strengths:
- Using LO-BCQ algorithm utilizing block clustering and dedicated codebooks effectively minimizes mean squared error (MSE) while maintaining high accuracy.
- The paper considers several datasets to evaluate, especially include the MMLU boolq and Race datasets.
- Achieving less than 1% loss when quantizing weights and activations to 4-bits.

weaknesses:
- The experiment shows that LO-BCQ takes good performance but it is built on multiple codebooks and small block division, the current LLM has a huge number of parameters, 8 elements for a block division will lead to a large computational overhead, and it is still a question whether the good performance can be maintained if the size of the block is increased.

- LO-BCQ's fine-grained quantization does not seem to have an advantage over per-group quantization such as Atom[1], which quantizes 128 consecutive elements as a group, and achieves excellent results in W4A4, and even better if Atom's grouping is adjusted to 64 or smaller. In addition, LO-BCQ introduces operations such as clustering, indexing, updating codebooks, etc., which costs more computational overhead.
[1]Zhao, Yilong, et al. "Atom: Low-bit quantization for efficient and accurate llm serving.

questions:
- In Table 4 Ablation experiments, only the case where $L_{b}$ is less than 8 is discussed, what is the performance of the model if the size of $L_{b}$ is aligned with the group quantization e.g. 128, 64?
- The performance improvement of LO-BCQ seems to partly come from the clustering of blocks, the value of activation in LLM has a large difference in the distribution of different datasets, does the use of pre-calibrate for activation to calculate the codebook of activation have an impact on the language modeling ability of LLM?
- What is the impact on model inference speed of performing clustering with block size 8, indexing, and looking up codebooks during model inference? Does it mask the gains from low-bit quantization?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730815272821

thread_type:
official_review

<thread_object>
creator: reviewer_miqq

<content>
summary:
The paper presents Block Clustered Quantization (BCQ), a post-training quantization (PTQ) method designed to compress both weights and activations in large language models (LLMs) to 4-bit precision. BCQ divides tensors into small blocks, clusters these blocks based on their statistical properties, and then quantizes each cluster using a dedicated codebook, reducing memory footprint and computational load while preserving model accuracy. The Locally Optimal Block Clustered Quantization (LO-BCQ) algorithm iteratively refines clusters and codebooks to minimize quantization error. With solid experiments with varying configurations on Llama2 and GPT3 models,  it has shown LO-BCQ outperforms the states-of-the-art methods on W4A4 quantization.

strengths:
1. Introducing iterative block clustering to derive an optimal quantization codebook for each block is novel, providing a robust and flexible mapping to the most accurate representation of each block. This approach, I believe, is the core factor driving the success of this method.
   
2. It is a compelling finding that the codebook is universally applicable for quantizing any tensor, at any layer, across different models.

weaknesses:
I find this paper impressive, with no obvious weaknesses in its methodology or presentation. Please refer to the Questions section for some discussion points

questions:
1. How does this method perform on W2A2 and W8A8 quantization, and how does it compare to other methods? What are the bottlenecks when applying this method to more aggressive quantization?

    2. As shown in Fig. 6, the range of values within some codebooks is smaller than [-1, 1], which appears to compensate for the large L_A. Have you tested even smaller L_A values, and how do they perform?

    3. 223: fond → found.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732741630756

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We would like to thank all the reviewers for their insightful comments and questions. We appreciate the interest in codebook-based approach for achieving 4-bit weight and activation quantization in LLMs. We would like to emphasize that to the best of our knowledge, we are the first to achieve < 1% loss in downstream task accuracy when both LLM activations and weights are quantized to 4-bits during PTQ (no finetuning).

## Key Modifications:
The main criticism of our work is regarding the group-size (referred as block array size in our paper) of quantization. We agree with the reviewers that a small group-size such as 8 can lead to large computational overheads due to high precision scale factor multiplication per-group. However, we would like to clarify that the block-array size denoted by ‘$L_A$’ in our work is equivalent to group-size, that is, each block array shares a scale-factor. Therefore, in our work, scale factors are shared among groups of size 64, rather than 8, which has much smaller overhead.

In response to reviewers’ suggestion to compare inference accuracy of LO-BCQ against other weight-only quantization proposals, we report perplexity loss on Wikitext dataset during weight-only LO-BCQ. We explore a large group-size of 128 and show that LO-BCQ achieves lower perplexity loss compared to other methods.  Here, the LO-BCQ configuration is denoted by tuple $\{L_A, L_b, N_c\}$ = {Length of block array, Length of block, Number of codebooks}. LO-BCQ with $2$, $4$, $8$ and $16$ codebooks result in an effective bitwidth of $4.19$, $4.31$, $4.44$ and $4.56$, respectively.

### Wikitext perplexity loss compared to unquantized baseline
| Method                      | Llama2-7B | Llama2-70B |
|-------------|-----------|------------|
| GPTQ                        | 0.22     | 0.10      |
| AWQ                         | 0.13     | **0.09**  |
| OmniQ                       | 0.27     | 0.15      |
| QuiP#                       | 0.19     | 0.10      |
| **LO-BCQ** \{128,8,2\} | 0.14     | **0.09**  |
| **LO-BCQ** \{128,8,4\} | 0.12     | **0.07**  |
| **LO-BCQ** \{128,8,8\} | **0.09** | **0.06**  |
| **LO-BCQ** \{128,8,16\} | **0.08** | **0.05**  |

The superior performance of LO-BCQ is from effectively designing codebooks that minimize quantization error and flexibly mapping each block to a codebook that best represents it.

## Other modifications
- Figure1: We include comparison to Atom which is an excellent group quantization-based proposal for quantizing both weights and activations in LLMs-
- Table 3. (b): We report weight-only quantization results for LO-BCQ with a large group size of 128 and compare it to GPTQ, AWQ, OmniQ and QuiP#.
- Table 7: We report Wikitext-103 perplexity results with Nemotron4-340B model and various LO-BCQ configurations
- We fixed the spelling errors pointed out by the reviewers.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents a novel quantization technique, Block Clustered Quantization (BCQ), aimed at significantly reducing the storage and computational needs of large language models (LLMs) while maintaining near-state-of-the-art inference accuracy. This is achieved through the decomposition of operand tensors into blocks, clustering these blocks based on statistical characteristics, and employing tailored quantization codebooks for each cluster. The authors argue that the proposed method is innovative and addresses the critical challenge of achieving efficient sub-8-bit quantization without further training costs.

The reviewers exhibited varied perspectives on the paper’s merits. While some acknowledged the algorithm's technical novelty and its potential for improving quantization accuracy, concerns were raised regarding the lack of clarity in differentiating BCQ from existing group-wise quantization methods. Notably, one reviewer recognized the paper's innovative approach yet categorized it as marginally below the acceptance threshold, indicating mixed opinion among the reviewers.

Despite receiving a rebuttal from the authors, significant concerns remained unaddressed, particularly surrounding inference latency. While the authors asserted that their method improves inference accuracy, several reviewers questioned whether the added complexity of block clustering and codebook indexing could offset these gains, particularly in practical applications. One reviewer emphasized that the proposed method might not deliver substantial improvements in inference speed, echoing concerns that persisted beyond the authors' responses.

In conclusion, while the paper has merit and could contribute to the field of quantization in LLMs, the unresolved issues regarding inference latency and the clarity of its novelty compared to existing methods hinder unanimous support for acceptance. Based on the reviewers' insights, the paper does not currently meet the criteria for acceptance due to these outstanding concerns.