TesseraQ: Ultra Low-Bit LLM Post-Training Quantization with Block Reconstruction

Authors: Yuhang Li, Priyadarshini Panda
Primary area: foundation or frontier models, including LLMs
Keywords: LLM, Quantization, Efficient Inference

Abstract:
Large language models (LLMs) have revolutionized natural language processing, albeit at the cost of immense memory and computation requirements. Post-training quantization (PTQ) is becoming the \emph{de facto} method to reduce the memory footprint and improve the inference throughput of LLMs.
In this work, we aim to push the upper limit of LLM PTQ by optimizing the weight rounding parameters with the block reconstruction technique, a predominant method in previous vision models.
We propose TesseraQ, a new state-of-the-art PTQ technique, to quantize the weights of LLMs to ultra-low bits.
To effectively optimize the rounding in LLMs and stabilize the reconstruction process, we introduce progressive adaptive rounding. This approach iteratively transits the soft rounding variables to hard variables during the reconstruction process. Additionally, we optimize the dequantization scale parameters to fully leverage the block reconstruction technique.
We demonstrate that TesseraQ can be seamlessly integrated with existing scaling or clipping-based PTQ algorithms such as AWQ and OmniQuant, significantly enhancing their performance and establishing a new state-of-the-art.
For instance, when compared to AWQ, TesseraQ improves the wikitext2 perplexity from 14.65 to 6.82 and average downstream accuracy from 50.52 to 59.27 with 2-bit weight-only quantization of LLaMA-2-7B. 
Across a range of quantization schemes, including W2A16, W3A16, W3A3, and W4A4, TesseraQ consistently exhibits superior performance.

Decision: reject
Reviewer scores: 5, 6, 6, 5, 3

Reviews and discussion:
<thread>
creation date:
1729108644470

thread_type:
official_review

<thread_object>
creator: reviewer_wxs8

<content>
summary:
TesseraQ is a new PTQ method for ultra-low bit quantization of LLM weights. It features progressive adaptive rounding and optimized dequantization scale parameters, improving block reconstruction. Integrated with PTQ algorithms like AWQ and OmniQuant, TesseraQ significantly enhances performance.

strengths:
The authors introduce two techniques to optimize more parameters under PTQ settings, enhancing the performance of quantized models through block-wise reconstruction. These approaches, combined with existing transformation-based methods, deliver improved model accuracy without adding inference overhead.

weaknesses:
1. While the paper claims to establish a new state-of-the-art, SpinQuant, as discussed in the related work, appears to far outperform the proposed method. It would be beneficial if the authors could initialize their model from SpinQuant and then apply their method to demonstrate its effectiveness.
2. Line 141 mentions that GPTQ can cannot be used to improve other transformation-based methods. However, GPTQ can significantly enhance the performance of rotation-based transformation like SpinQuant and QuaRot.
3. The rationale for using progressive freezing instead of a dynamic regularization term (i.e., AdaRound) for optimizing rounding values is not clearly articulated, nor is it convincingly demonstrated why this method surpasses AdaRound. The explanations in Lines 075 to 077 lack substantial evidence.
4. The paper does not compare its progressive rounding approach with existing rounding methods such as AdaRound, AdaQuant, and FlexRound, missing an opportunity to justify the need for progressive rounding.
5. Lines 230 to 231 discuss the potential negative impacts of optimizing the scale parameter $s$ in quantization. However, experimental evidence is needed to substantiate that learning $s$ is detrimental, as the current argument does not conclusively link optimization of $s$ to performance degradation.
6. The paper optimizes significantly more parameters than previous methods using limited data, raising concerns about potential overfitting. A user study or chat test could provide essential validation.
7. Line 473 states a significantly larger value than AWQ/OmniQuant, but it's unclear what this comparison signifies as these methods do not optimize rounding to flip variables.
8. The necessity of the sigmoid function $\sigma(\cdot)$ in Equation (9) should be experimentally validated. Why not use $\mathbf{v} \times \mathbf{s} \times (\theta^{q}-\mathbf{z})$ with $\mathbf{v}$ initialized to $\mathbf{1}$?
9. Further evaluation is needed on the effect of larger $t$ values in the expression $\frac{1}{\exp(tx)}$ and other scheduling functions in the **PAR Schedule** to demonstrate the method's robustness to different schedules.
10. More comparisons with block-wise reconstruction or learning-based methods, e.g, DuQuant (https://arxiv.org/abs/2406.01721) and QLLM (https://arxiv.org/pdf/2310.08041) could help confirm its effectiveness.
11. The author does not provide results for the proposed method without using transformation-based initialization.

questions:
Why didn't the authors compare their method with OmniQuant for weight-activation quantization?
</content>
</thread_object>

</thread>
<thread>
creation date:
1729653238228

thread_type:
official_review

<thread_object>
creator: reviewer_eujb

<content>
summary:
Large language models (LLMs) has recently gained widespread attention，but face challenges in memory and computation requirements.  Post-training quantization (PTQ) is a common approach to address these issues. This work mainly proposes some optimization methods for PTQ targeting LLM.The proposed TesseraQ, a novel PTQ technique, can quantize LLMs to ultra-low bits. It introduces progressive adaptive rounding to iteratively transition soft rounding variables to hard ones during reconstruction and optimizes the dequantization scale parameters. TesseraQ can be integrated with existing PTQ algorithms like AWQ and OmniQuant. Experiments show that TesseraQ significantly improves performance.

strengths:
The paper proposes a new rounding optimization design called TesseraQ， it can be combined with other PTQ methods to achieve new performance levels, establishing a new technological level for quantifying LLM in terms of confusion, downstream accuracy, and hardware performance.

weaknesses:
1.  3.2 Some expressions are not very clear, and some variables have not been explained
2.  The new rounding error optimization method proposed in the paper mentions that compared to previous methods ( https://arxiv.org/pdf/2004.10568 ) does not rely on regularization loss , but HS and sorting were introduced. It doesn't have a significant advantage.For example, computational complexity and convergence speed.
3.  The paper combines the proposed rounding error optimization method with other methods for experimentation and comparison with other quantization methods, but does not use other rounding error optimization methods for the same experiment, and does not demonstrate the superiority of this method.For example, the Adaround mentioned in the article.
4. When comparing quantitative indicators such as W4A4/W3A3 that include activation, the QuaRot initial method was used, but it was not compared with using the QuaRot method alone. The advantage of the proposed new rounding error optimization method in improving accuracy cannot be clearly demonstrated.Suggest to use QuaRot as a baseline separately to include the results.

questions:
1. The new rounding error optimization method proposed in the paper mentions that compared to previous methods ( does not rely on regularization loss , but HS and sorting were introduced. Does it have advantages in training convergence speed and video memory usage?
2. The paper combines the proposed rounding error optimization method with other methods for experimentation and comparison with other quantization methods, but does not use other rounding error optimization methods for the same experiment, Can relevant experiments be supplemented ?
3.When comparing quantitative indicators such as W4A4/W3A3 that include activation, the QuaRot initial method was used, but it was not compared with using the QuaRot method,Can relevant experiments be supplemented?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730319479231

thread_type:
official_review

<thread_object>
creator: reviewer_cbes

<content>
summary:
The paper presents a novel method to compress large language models (LLMs) into ultra-low-bit formats while maintaining high performance. It introduces Progressive Adaptive Rounding (PAR), which gradually decides rounding schedules for groups of parameters within LLMs while retaining the training flexibility via Sigmoid-based soft-hard transition, improving quantization stability. By integrating with block-wise reconstruction instead of layer-wise optimization, TesseraQ captures inter-layer dependencies more effectively, and de-quantization scale tuning further reduces reconstruction errors. The method integrates seamlessly with existing quantization approaches like AWQ and OmniQuant, significantly improving perplexity and accuracy across benchmarks. TesseraQ achieves state-of-the-art results, offering a practical solution for deploying LLMs efficiently in memory-constrained environments.

strengths:
The paper offers an interesting and effective add-on for block-wise reconstruction for LLM quantization with optimized rounding policy. Especially for Ultra-low bit scenarios, such combination advances the such field where prior quantization methods struggle to maintain performance for various tasks.

The methodology is well-designed, with clear theoretical motivation for each component, including PAR and dequantization scale tuning. The experimental results are robust, demonstrating TesseraQ’s superiority across multiple benchmarks such as WikiText2 and downstream reasoning tasks. The paper carefully evaluates TesseraQ’s effectiveness by comparing it to several existing methods, providing quantitative evidence that supports the proposed contributions.

In terms of the overall writing quality, the paper is generally well-organized and explains technical concepts like rounding optimization and block reconstruction in a clear and structured manner.  

The significance of the paper lies in its potential impact on LLM deployment, particularly for memory-constrained environments such as consumer devices and edge processors. TesseraQ pushes the boundaries of ultra-low-bit quantization, a growing need as LLMs become larger and more ubiquitous.

weaknesses:
1. The paper proposes PAR instead of the common STE implementation. There is a lack of justification why direct utilization of STE for the rounding policy estimation would result in worse performance so that PAR is significant in the overall design. 

2. The paper purposes an effective rounding optimization add-on for enhancing the block-wise quantization for LLM, but there is a lack of theoretical analysis of the different between: A) BLOCK-WISE & Normal Rounding (round to the nearest) and  B) the proposed BLOCK-WISE & Rounding Optimization. If A) is well trained and fully converged, would it theoretically have the similar mathematical foundations as B)?

questions:
The paper is generally clear and well-written, the algorithm has been fully presented. While according to my perspective in the weakness section, I have the following question:

1. Why does the paper adopt Progressive Adaptive Rounding (PAR) instead of the more common Straight-Through Estimator (STE) for rounding policy estimation? What specific limitations or performance issues with STE justify the need for PAR in the overall design? If STE is applied, whether such progressive-stage training is still required?

2. Could the authors justify the theoretical basis for the improvement achieved by rounding optimization over the baseline?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730703038734

thread_type:
official_review

<thread_object>
creator: reviewer_co5y

<content>
summary:
This paper introduces TesseraQ, a post-training quantization method for LLMs that focuses on optimizing weight rounding parameters using block reconstruction techniques. The key design is a Progressive Adaptive Rounding approach that iteratively converts soft rounding variables to hard variables, along with dequantization scale tuning.
﻿
The method is designed to work with existing quantization approaches like AWQ and OmniQuant, significantly improving their performance particularly for ultra-low bit scenarios (2-4 bits). Results show major improvements in both perplexity metrics and downstream task accuracy across different LLaMA models and quantization schemes.
﻿
The main contribution is enabling better ultra-low bit quantization of LLMs through tuning of quantization rounding while maintaining model performance, demonstrated through comprehensive empirical results.

strengths:
1. The paper presents a clear motivation for ultra-low bit LLM quantization, with a well-justified technical solution. The authors effectively demonstrate how current methods struggle with low-bit quantization and explain why optimizing the entire weight tensor is necessary. The progressive adaptive rounding approach is sound to handle the problem.

2. The writing quality is exceptionally clear and professional. Complex technical concepts are explained systematically, from basic quantization principles to advanced optimization techniques. The figures and algorithms are informative and well-designed, particularly Figure 1 which clearly illustrates the workflow.

3. The experimental results show substantial improvements over strong baselines. For example, improving WikiText2 perplexity from 37.37 to 8.05 on LLaMA-2-7B with W2A16 quantization is a significant achievement. The ablation studies effectively isolate the contributions of PAR and DST.

4. The practical utility is demonstrated through real-world evaluations. The authors test different hardware configurations, measure actual throughput, and analyze memory consumption.

weaknesses:
1. The paper lacks detailed technical comparisons with similar methods. While mentioning prior approaches like AdaRound and AdaQuant,  it doesn't experimentally demonstrate why the proposed methods are superior to the these methods. The rationale behind several design choices (sigmoid reparameterization; harden score metric in Eq 6; avoiding scale optimization in the quantization step) isn't fully justified through comparative experiments. 

2. The evaluation demonstration has problems. 
- The inconsistency in model selection between Tables 1, 2 and 3 makes it difficult to draw comprehensive conclusions. 
- Runtime and GPU memory analysis is limited to the 7B model.  As the runtime and memory consumption are important metircs, the authour should provide comprehensive efficiency metrics across different model sizes. 
- The paper would be stronger with complete results for newer models like LLaMA-3 across all experiments.

3. The technical novelty is relatively incremental. Both block reconstruction and scale tuning are well-established in PTQ literature. While the paper makes useful improvements, it builds primarily on existing techniques rather than introducing fundamentally new concepts.

questions:
1. The choice to restrict rounding to ±1 steps seems limiting. Why not allow multiple step sizes as in AdaQuant? This decision could impact optimization flexibility, especially for ultra-low bit quantization where precision is crucial. Has this design choice been empirically validated against alternatives?

2. The effectiveness with weight transformation techniques is unclear. Specifically, would TesseraQ maintain its benefits when applied after Hadamard matrix transformations (as in QuaRot)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730704786301

thread_type:
official_review

<thread_object>
creator: reviewer_ck8i

<content>
summary:
This paper presents a PTQ method，aTesseraQ, for LLM.  The author propose optimize weight rounding through a progressive
approach based on block reconstruction. Besides, the author also propose dequantization scale tuning to mitigate the reconstruction error.
The experiments on LLaMa models shows good performance, and this training-based PTQ method could be implemented with other PTQ methods.

strengths:
1. The idea of optimizing the weight rounding error in LLM is interesting, while most previous methods focus on the weight clipping or distribution transmation.
2. The author proposes Progressive Adaptive Rounding strategy by deviding all rounding variables into harden and soften.
3. The performance in W2A16 setting is good and TesseraQ is orthogonal to other post-training quantization (PTQ) methods for LLM.

weaknesses:
1. unclear writing: The writing in Section 3.2 of the paper is not sufficiently clear, making it somewhat difficult to read and understand. Moreover, this section constitutes the core content of the methodology and pertains to the detailed description of the method itself.
2. Outdated Comparison: The comparison methods in Table 1 appear to be somewhat outdated. The authors should compare with more advanced weight-only methods to demonstrate its superiority. 
3. Besides, other tables comparison seems inadequate. The authos should also report the comparison results with other Rounding method, such as Adaround and signRound.
4. The author should discuss the method's limitation.

questions:
1. Why the author didn't compare with Adaround and signRound in Table1 and other experiments?
2. Compared with the traditional Layer-wise Rounding method Adaround, what is the superiority of the proposed TesserQ, is the block-wise reconstruction granulity or the Harden and Soften design? Why? The author should add more related discussion and results to analysis it.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732040752617

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to All Reviewers

comment:
We sincerely thank all reviewers for their thoughtful and constructive feedback. We are encouraged that they agree that our method is interesting, effective, and significantly improves the uniform quantization performance of LLMs. We will try our best to address each reviewer’s questions and concerns point-to-point. We also welcome any further discussion of our paper. 

Here we would like to highlight some major changes to our paper:

1. **Updated more perplexity results to Table 1**. Following suggestions by Reviewer co5y, we run our TesseraQ on LLaMA-3-8B and 70B models and compare it against existing methods like GPTQ, AWQ. Moreover, as suggested by Reviewer Ck8i, co5y, we compare two additional baseline methods, SignRound, and GPTQ combined with QuaRot. We show that our method consistently outperforms the existing methods on all series of LLaMA models. For example, on LLaMA-3-8B W2A16g128 quantization, our method obtains **10.03** perplexity while the AWQ model is 334.1 and GPTQ+QuaRot is 17.43. 
Some representative results are listed here:
| Method         | LLaMA-3-8B W2A16g128 | LLaMA-3-8B W3A16g128 | LLaMA-3-70B W2A16g128 | LLaMA-3-70B W3A16g128 |
|----------------|----------------------|----------------------|-----------------------|-----------------------|
| GPTQ + QuaRot  | 17.43                | 7.42                 | 30.89                 | 4.98                  |
| AWQ            | 334.1                | 8.24                 | 10.98                 | 4.63                  |
| TesseraQ + AWQ | **10.03**            | **6.90**             | **7.47**              | **4.13**              |

2. **Updated more WA quantization results to Table 3**. Following suggestions by Reviewer co5y and wxS8, we additionally run TesseraQ on LLaMA-1/2-7B models for weight-activation quantization for comprehensive comparison. In addition, we add two more baseline results like Atom and QLLM in Table 3. The results demonstrate that TesseraQ can be used to initialize both scale-transformation and rotation-transformation models in weight-activation quantization scenarios, achieving even better performance. For example, on W4A4 quantized LLaMA-2-7B, our method obtains **55.12%** average accuracy while previous methods like AWQ, OmniQuant, OS+, and QLLM obtain 49~51% average accuracy. 

3. **Provided an ablation study on rounding optimization**. As suggested by all reviewers, a comparison to existing rounding learning methods like AdaRound, and AdaQuant, is needed. To clarify, very few works have tried implementing rounding optimization on LLMs quantization and GPTQ (Frantar et al.) claimed that these methods *are complex and challenging to scale to billions of parameters*. Nevertheless, we agree that a more practical study would benefit our work. Therefore, we implemented AdaRound and several variants to demonstrate the effectiveness of our proposed method with LLaMA-2-7B W2A16g128 quantization. Please check our new results in Appendix A. We also summarize the results in the following table. In summary, our PAR consistently outperforms the other two rounding methods regardless of objective. We think the reason is that we explicitly control the hardness of rounding variables through the progressive approach. On the other hand, AdaRound and AdaQuant are less optimized on LLMs and may require more hyper-parameter search. 
| Rounding          | Objective | Wikitext2 ($\downarrow$) | C4 ($\downarrow$) |
|-------------------|-----------|--------------------------|-------------------|
| None (AWQ)        | Layer     | 14.65                    | 18.67             |
| AdaRound          | Layer     | 10.68                    | 15.67             |
| AdaQuant          | Layer     | 16.78                    | 21.34             |
| PAR               | Layer     | **9.43**                 | **12.79**         |
| None (OmniQuant)  | Block     | 11.06                    | 16.34             |
| AdaRound          | Block     | 9.05                     | 11.45             |
| AdaQuant          | Block     | 10.05                    | 14.87             |
| PAR               | Block     | **6.82**                 | **10.77**         |

4. **Revise Sections 3.1 and 3.2 for better clarity**. We justify the reason why we chose the rounding optimization framework and the reason we introduced Progressive Adaptive Rounding. 

For other technical questions, please check our detailed response to each reviewer.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces TesseraQ, a novel post-training quantization (PTQ) technique for large language models (LLMs), aiming to improve performance by optimizing weight rounding through a block reconstruction approach. The authors also propose a progressive adaptive rounding strategy and the optimization of dequantization scale parameters. Experimental results show that TesseraQ significantly improves perplexity and downstream task accuracy when applied to the LLaMA-2 model, particularly with ultra-low bit quantization. The authors also highlight that TesseraQ can be seamlessly integrated with existing PTQ methods, offering a flexible and effective solution for optimizing LLMs.

The strengths of this paper lie in its innovative approach to weight rounding and the proposed progressive adaptive rounding strategy. The experiments are well-conducted, showing that TesseraQ outperforms previous methods, such as AWQ, on multiple quantization schemes. Additionally, the ability to integrate TesseraQ with existing PTQ methods enhances its practical applicability. The results, particularly in terms of perplexity reduction and downstream task performance, are promising and contribute to advancing the field of LLM optimization.

However, several concerns were raised by reviewers that the authors attempted to address but did not fully resolve. One major issue is the inconsistency in the reported results, especially in comparison to similar methods like SpinQuant and QuaRot. The reported perplexity for LLaMA-2-7B is notably higher than previous benchmarks, which raises doubts about the generalizability of the method. The authors have not sufficiently aligned their results with these previous studies, and the lack of detailed comparison with full versions of baseline methods undermines the claimed improvements. Additionally, the experimental setup is not sufficiently detailed, leaving unclear how some of the discrepancies arose.

Given these unresolved issues, particularly the divergence in results and the absence of a thorough comparison with full baseline versions, I recommend rejecting the paper at this time. While the proposed method is promising and the experimental results are noteworthy, the issues with result alignment, insufficient comparisons, and lack of methodological clarity need to be addressed before this work can be considered for acceptance. A more thorough evaluation across different datasets and models, along with an in-depth experimental setup description, would strengthen the paper significantly.