Low-Rank Correction for Quantized LLMs

Authors: Meyer Scetbon, James Hensman
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: Quantization, LLM, Low-rank

Abstract:
We consider the problem of model compression for Large Language Models (LLMs) at post-training time, where the task is to compress a well-trained model using only a small set of calibration input data. 
In this work, we introduce a new low-rank approach to correct for quantization errors of \emph{activations} in LLMs: we propose to add low-rank weight matrices in full precision that act on the \emph{unquantized} activations. We then solve a joint optimization problem over the quantized representation of the weights and additional low-rank weight matrices to quantize both weights and activations.
We focus on the case of 4-bit weight-and-activation quantization (W4A4). Using ranks equivalent to 10\% of the original weight matrix size, our approach reduces the accuracy gap with the original model by more than 50\%. Using ranks equivalent to 30\% of the original weight matrix,  the accuracy gap is closed completely. We demonstrate our results on four recent LLMs, namely Llama-2, Llama-3, Phi-3 and Mixtral models.

Decision: reject
Reviewer scores: 6, 6, 5, 5, 3

Reviews and discussion:
<thread>
creation date:
1730528439129

thread_type:
official_review

<thread_object>
creator: reviewer_lqqc

<content>
summary:
This work presents a low-rank approach aimed at enhancing the performance of post-training quantization. Specifically, it refines the Low-Rank Correction method by incorporating selected low-rank matrices in full precision during the forward pass to help reduce quantization errors. The experimental results indicate competitive performance, showing improvements over current methods, espeically at W4A4 quantization levels.

strengths:
1. The studied problem is important.
2. This paper provides a new method of low rank correction to consider both weights and activations are quantized situation.
3. This paper shows a good experimatal performance on low bit quantazation W4A4.

weaknesses:
1. The paper does not provide sufficient proof of the correctness of its algorithm. There is no error analysis for the approximation methods presented. While the authors detail their approach for solving Equations 3 and 4, they do not demonstrate that these solutions are equivalent to Equation 2, which is the main problem of interest.
2. The theoretical analysis provided by the authors is overly simplified. The assumptions of full rank in Propositions 3.3 and 3.4 are questionable. Empirical studies have suggested that activations in large transformer models often exhibit a structure that can be approximated as low rank, making these assumptions potentially unsuitable.
3. The paper does not include an analysis of the time complexity associated with adding low-rank weights. This omission leaves a gap in understanding the computational implications of the proposed method.
4. The contribution and novelty of the paper are somewhat limited. While the authors improve the accuracy of post-training quantization through low-rank correction, additional analyses on aspects such as time complexity, convergence, and error bounds would enhance the paper’s competitiveness and impact.

questions:
See weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730681427018

thread_type:
official_review

<thread_object>
creator: reviewer_xtpe

<content>
summary:
The paper studies post training quantization. They achieve a new SoTA for W4A4. This and the introduction of their algorithm is the main contribution of the paper. The main idea is to solve Equation (2) using alternating minimization by optimizing either for quantized weights or for low rank matrices. The authors also initialize the low rank matrices carefully. In terms of experimental results, the authors report wikitext-2 perplexity as well as from lm-eval (PIQA, HellaSwag, Arc-Easy, Arc-Challenge, Winograd and Lambada). The numbers demonstrate that the introduced approach beat their main benchmark QuaRot. Surprisingly, one iteration of alternating minimization is enough to achieve good performance.

strengths:
please see summary

weaknesses:
minor typos:

Line 57: summurized -> summarized

Line 191: "... we drop the dependency in l of our notations" -> replace "in" with "on"

Line 353: "Our ambition is to close the gap between out main benchmark" -> replace "out" with "our"?

Line 405: "equan" -> equal

Line 421: "prononced" -> pronounced

Line 455: "In this work we have not study the computational" -> replace "study" with "studied"

questions:
The paper contains several propositions but they lack proofs.

The approach of adding a low rank matrix is similar to approaches in these papers that the authors could cite: "LoRA: Low-Rank Adaptation of Large Language Models" and "QLoRA: Efficient Finetuning of Quantized LLMs".

My main concern with the paper is that it doesn't cite or discuss the above 2 related papers, which seem quite related. Also, the paper lacks several proofs. Furthermore, the paper could consider generative tasks for benchmarks such as gsm8k. Typically, generative tasks are harder for quantized model than multiple choice tasks. Given all this, my overall rating for the paper is "marginally below the acceptance threshold".
</content>
</thread_object>

</thread>
<thread>
creation date:
1730692492691

thread_type:
official_review

<thread_object>
creator: reviewer_rmtb

<content>
summary:
This paper introduces a new method called LRC (Low-Rank Correction) for quantizing large language models (LLMs) to 4-bit weights and activations (W4A4) while minimizing accuracy loss. The key idea is to jointly optimize for a quantized weight matrix acting on quantized activations and a full-precision low-rank weight matrix acting on the original unquantized activations. This allows LRC to significantly reduce the quantization error and close the accuracy gap with the full-precision model.

strengths:
This paper is its novel approach to quantizing large language models to 4-bit weights and activations while maintaining high accuracy. The LRC method's ability to optimize jointly for a quantized weight matrix and a full-precision low-rank correction matrix, which is connected to the original unquantized activations, effectively reduces quantization error. This innovative technique sets LRC apart from previous approaches and demonstrates its potential for enabling highly compressed models with minimal performance degradation.

weaknesses:
The paper does not analyze the computational cost associated with the added low-rank correction matrix. While the method effectively reduces quantization error, the impact on inference time and memory usage is not thoroughly explored. This is an important consideration for the practical deployment of the LRC method.

The authors leave the ideal implementation of the low-rank computation for future work. Without a concrete implementation strategy, it may be difficult for practitioners to immediately adopt the LRC method in real-world applications. Providing guidance or preliminary implementation details could have made the paper more impactful.

Although the paper identifies activation quantization as the primary source of error, it does not propose novel activation quantization schemes. The authors rely on existing techniques like round-to-nearest and suggest that future work on improved activation quantization could lead to better results. Addressing this limitation within the paper could have further strengthened the contribution.

questions:
How does the computational cost of the low-rank correction matrix scale with the size of the language model? Is there a trade-off between the compression ratio and the computational overhead introduced by LRC?

Can the LRC method be extended to other model compression techniques, such as pruning or knowledge distillation? How would the low-rank correction approach interact with these techniques?

How sensitive is the performance of LRC to the choice of calibration dataset used for computing the activation statistics? Would using a more diverse or domain-specific calibration dataset lead to better results?

The authors suggest that improved activation quantization schemes could further enhance the performance of LRC. What specific properties should these improved schemes have, and how might they be developed?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730722162644

thread_type:
official_review

<thread_object>
creator: reviewer_ijc7

<content>
summary:
This work studies a brand-new approach on the post-training quantization problem of LLMs. In particular, the authors suggest introducing a low-rank adaptation to relieve the accuracy loss of quantization, which uses a small size dataset to generalize their low-rank adaptation matrix U, V. This work is novel and easy to implement.

strengths:
* The idea of introducing low-rank adaptation to correct the quantization error is good, and trivially effective. In my perspective, these adaptation-based methods are worthy of further and comprehensive study.
* The derivation in this paper provides concise intuition, which is easy to follow.
* The topic of efficient LLM deployment is becoming vital currently, this method has considerable potential in addressing such PTQ problems on LLMs.

weaknesses:
* It is not clear how the rank of adaptation would influence the efficiency. This would become my main concern for this paper. I strongly recommend the authors to add an experiment to evaluate its enhancement of memory usage and speed.
* The presentation of this paper is good, but not excellent enough. The authors should add an introduction of GPTQ method and Cholesky in their appendix (since they are parts of the main algorithm) for presenting to the broader audience.
* The choice of dataset in $X$ should be carefully considered, but I don’t see any analysis about how $X$ will affect the performance. I recommend the authors to add a discussion about this and their assumption for rigorousness.

questions:
* Please answer the issues and questions in the Weakness and point out my potential misunderstandings. I am happy to discuss and enhance my rate.
* Why is the last line (LRC (5)) for the Mistral model missing both in Table 1 and Table 5?
* From Table 3, LRC fails to beat SVD on average score, can you explain why this happens?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730806486136

thread_type:
official_review

<thread_object>
creator: reviewer_tkaq

<content>
summary:
The paper proposes LRC (Low Rank Correction), a method to perform error correction for quantization of weights *and* activations (a common approach to perform model compression in LLMs). They add a full precision low-rank weight matrices in the forward pass that act on the unquantized activations and accounts for errors arising from both weights and activations.

strengths:
- They paper is written well and easy to understand. The scheme they propose is sound and intuitive in its formulation.
- Their method can use any weight quantization technique as a subroutine (they use GPTQ in the paper), which allows other tools/papers to plugin their own method.
- They perform sensible ablations in order to clearly identify the impact of the weight only quantization vs activation quantization, and when low rank error correction offers value. Moreover, they also show that the quantization is replaceable by any technique which can use the weight matrix and the covariances matrices to output a quantized version.

weaknesses:
# Major
- *Limited Contribution*: The paper stitches together many well known building blocks in the PTQ literature to build a sane, effective technique. In my opinion, it is a sound engineering feat, but still has high overlap with the previous work on the topic by Zhang et al (2024) and Ou et al (2024). The authors do differentiate themselves by the fact that they do a joint optimization over the low rank and quantized matrices which is key to the delta over the previous work. However, this is also a well known technique and has been applied in other works such as [1]
- The experiments are performed on models which do not overlap with the QuaRot paper (their primary baseline) and hence it is difficult to compare directly. Moreover, their method uses additional bits (aka low rank correction factors) and hence comparison to QuaRot is not a strictly fair comparison, a small increase in model size though it may be. 
- The authors mention a number of contemporary work (Quip, Quip#, LQER etc) but provide an empirical comparison against a very limited baseline (QuaRot).
- The authors claim that they can close the accuracy gap using rank equivalent to 30%, this seems to be true only for Phi3 which the authors present as an unqualified statement. Moreover, I cannot find the corresponding results in the evaluation section

# Minor
- In L431, the authors use the phrase "10% additional ranks" which I believe is not meaningful
- L399-407 sub section "on the effect of rank" does not mention the table/fig where the results can be found
- Table 1, for Phi3, LRC(1) outperforms LRC(5), a weird anomaly since one would expect performance to increase monotonically with iters
- Best results not provided in bold font
# References
1. Saha, Rajarshi, et al. "Compressing Large Language Models using Low Rank and Low Precision Decomposition." arXiv preprint arXiv:2405.18886 (2024).

questions:
- Why is it necessary to store the low rank matrices in full precision? Couldn't they also be quantized?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732241544906

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Author Rebuttal by Authors

comment:
**We thank the reviewers, AC and SAC assigned to this paper for their time and work looking into our submission.**

We thank them in advance for reading our rebuttal and interacting with us for a few more days during the discussion period.

We were happy to see that the paper was overall well received by all 5 reviewers:

**TKaQ:**  *They paper is written well and easy to understand. The scheme they propose is sound and intuitive in its formulation.*

**iJC7:** *The idea of introducing low-rank adaptation to correct the quantization error is good, and trivially effective.* 

**RmtB:** *This innovative technique sets LRC apart from previous approaches and demonstrates its potential for enabling highly compressed models.*

**xtpe:** *They achieve a new SoTA for W4A4.*

**LQqC:** *The studied problem is important. The experimental results [...] show improvements over current methods, especially at W4A4 quantization levels.*

The most important weaknesses highlighted by reviewers point to:

- the lack of experimental results on the efficiency of our approach in term of memory and speed.

&#8594; *We have run novel experiments following their remarks. More precisely, we have implemented a cuda kernel using Cutlass to show the effect of the rank on the latency of LRC. Additionally, we have shared a table comparing the sizes in GB of the different LLMs considered in this work (unquantized, quantized without additional low-rank terms, and quantized with additional low-rank weight matrices).* 

-  some clarifications on the effect of the calibration datasets. 

&#8594; *We followed the suggestion of the reviewers and added an experiment to compare the performances obtained by Phi-3 when quantized at W4A4 using either wikitext2 or alpaca as the calibration dataset.* 

- the missing proof.

&#8594; *We apologize for this omission. We have added all the proofs in the Appendix.*

We believe we have addressed all the points raised by the reviewers and have already implemented all the changes in our new version of the manuscript.

At the moment, all reviewers except **xtpe** have scored our paper as good (3) in presentation. We have received an average 2.4 and 2.2 grades in soundness and contribution respectively, and we hope our rebuttal alleviates these concerns.

We believe that the very supportive words found in all reviews are not reflected in the current distribution of (fairly low) scores of 5,5,5,5,3. If reviewers agree with our assessment, we humbly ask them to reconsider their score.
</content>
</thread_object>

</thread>


Metareview:
metareview:
Dear Authors,

Thank you for your valuable contribution to ICLR and the ML community. Your submitted paper has undergone a rigorous review process, and I have carefully read and considered the feedback provided by the reviewers.

This work proposes a low-rank approach to enhance the performance of post-training quantization of large language models. The approach is evaluated on some recent language models.

The paper received mixed review scores (6,6,5,5,3). Reviewers pointed out critical issues including (i) over-simplified theoretical analysis, (ii) lack of a time complexity analysis, (iii) limited novelty of the method -- considering similar low-rank correction ideas recently proposed in the LLM quantization literature. Thank you for providing a detailed rebuttal. However, the rebuttal was not convincing enough for three reviewers to increase their scores.

Given the current form of the paper and the reviewer discussion, I regret to inform you that I am unable to recommend the acceptance of the paper for publication at ICLR. I want to emphasize that this decision should not be viewed as a discouragement. In fact, the reviewers and I believe that your work has valuable insights and, with further development and refinement, can make a meaningful impact on the field.

I encourage you to carefully address the feedback provided by the reviewers and consider resubmitting the paper. Please use the comments and suggestions in the reviews to improve and refine your work.

Best,
AC