Low Rank Quantization Adaptation for Large Language Model

Authors: Xiwei Xu, Yuexiao Ma, Wenting Lin, Yuhang Wu, Zelan Yang, Wanchen Sui, Shen Li, Yong Li, Xiawu Zheng, Fei Chao, Rongrong Ji
Primary area: foundation or frontier models, including LLMs
Keywords: Quantization, Low-Rank Adaptation, LLM

Abstract:
As the parameters of Large Language Models (LLMs) increase, quantization has emerged as a potent strategy for model compression and acceleration. Concurrently, Low-Rank Adaptation (LoRA) has been recognized as an effective method for enhancing LLM performance. However, integrating LoRA with quantization presents significant challenges, particularly in preserving the quantization format after model optimization. In this paper, we introduce Low rank Quantization Adaptation (LoQA) for LLM, a novel approach that effectively fine-tunes holistic quantization parameters. Specifically, we first propose a new perspective of quantization operator, which is compatiable with LoRA and mathematically equivalent to the original operator. In this way, all the parameters (scale and zero point) are finetuned simultaneously, and thus yields notable improvements in model performance.Thanks to the expanded optimization landscape, LoQA is broadly applicabile to various Post-Training Quantization (PTQ) techniques, ensuring better generalizability in practical deployments. To maintain the stability of the optimization, we further propose a LoRA scaling strategy that leverages quantization data to adjust the norm of the low rank adaptation, regulating the speed of convergence in optimization and preventing inappropriate LoRA scaling, which could lead to overfitting or underfitting. Compared to existing methods, LoQA consistently achieves performance gains across a wide range of models, proving its effectiveness and adaptability.

Decision: reject
Reviewer scores: 5, 5, 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1730114682105

thread_type:
official_review

<thread_object>
creator: reviewer_2qnk

<content>
summary:
This paper introduces Low Rank Quantization Adaptation (LoQA) for LLMs, a novel parameter-efficient fine-tuning method for quantized models. LoQA aims to reduce memory consumption across model weights, gradients, and optimizer parameters. It includes two core modules: **HQ-LoRA** (Holistic Quantization Low-Rank Adaptation) and **QBAS** (Quantized Bit-Aware Scaling), aligning quantization parameter granularity with LoRA parameters in group quantization.

strengths:
- The proposed method is well-motivated and sounded.
- This approach is straightforward yet effective, especially in ultra-low bit-width scenarios.
- Code is provided for reproducibility.

weaknesses:
- **Limited novelty**: The method’s innovation is incremental, building on QA-LoRA and closely resembling group quantization applied to LoRA parameters. In Sec 3.2, LoQA uses two LoRA components, whereas QA-LoRA requires only one, which may lead to a higher parameter count.
- **Experimental limitations**:
  - LoQA does not demonstrate clear improvements on LLaMA 2, and for LLaMA 3, only training loss rather than performance results is presented. Providing promising results on more advanced models would strengthen the paper’s contribution.
  - Important baselines, such as **IR-QLoRA** and **QLoRA**, are absent in Tables 5,7,9, affecting comparison fairness.
  - While the authors claim broad applicability of LoQA across post-training quantization techniques, only **GPTQ** settings are evaluated.
- **Poor presentation and writing**:
  - Figure 2 lacks clarity in distinguishing between QA-LoRA and LoQA.
  - In Sec 3.3, the rationale for setting $maxq$ to $2^{N-1}$ is not sufficiently analyzed.
  - In some tables (e.g., Tables 4, 7, 8), entries are missing bold formatting to indicate the best results.
  - The LoRA rank used in the compared methods is not clearly specified.
  - The "4+16 bit" setting is not formally defined.
  - "Llama-7b" and "Llama-7B" are inconsistently referenced throughout the paper.
  - "Post-fine-tuning" in line 269 is incomplete.

questions:
Please refer to the Weaknesses above for questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730174382021

thread_type:
official_review

<thread_object>
creator: reviewer_jkth

<content>
summary:
This paper proposes a novel quantization-aware, parameter-efficient tuning method for large language models (LLMs) based on QA-LoRA. The approach reparameterizes both scaling and zero factors to mitigate quantization errors. Experimental results demonstrate its effectiveness.

strengths:
1. Deployment is a crucial consideration for large language models (LLMs), with quantization-aware training being one of the key challenges in their deployment.

2. The proposed method is simple yet promising, as indicated by the experimental results.

weaknesses:
1. While I understand the core idea the author aims to convey, I believe the writing could be improved by providing a more general analysis of approaches to quantization-aware (QA) training or reparameterization of quantization parameters. This would be more important to the community and the contributions will be more important.
2. The contributions appear to be incremental.
3. Figure 2 could be enhanced: it currently labels the LoRA parameters 𝐴 and 𝐵 for QLoRA, but lacks annotations for QA-LoRA and the proposed method. Additionally, the concept of Holistic LoRA is not clearly conveyed.
4. This method seems have limitations which only support group-wise quantization.

questions:
1. Any inference time results?
2. other questions please refer to the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730400569708

thread_type:
official_review

<thread_object>
creator: reviewer_n8i8

<content>
summary:
This paper aims to enable efficient LLM fine-tuning by combining quantization with LoRA. Rather than keeping LoRA inference in FP/BF16 during the forward pass, it proposes a holistic quantized LoRA approach where LoRA is fused into the quantization set to enable fully quantized inference. Additionally, the paper introduces a method to determine the LoRA scaling factor based on the target bit precision.

The proposed method achieves higher fine-tuning performance and training efficiency compared to QLoRA and QA-LoRA at 4/3/2-bit precision.

strengths:
- The related work and motivation are clearly presented, making the problem the paper addresses easy to understand.
- The holistic quantized LoRA approach follows a similar direction to QA-LoRA, aiming to achieve fully quantized inference by fusing LoRA. The paper introduces a way to learn both the scaling and zero-point parameters, which adds flexibility to the method.

weaknesses:
- The holistic quantization method formulation is somewhat difficult to follow. Figure 2 is overly simplified, and would benefit from revision to visually clarify the equations in Section 3.2.
- Experiments are limited to the LLaMA family, and there is a lack of results on LLaMA-3. If dataset is an issue, it may be preferable to use more recent open-source datasets (e.g., SlimPajama) instead of the relatively old and small sample Alpaca dataset. Alternatively, comparing performance on domain-specific fine-tuning tasks (e.g., GSM8K, OASST1) as done in LoftQ/LQ-LoRA could provide additional insights.
- The main contribution of this paper appears to be preserving the quantization set in inference. However, the proposed method lacks significant innovation compared to QA-LoRA, and there is no report on the actual efficiency gains achieved during fully quantized inference. Section 4.3 only briefly mentions the potential for kernel optimizations, which seems insufficient given that fullly quantized inference is the main focus of this paper.

In summary, while the proposed method consistently outperforms QA-LoRA and addresses an important problem, the limited experimental results and unclear explanations of key methods indicate room for improvement.

questions:
- In Section 3.2, the proposed method is still not entirely clear. On Line 249, what exactly do you mean by “two LoRA variants”?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730632248785

thread_type:
official_review

<thread_object>
creator: reviewer_45yq

<content>
summary:
The paper introduces Low-Rank Quantization Adaptation, called LoQA, a new approach for fine-tuning large language models (LLMs) while preserving quantization. LoQA consists of two techniques: (1) Holistic Quantization Low-Rank Adaptation (HQ-LoRA): This approach fine-tunes all quantized parameters, enabling broader optimization without losing the quantized model structure. (2) Quantized Bit-Aware Scaling (QBAS): This technique dynamically adjusts scaling factors based on bit-widths, enhancing performance stability across various quantization levels. LoQA is tested against state-of-the-art quantization methods and is effective across different model sizes and bit configurations. It yields significant improvements, particularly under ultra-low bit-width scenarios.

strengths:
+ LoQA introduces a novel quantization-aware fine-tuning method that can support quantized LLMs to effectively balance model compression with performance.
+ Experiments show that LoQA consistently outperforms previous methods in accuracy and efficiency, even in ultra-low bit-width configurations.
+ The method supports multiple bit-widths and integrates smoothly with existing post-training quantization methods, making it adaptable to a wide range of applications.
+ LoQA is evaluated using the MMLU and common sense QA tasks, with consistent results.

weaknesses:
- LoQA requires more training time and memory compared to some baseline methods, which may hinder its practicality in resource-constrained environments (e.g., compared with QA-LoRA)
- The study did not use the latest datasets or the newest training paradigms, which might limit the generalizability of the results to other, more current LLMs (as also mentioned in Appendix).

questions:
Would you please describe how LoQA could be implemented in production?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730646666439

thread_type:
official_review

<thread_object>
creator: reviewer_fbjk

<content>
summary:
This paper extends the QA-LoRA method by introducing Low-Rank Quantization Adaptation (LoQA) to enhance the fine-tuning of large language models (LLMs) within a quantized framework. LoQA addresses the integration challenges between quantization and Low-Rank Adaptation (LoRA), proposing Holistic Quantization Low-Rank Adaptation (HQ-LoRA), a quantization-compatible adaptation technique that allows finetuning of all quantization parameters (including scale and zero points) resulting in improved model performance. Additionally, the paper introduces Quantized Bit-aware scaling (QBAS), which adjusts LoRA scaling to account for the impact of integer weights at varying bit-widths. Overall, LoQA extends the QA-LoRA paper by modifying not only the quantization zero points (as in QA-LoRA) but also the quantization scale.

strengths:
Strengths:

-Considering the quantization scaling as well during finetuning gives much more representation power compared to earlier approaches like QA-LoRA, which focus only on quantization zero points. Also, as demonstrated by empirical results, it usually results in better generalization across diverse set of models and bit budgets.

-LoQA especially outperforms previous methods on low-bit budgets, which would make it useful for ultra resource-efficient settings.

-In the 4bit regime, LoQA seems to perform competitively to algorithms like IR-QLoRA that use a mix of 4/16 in their inference deployment

-Given that it doesn’t carry 16bit LoRA parameters to the inference phase (as opposed to methods like QLoRA), it would bring considerable inference efficiency compared to "4+16" type of methods.

weaknesses:
In general, this method is interesting - however, in terms of novelty,  this work is indeed an incremental work on top of the QA-LoRA work. Moreover, this paper could benefit from improvements in the presentation and description of the experiments.

Weaknesses:

-The paper presentation could be improved, especially in the experiment section. Implementation details are underspecified in many parts. Given that this research effort is based on empirical results, this lack of experimental setting would make it hard to reproduce the results for the community.

-For experiments comparing LoQA with QA-LoRA (as well as other similar methods), the corresponding ranks of the methods are not reported in most cases, which makes the reader question if the improvement could be due to using more parameters by LoQA or not.

-There’s lot of focus on MMLU evaluation after finetuning on alpaca/Flan datasets, which might not be representative of how this algorithm would actually work for other type of tasks/datasets. The only exception, Table 5, focuses on commonsense reasoning, which the performance is very close to QA-LoRA, while it uses half of your training complexity by having a single set of LoRA parameters for zero points.

-Table captions in many cases are not "self contained" and often have a very general caption, which make it hard to go through different tables and understand the results.

questions:
-Regarding QBAS: the scaling that you proposed is then being multiplied by a hyperparameter. Are you using the same $alpha$ across different bits?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732502207540

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We sincerely thank all the reviewers and area chairs for their time and efforts in reviewing our paper. We appreciate the insightful and constructive comments provided by the reviewers. We have addressed each of the points raised and made revisions accordingly. Below is a summary of the main contributions:

**Novelty**

We propose a novel and efficient fine-tuning method, LoQA, for quantized large language models. This method seamlessly integrates Low-Rank Adaptation (LoRA) with group-wise quantization into a unified framework. Compared to previous work, LoQA makes two fundamental contributions: (1) it introduces the novel concept of merging LoRA with scale quantization parameters, and (2) it introduces QBAS to address the abnormal magnitude issues in learning scale parameters. These advances represent significant improvements in the versatility and accuracy of such methods.

**Strong and Comprehensive Empirical Results**

The experimental results demonstrate consistent improvements across different LLM architectures and quantization bit-widths. Our experiments cover a variety of architectures (e.g., LLaMA1, LLaMA2, LLaMA3, OPT) and model sizes, and showcase results across different datasets (Alpaca, Flan v2, GSM8K) and bit-widths. LoQA exhibits its adaptability across models and sizes, with significant improvements in both Commonsense QA and MMLU accuracy. Despite using only half the number of parameters, LoQA achieves a substantial accuracy boost. Extensive experiments show that LoQA consistently outperforms previous fine-tuning methods that retain quantized formats and, in many cases, matches the performance of state-of-the-art 4+16 bit methods. Notably, in ultra-low bit-width scenarios, LoQA's effectiveness is even more pronounced, with its 2-bit version outperforming the current 2+16 bit state-of-the-art method by 4.7%, and even surpassing the original 16-bit model.

**Inference Efficiency**

LoQA preserves the inference speed advantages of quantization by merging the low-rank adaptation weights, while also being compatible with the latest inference engines such as Marlin and BitBlas. This results in significant improvements in inference speed, which is crucial for deploying LLMs in resource-constrained environments.

We also appreciate the constructive suggestions and questions raised by the reviewers, which have led to fruitful discussions and additional experiments. We have made appropriate revisions based on these suggestions, with all changes marked in blue for easy reference. We believe these revisions further strengthen the quality of the paper. Below are the major revisions:

1.We have re-drawn Figure 2 to improve clarity, and added additional explanations for the "LoRA-Merge" section with different color blocks to aid reader comprehension.
2.We revised the titles of each table to make them more self-contained and reader-friendly, and added explanations for parameters like “rank” and “4+16-bit” to prevent any misunderstandings.
3.We added comprehensive implementation details in Appendix A.
4.We included a comparison of Commonsense QA results on the Alpaca dataset to further highlight the accuracy advantages of our method.
5.We included evaluation results using domain-specific datasets GSM8K on LLaMA3.
6.We added Appendix D, which provides a visual analysis of the QBAS method, comparing magnitude changes before and after using QBAS to clearly demonstrate its effect.
6.We included Appendix F, where we tested our method on Flan-v2 with OPT-6.7B, expanding our evaluation beyond the LLaMA series.
7.We added Appendices G and F, which include detailed information on training and inference speed experiments.

Finally, **we have addressed all raised questions and concerns**, and detailed responses to individual reviewers are provided below. We look forward to any additional comments.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes Low-Rank Quantization Adaptation (LoQA), an approach to fine-tuning LLMs within a quantized framework that introduces two primary techniques: Holistic Quantization LoRA (HQ-LoRA) and Quantized Bit-Aware Scaling (QBAS), which aim to enhance model performance and efficiency during quantization. The research demonstrates consistent improvements, particularly in ultra-low bit-width scenarios, with most reviewers acknowledging its potential to balance model compression and performance (45yq, jKth, 2qNk). Reviewers noted strengths including the method's ability to fine-tune quantization parameters comprehensively, outperform previous approaches like QA-LoRA, and show promising results across different model sizes and bit configurations (FBJK, 45yq, n8i8).

However, the paper exhibits several significant weaknesses that led to its negative rating (all reviewers gave a 5). Reviewers consistently pointed out limited novelty, with the work being largely incremental to previous research (2qNk, jKth, FBJK). Experimental limitations were prominent, including underspecified implementation details, restricted dataset usage (primarily Alpaca/Flan), and lack of comprehensive comparisons with other SOTA (FBJK, n8i8, 2qNk). Presentation issues were also critical, with reviewers highlighting unclear figures, inconsistent terminology, missing context in tables, and insufficient explanation of key methodological components (n8i8, 2qNk, jKth). Additionally, concerns were raised about the method's training complexity, limited generalizability, and the absence of clear inference time results (45yq, jKth, 2qNk).