IntLoRA: Integral Low-rank Adaptation of Quantized Diffusion Models

Authors: Hang Guo, Yawei Li, Tao Dai, Shu-Tao Xia, Luca Benini
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Parameter efficient fine tuning, Network quantization, Diffusion model

Abstract:
Fine-tuning large-scale text-to-image diffusion models for various downstream tasks has yielded impressive results. However, the heavy computational burdens of tuning large models prevent personal customization. Recent advances have attempted to employ parameter-efficient fine-tuning (PEFT) techniques to adapt the floating-point (FP) or quantized pre-trained weights. Nonetheless, the adaptation parameters in existing works are still restricted to FP arithmetic, hindering hardware-friendly acceleration. In this work, we propose IntLoRA, to further push the efficiency limits by using integer type (INT) low-rank parameters to adapt the quantized diffusion models. By working in the integer arithmetic, our IntLoRA offers three key advantages: (i) for fine-tuning, the pre-trained weights are quantized, reducing memory usage; (ii) for storage, both pre-trained and low-rank weights are in INT which consumes less disk space; (iii) for inference, IntLoRA weights can be naturally merged into quantized pre-trained weights through efficient integer multiplication or bit-shifting, eliminating additional post-training quantization. Extensive experiments demonstrate that IntLoRA can achieve performance on par with or even superior to the vanilla LoRA, accompanied by significant efficiency improvements.

Decision: reject
Reviewer scores: 5, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730356025035

thread_type:
official_review

<thread_object>
creator: reviewer_c58y

<content>
summary:
The paper introduces IntLoRA, which uses integral low-rank parameters to adapt quantized diffusion models. First, Adaptation-quantization separation (AQS) is proposed to overcome the difficulty of vanilla zero-initialized LoRA parameters is not quantization-friendly. Then, Multiplicative low-rank adaptation is proposed to enable the fusion of quantized weights and LoRA parameters. Variance matching control is further proposed to make the distribution suitable for log2-quantization. Experimental results show that IntLoRA improves the efficiency of adaptation while maintaining performance.

strengths:
1.	Reducing storage costs is significant for adaptation tasks.
2.	The idea of multiplicative low-rank adaptation is novel.

weaknesses:
1.	The training costs are not reported in the paper.
2.	In section 5.2, the authors mentioned that “IntLoRA is the only one that achieves INT type adaptation weights”, which is not true since EfficientDM also uses quantized LoRA weights.

questions:
See weaknesses above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730429947036

thread_type:
official_review

<thread_object>
creator: reviewer_sb93

<content>
summary:
The paper introduces IntLoRA, a pioneering method for adapting quantized diffusion models to optimize both training and inference efficiencies. It tackles the arithmetic inconsistency between low-rank parameters and pre-trained model weights by employing integer-based low-rank parameters. During training, the method applies quantization to the pre-trained model weights, and post-training, integrates these integer low-rank parameters seamlessly for efficient inference. To facilitate quantization-aware training, the authors propose the Adaptation-Quantization Separation (AQS), which allows for the coexistence of zero-initialized gradients and a quantization-friendly distribution. Additionally, a Variance Matching Control (VMC) mechanism is developed to finely tune the channel-aware variance, ensuring precise adaptation control.  Extensive experimental results indicate that IntLoRA not only competes with but often surpasses traditional methods in performance, significantly reducing memory usage and computational demands.

strengths:
The paper demonstrates a strong and clear motivation by addressing a critical challenge in the domain of machine learning models—specifically, the adaptation of quantized diffusion models using integer low-rank parameters. This innovative approach effectively bridges the gap between maintaining precision and enhancing computational efficiency. Furthermore, the paper successfully resolves the prevalent issue of arithmetic inconsistency between pre-trained weights and adaptation parameters.

weaknesses:
1.	The clarity of some explanations could be improved, particularly concerning why the all-zero distribution is not quantization-friendly. The text should elucidate why such a distribution necessitates a distinct quantizer design at the beginning of tuning. Providing a detailed example or analytical explanation on gradients would greatly enhance understanding, highlighting the specific challenges and potential impacts on the model’s performance during quantization.

2.	In the description of the VMC, the authors introduce a scalar $\alpha$ used as an exponent of $r$ to finely adjust the search for the optimal auxiliary matrix. However, the method for conducting this search remains unclear. It would be beneficial for the authors to specify the objective function used in this search process. Clarifying how $\alpha$ and $r$ are optimized, including the criteria and algorithms employed, would provide a more comprehensive understanding of the method’s effectiveness and implementation.

questions:
1. The proposed method, IntLoRA, demonstrates substantial potential to enhance the performance of quantized models beyond simple adaptation. Integrating IntLoRA’s innovative techniques, such as integer low-rank parameters and network-wide reconstruction calibration, not only maintains but also significantly boosts the efficacy of models operating under quantization constraints.

2. The authors introduce the use of an auxiliary matrix in the AQS. It would be helpful to clarify whether the introduction of this auxiliary matrix introduces additional quantization difficulty like outliers to the pre-trained model weights.

3. The paper overlooks an important baseline in its experimental setup or discussion. The study “EfficientDM” by He et al., 2023, which similarly employs low-rank parameters for fine-tuning pre-trained models, is notably absent. Like IntLoRA, EfficientDM also allows for the merging of low-rank parameters into pre-trained model weights post-training. Including this baseline could enhance the comparative analysis, providing a clearer benchmark of IntLoRA’s performance against existing methodologies with similar strategies.

4. The authors merge integer low-rank parameters with quantized pre-trained model weights after training, a process that might lead to value overflow. It would be beneficial for the paper to address how this potential issue is managed. Specifically, detailing the mechanisms or techniques employed to prevent or mitigate overflow—such as scaling factors, normalization methods, or safeguards within the merging algorithm—would provide a clearer understanding of the reliability of the proposed method in handling data integrity during integration.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730604566933

thread_type:
official_review

<thread_object>
creator: reviewer_qtl5

<content>
summary:
This paper introduces IntLoRA, a method that quantizes the LoRA branch to integers for parameter-efficient fine-tuning (PEFT) in diffusion models. Traditional approaches typically use 16-bit precision for LoRA branches and focus on quantizing only the base model during fine-tuning. However, for inference, these methods often require re-quantization when the LoRA branches are fused with the base model. Unlike prior methods, such as QLoRA, IntLoRA fine-tunes the LoRA branches in a quantization-aware manner. To maintain adapter performance, it incorporates techniques like Adaptation-Quantization Separation (AQS), Multiplicative Low-rank Adaptation (MLA), and Variance Matching Control (VMC), and supports both linear and log2 quantization. Extensive experiments across various downstream tasks—including subject-driven generation, controllable generation, and style customization—demonstrate the effectiveness of IntLoRA in both visual quality and model size reduction.

strengths:
* The paper presents a strong motivation, addressing a key limitation in current quantization methods: the need for re-quantization after applying adapters. This limitation has prevented the wide adoption of quantization methods in diffusion models.
* The concepts of Adaptation-Quantization Separation (AQS), Multiplicative Low-rank Adaptation (MLA), and Variance Matching Control (VMC) are innovative and well-supported by mathematical analysis. Their effectiveness is further validated through comprehensive experimental results.
* The authors' support for log2 quantization, a challenging but highly hardware-efficient setting that further enhances the practical utility of IntLoRA.

weaknesses:
* The paper exclusively focuses on weight-only quantization, where activations remain at 16-bit precision. Since diffusion models are primarily compute-bound, quantizing only the weights does not yield speedups during inference. Additionally, in a weight-only quantization setting, users can choose not to fuse adapters, thereby avoiding re-quantization. As a result, the primary benefit of IntLoRA is just a reduction in adapter size compared to conventional PEFT methods—a relatively minor contribution, given that LoRA adapters are already small.
* he proposed PEFT method does not reduce training memory usage compared to existing PEFT methods (e.g., QLoRA). Furthermore, quantization-aware training of adapters may slow down training and even increase memory usage. The paper should discuss the training speed and memory usage.
* Clarification on storage requirements for $\mathbf{R}$ is necessary. Storing $\mathbf{R}$ after training the adapters would require significant storage, which may limit the method's practicality. If $\mathbf{R}$ is not stored, the authors should explain how it is obtained during inference. Also, note that $\mathbf{R}$ should derive from $\mathbf{W}_{\text{round}}$ rather than $\mathbf{W}$, as only quantized weights are accessible during PEFT.
* Writing:
  * Using "FP" to refer to original LoRA training is ambiguous, as there are low-precision formats like FP8 or FP4. Consider using "16-bit" instead. Note also that QLoRA uses NF4 for weight representation, which is not integer-based. Therefore, Lines 190–202 need to be rewritten.
  * Line 156: The dimension of $\mathbf{B}$ should be $d \times C_{\text{in}}$.
  * Figure 2: The caption lacks detail, and the figure is difficult to interpret.
  * Line 229: Refer to Section 5.3 on how to determine $\mathbf{R}$.
  * Line 265: Need to clarify $\rho$ is the correlation coefficient.
  * Table 4: Correct "FP addtion" to "FP addition".
  * Figure 7: Add axis labels for clarity.

questions:
* Line 252: Could you provide any insights or justification for the assumption that $\mathbf{A}\mathbf{B}$ is orders of magnitude smaller than $\mathbf{R}$?
* Line 260: What is the purpose of a zero-mean adaptation term, and why is it better?
* Table 4: Why does the storage reduction exceed $8\times$ (e.g., $9.8/1.2 > 8$ and $0.34/0.04 > 8$)? If the model is stored in 32 bits, the reduction should be slightly less than $8\times$ due to the additional overhead from zero points and scaling factors. If stored in 16 bits (the preferred format), the reduction should not exceed $4\times$.
* Figure 6: Could you clarify the difference between VMC $\sigma_{\mathbf{R}}$ and the appropriate $\sigma_{\mathbf{R}}$? Additionally, how can one observe the large quantization error from the second image in the second row of the figure?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730777943783

thread_type:
official_review

<thread_object>
creator: reviewer_wq3r

<content>
summary:
The authors propose IntLoRA, which employes INT low-rank parameters to adapt the quantized diffusion models, to address the arithmetic inconsistency between the quantized pre-trained weights and adaptation weights. All the process will operate directly on INT arithmetic in training and merged weights at inference stage will also be on INT format. Apparently, it will speed up both the training and inference stage.

strengths:
1) The presentation of challenges and contributions are clear to me. 
2) The presentation of methods and experiments are clear to me.

weaknesses:
1) Recent works, e.g., EfficientDM and LoftQ, have worked out quantized diffusion models using LoRA. Different from this paper, EfficientDM focus on quantized merged weights at inference stage. Can you compare with it? BTW, I am doubting the importance of the speed-up for the training stage as the pretrained weights is fixed, only small adapters are updated.  Can you show how much speed-up as compared to full-precision? or at least theorotically show the improvement?

2) From my understanding, the proposed methods also can be used in NLP tasks. As the baselines have results on NLP tasks in their papers, why not verify the proposed methods on NLP tasks? Could you show some experimental results on NLP tasks?

3) For diffusion models, seems it have standard/common verification on some tasks, please check EfficientDM paper. As all baselines are implemented by authors, I just doubt the reliablity because of hyper-parameter tuning. Maybe it is better compare with SOTA results in paper.

questions:
Have asked in Weakness section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732197630579

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Author Rebuttal

comment:
## [G1. Remarks by authors]

We would like to express our sincere gratitude to all the reviewers for taking their time reviewing our work and providing fruitful reviews that have definitely improved the paper. We are encouraged that they find our method

- "a strong and clear motivation " (Reviewer wQ3r, qTL5, SB93)
- "addressing a key limitation in current quantization methods" (Reviewer SB93 qTL5)
- "innovative and well-supported by mathematical analysis" (Reviewer qTL5, c58Y)
- "comprehensive experimental results"(Reviewer qTL5)

If you have any further questions, we will actively discuss with you during the author-reviewer discussion period. Due to limited time, the PDF has not been fully revised based on the reviewers' comments, we will upload the fully revised version in a few days.

## [G2. Detailed comparison with EfficientDM] 

Despite both appear "diffusion" and "quantization", our IntLoRA focuses on the **quantization-aware fine-tuning**, instead of the **diffusion model quantization** of EfficientDM.  Importantly, **EfficientDM is more suitable for diffusion modeling quantization**, since directly quantizing the merged weights is sufficient given quantization aims to obtain **one** quantized model. For downstream task adaptation, where **multiple** quantized downstream models are required, the proposed **IntLoRA is more suitable for downstream adaptation**, which allows separately quantized weights and adapters as well as seamless weight merge.

However, considering Reviewer wQ3r, SB93, and c58Y's kind suggestions, we compare our IntLoRA in detail with EffiicentDM as follows.


### [G2-1: Limitations of EffieicntDM for downstream adaptation]

EfficientDM focuses on the quantized merged weights, i.e., $\mathcal{Q}(\mathbf{W+AB})$. An obvious drawback 
is that it still needs to load the large FP32/FP16 weights during downstream adaptation, which is not suitable for user customization on consumer-level GPUs. For example, just loading the FP32 FLUX.1-dev can consumes over 28GB GPU memory. Moreover, $N$ downstream task need to store $N$ large models in EfficientDM, which also leads to noticeable challenge for customized model sharing among users like Civitai. As a comparison, our IntLoRA only need to load the quantized INT4/INT8 model during tuning,  and store 1 quantized pre-training weight & $N$ quantized lightweight LoRA weights to reduce storage cost. 


### [G2-2: Applying EffiicentDM to downstream adaptation]

We also apply the EfficientDM technique to the downstream adaptation task of Dreambooth fine-tuning. We average the results of 15 text-subject pairs for evaluation. The results are shown below.

| setup          | nbits | DINO   | CLIP-I  | CLIP-T  | LPIPS   |
|----------------|-------|--------|---------|---------|---------|
| LoRA-FP       | W32   | 0.4828 | 0.6968  | 0.2954  | 0.8076  |
| EfficientDM   | W8    | 0.4055 | 0.6649  | 0.2855  | 0.8123  |
| IntLoRA-MUL   | W8    | 0.4094 | 0.6630  | 0.2857  | 0.8048  |
| EfficientDM   | W4    | 0.3733 | 0.6555  | 0.2847  | 0.8085  |
| IntLoRA-MUL   | W4    | 0.3749 | 0.6599  | 0.2867  | 0.8011  |

It can be seen that our IntLoRA-MUL achieves better performance than EfficientDM. Moreover, IntLoRA also enjoys benefits described in the above [G2-1] 

### [G2-3: Applying IntLoRA to diffusion model quantization]

Since our IntLoRA can potentially be used in diffusion model quantization as well, we also apply our IntLoRA to the quantization experimental settings of EfficientDM using its official code. In detail, we use the LDM-4 on ImageNet256$\times$256 image generation tasks, and train the ddim_step=20 model with 500 training epochs. The results are as follows. We keep other hyper-parameters the same except replacing the original EfficientDM LoRA layer with our IntLoRA layer.

| methods         | nbits(W/A) | IS      | FID   | sFID   | precision | recall |
|-----------------|------------|---------|-------|--------|-----------|--------|
| EfficientDM    | W4A4       | 178.20  | 13.42 | 26.67  | 0.70      | 0.58   |
| IntLoRA-SHIFT  | W4A4       | 116.50  | 20.20 | 26.79  | 0.63      | 0.58   |
| IntLoRA-MUL    | W4A4       | 199.20  | 10.43 | 24.02  | 0.79      | 0.55   |


It can be seen that our IntLoRA-MUL achieves even better performance than the EfficientDM on weight-bits-4&activation-bits-4 setups.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper titled "IntLoRA: Integral Low-rank Adaptation of Quantized Diffusion Models" introduces IntLoRA, a novel parameter-efficient fine-tuning (PEFT) method tailored for quantized diffusion models. The primary innovation lies in utilizing integer-based low-rank adaptation parameters, enabling seamless integration with quantized pre-trained weights through integer multiplication or bit-shifting. This approach aims to enhance both training and inference efficiencies by reducing memory usage, storage costs, and eliminating the need for additional post-training quantization (PTQ). The authors present comprehensive experimental results demonstrating that IntLoRA achieves performance comparable to or surpassing traditional LoRA methods while offering significant efficiency improvements across various downstream tasks, including computer vision and natural language processing (NLP).

Despite the promising contributions, several critical concerns undermine the paper's suitability for acceptance. Firstly, the comparison with existing methods such as EfficientDM and LoftQ is insufficiently addressed, particularly regarding the applicability and performance on standard benchmarks. Reviewer wQ3r highlighted the lack of comparisons on established NLP tasks and questioned the reliability of the experimental setups, suggesting potential biases due to hyper-parameter tuning. Although the authors provided additional experiments and clarifications during the rebuttal, the persistent concerns from Reviewer wQ3r regarding the method's novelty and practical significance remain unresolved. Additionally, the paper's focus on weight-only quantization limits the claimed inference speedups, as activation quantization is not comprehensively explored. These weaknesses, coupled with the unresolved doubts about the method's generalizability and comparative advantage, lead to the recommendation to reject the paper in its current form.