IntLoRA: Integral Low-rank Adaptation of Quantized Diffusion Models
Authors: Hang Guo
Yawei Li
Tao Dai
Shu-Tao Xia
Luca Benini
Venue: ICML 2025 poster (2025)
Keywords: Low-rank Adaptation
Network Quantization
Diffusion Models
Primary area: applications->computer_vision
TL;DR: This paper presents the integral low-rank adaptation which can adapt the quantized diffusion models by int-multiplication or bit-shifting.

Abstract: Fine-tuning pre-trained diffusion models under limited budgets has gained great success. In particular, the recent advances that directly fine-tune the quantized weights using Low-rank Adaptation (LoRA) further reduces training costs. Despite these progress, we point out that existing adaptation recipes are not inference-efficient. Specifically, additional post-training quantization (PTQ) on tuned weights is needed during deployment, which results in noticeable performance drop when the bit-width is low. Based on this observation, we introduce IntLoRA, which adapts quantized diffusion models with integer-type low-rank parameters, to include inference efficiency during tuning. Specifically, IntLoRA enables pre-trained weights to remain quantized during training, facilitating fine-tuning on consumer-level GPUs. During inference, IntLoRA weights can be seamlessly merged into pre-trained weights to directly obtain quantized downstream weights without PTQ. Extensive experiments show our IntLoRA achieves significant speedup on both training and inference without losing performance.

IntLoRA:
Integral Low-rank Adaptation of Quantized Diffusion Models
Hang Guo Yawei L
Tao Dai
Shu-Tao Xia 14
Luca Benini?
Abstract
(a) Previous pipeline of tuning-theen-quantzatto
Fine-tuning pre-trained diffusion models under limited budgets has gained great success, In par- ticular, the recent advances that directly fine-tune the quantized weights using Low-rank Adaptation (LoRA) further reduces training costs. Despite these progress, we point out that existing adapta- Specifi-
Quantized Weights
BIPP16L
Merged Weights W W AB (FF16)
Quzntized Merged Waights W((INT9)
Wingle Morge
PTG
(INTA)
AGP16)
(b) Our integer adaptation w/o additional PTQ
tion recipes are not inffrencceefffciien cally, additional post-training quantization (PTQ) on tuned weights is needed during deployment, which results in noticeable performance drop when the bit-width is low. Based on this observa- tion, we introduce IntA,,, which adapts quan- tized diffusion models with integer-type low-rank parameters, to include inference efficiency during tuning. Specifically, IntLoRA enables pre-trained weights remain quantized during training, facil- itating fine-tuning on consumer-level GPUs. Dur- ing inference, IntLoRA weights can be scamlessly merged into pre-trained weights to directly obtain quantized downstream weights without PTQ. Ex- tensive experiments show our IntLoRA achieves significant speedup on both training and inference without losing performance. Code is available https//ggituuh.com//ssguu/o
Quantized Weights
BLITTA
Quantized
Int- Int-Muitiplication Bit-shifting
W(NT4)
Weights (INVT4)
ACDT4
Figure I: (a) The arithmetic inconsistency between the pre- trained and adaptation weights leads to the merged weights still in FP16 6. Consequently, additional PTQ is needed for low-bit inference. (b) Our IntLoRA allows to work directly on INT4 arithmetic, ensuring the merged weights seam- lessly in INT4 format and streamlining the whole process.
els for downstream tasks poses challenges on consumer- level GPUs. For instance, only loading the FP32 FLUX.1- dev (BlackForestLabs, 2024) weights into GPUs can con- sume over 24GB of memory, let alone subsequent fine- tuning. Therefore, the huge fine-tuning costs hinder person- alized diffusion model customization.
To facilitate efficient training, recent advances have intro- duced parameter efficient fine-tuning (PEFT) (Houlsby et al., 2019; Jia et al., 2022) techniques, such as LoRA (Hu et al., 2021), fine-tune a limited number of parameters. With the reduced gradient and optimizer states, they can achieve comparable
1. Introduction
Recently, large-scale text-to-image diffusion models (Rom- bach et al., 2022; Saharia et al., 2022; Podell et al., 2023) have shown promising capabilities for image generation. Taking advantage of the strong generative prior of pre- trained parameters, range of downstream adaptation ap- plications have emerged, such as subject-driven genera- tion (Ruiz et al., 2023), style-customized generation (Sohn et al., 2023), and controllable generation (Zhang et al., 2023). However, fully fine-tuning large pre-trained mod-
better adaptation performance than fully fine-tuning. More recently, some works (Dettmers et al., 2024; Qin et al., 2024) have successfully married PEFT and network quantization to allow the low-rank adaptation directly on the quantized weights (as shown in Fig. (a)). Through reducing the bit-widths, the GPU costs during fine- tuning are further decreased.
'Tsinghua University 'ETH Zürich "Shenzhen University PPng Cheng Laboratory. Project Lead. 'Correspondence Tao Dai
Although the reduced training costs have facilitated user customization, obtaining an inference-aware tuning recipe remains an open challenge. Specifically, existing methods predominantly employ floating- (e.g., FP1 6) low-rank parameters during training, as a result, it is inevitable to convert the quantized pre-trained weights back to FP1 6 for
to: Yawei Li <yawei.li @ <daitao. @aail@@mam..........
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

IntLoRA: Integral Low-rank Adaptation of Quantized Diffusion Models
Performance subject-drivvn gener ation
Performance controllable generation
2. Related Work
+37Q
02
Int OelAOkO
Parameter-efficient fine-tuning of diffusion models. In or- der to reduce the fine-tuning cost of large models. parameter- efficient fine-tuning (PEFT) has recently gained great inter- ests (Lian et al., 2022; Chavan et al., 2023; Li & Liang, 2021; He et al., 2021; Jie & Deng, 2023). For example, prompt- based methods (Jia et al., 2022) append learnable prompts to modify the input space. Adapter-based methods (Houlsby et al., 2019: Chen et al., 2022) employ additional bottleneck structures as bypass branches for adaptation. Moreover, LoRA (Hu et al., 2021) adopts low-rank matrices to learn the weight updates for downstream tasks. In this work, we mainly focus on LoRA since it has been widely applied in diffusion model fine-tuning and can be merged into pre- trained weights without increasing inference costs.
030
220le
37
158
Figure 2: The utilization of PTQ on the downstream merged weights leads to severe performance degradation under low bit-width quantization.
arithmetic consistency to merge low-rank weights into pre- trained weights. During test-time, this pipeline necessitates additional post-training quantization (PTQ) on the FP 6 merged weights for accelerated inference, which is pipeline- complicated and incurs significant performance drop when the bit-width is low (see Fig. 2).
Network quantization of diffusion models. Quanti- zation (Nagel et al., 2021) is an effective technique to speed deep-learning models and can be categorized into quamtization-awaan training (QAT) (Jacob et al.. 2018; Li et al., 2024; 2022; Xu et al., 2023a) and post-training quan- tization (PTQ) (Wang et al., 2023; Nahshan et al., 2021; Li et al., 2021; Wei et al., 2022; Liu et al., 2023; Huang et al., 2024a). In the context of diffusion model quantiza- tion, existing works mainly focus on PTQ because of the significant overhead of retraining diffusion models. For example, PTQ4DM (Shang al., 2023) makes the first at- tempt to quantize diffusion models to 8 bits. After that, Q-Diffusion (Li et al., 2023) further achieves improved per- formance and lower bit-width. EfficientDM (He et al., 2023) introduces LoRA to fine-tune the pre-trained model to allow comparable performance with QAT. TFMQ-DM (Huang et al., 2024b) proposes to quantize the time-embedding layer individually for better performance.
To address these challenges, a potential solution is to also transfer the adaptation weights to integer arithmetic. In this way. all weights during fine-tuning are in integers, thus ensuring the merged weights naturally being quantized. De- spite these promising properties, it is non-trivial to accu- rately quantize the low-rank weights. For example, while zero initializing low-rank weights are advantageous for fine- tuning (Hu et al., 2021), it poses quantization challenges due to substantial quantization errors from small values. Fur- thermore, the additive form of the original LoRA forces the pre-trained and adaptation weights to share the same quan- tizer for seamless weight merging, which restricts available parameter space during fine-tuning.
In this work, we propose IntLoRA, which achieves integral low-rank parameters for both training and inference efficient diffusion models. In detail, we introduce the Adaptation- Quantization Separation (AQS) technique, which employs a task-agnostic auxiliary matrix to enable quantization- friendly low-rank parameters without disrupting the gradient trajectory of the original LoRA. Additionally, we present the Multiplicative Low-rank Adaptation (MLA), which re- formulates the mathematical structure of LoRA from addi- tion to multiplication. This remains mathematically equiv- alent to the original but allows for independent optimiza- tion of adaptation weights. Furthermore, we develop the Variance Matching Control (VMC) to align the pre-trained and auxiliary matrices. For implementation, we provide two versions, i.e., IntLoRAMUL, and IntLORASHIFT- The IntLoRAMU, learns quantized low-rank parameters and can be seamlessly merged through integer multiplication, while LntLoRRAsiie introduces log2-quantization and operates by bit-shifting the quantized weights for downstream adapta- tion. We evaluate our IntLoRA on various diffusion person alization tasks. Extensive experiments show that IntLoRA presents impressive efficiency and performance.
Joint PEFT and quantization for efficient fine-tuning. Benefiting from the scaling law, the pre-trained models have become increasingly large, which makes even loading mod- els challenging. To allow fine-tuning on consumer-level GPUs for user customization, some work attempts to ap- ply PEFT techniques directly on the quantized pre-trained weights Specifically, QLoRA (Dettmers et al., 2024) pro- poses to quantize the LLMs before fine-tuning the LLMs with LoRA. Despite the reduced GPU usage during train- ing due to the import of only the quantized model, QLORA does not maintain quantized at inference since the quan- tized weights need to be converted to FP1 6 again so as to be merged with the LoRA weights QA-LoRA (Xu et al., 2023b) develops group-wise quantization through sharing parameters across channels but at the cost of impairing the adaptation ability. IR- IR-QLoRA (Qin et al., 2024) analyzes the entropy loss of quantization from an information theory view, but it also needs to convert the quantized weights back to FP16 6 during inference.
2

---

IntLoRA: Integral Low-rank Adaptation of Quantized Diffusion Models
3. Preliminary
requires separately designed quantizer at the beginning of tuning, since the scaling factor = 0 leads to infinite X in Eq. (1). Second, the vanilla LoRA merges the FP 1 AB and W using addition. When both AB and W are quantized, it is essential to ensure that they share identical quantization parameters to enable PTQ-free weight merging, This requirement leads to constrained parameter space, thus limiting the adaptation ability.
The LoRA (Hu et al., 2021) introduces a low-rank matrix AW to learn the weight increments for adapting the pre- trained weights W Co to downstream tasks. In implementation, the AW is formulated as the matrix mul- tiplication of two low-rank matrices A E RCCu xd and B E Idd where the inner dimension d is the pre- defined rank. During fine-tuning, the pre-trained weight w is frozen and only the low-rank A, B are trainable. Since & min (Ci;; Cout]. ). the number of trainable parameters can be very small compared to full fine-tuning, thus reduc- ing the GPU footprint of gradients and optimizer states. The output during downstream fine-tuning is calculated as y= Wx + X- (AB)x, x where À is the LoRA scale to adjust the control strength. During inference, the task-specific AB can naturally be merged into the pre-trained weights, i.e., W
4.1. Integral Low-rank Adaptation
To address the above challenges, we propose IntLoRA to operate adaptation on the integer arithmetic. The overall pipeline is shown in Fig. 3.
ilaptation-quantzattiio separation. The vanilla LoRA adopts zero initialization on the adaptation parameter AB. Although this strategy can improve performance, the all- zero distribution is not quantization-friendy. To allow ac- curate quantization while maintaining the correct gradient, we propose the mechanism. The key observation is that the adaptation requires gradients from zero-initialized weights while the quantization does not, Therefore, we can split the adap- tation weights into the gradient-enabled zero part and the gradient-fie free nonzero part. Formally, let R be the auxiliary matrix to serve as the nonzero part, Q be the quant-dequant operator, then our AQS can be formulated as:
W + X- AB, without increasing additional costs.
Even though the LoRA can alleviate training costs through reduced gradients and optimizer states, it still needs to load huge FP16 pre-trained weights. Given the increas- ing pre-trained model size, it becomes impractical to only use LoRA A to fine-tune the diffusion models on consumer- level GPUs To further reduce the training memory, recent advancements (Dettmers et al., 2024; Xu et al., 2023b; Qin et al., 2024) have introduced network quantization to allow direct fine-tuning on the integer weights. Formally, given a tensor X, the target bit-width b, the quantization process can be defined as:
Quantization MllantiztioonSSpprrttion (AQS)
w Q(W sg(R)] + sg(R) + ABB
(2)
x 8- (elip(( 1+
2, 2, 2hh - 1) 1)- A 8. -x round z),
where sg(-) denotes the stop gradient operation. Thanks to the AQS, the AB be vero-initialized for the same gradient as the original LoRA, while sg( (R) + AB facilitate subsequent quantization by specifically designing the aux- iliary matrix R as discussed in Sec. 5.4. In the following part, we will ignore the sg(+) notation for clarity.
(1)
where [.] is the round function, s max miimin(XX is the scaling factor, and
-1
min( (X) is the zero-point.
Despite current methods allow user to train customized mod- els under a low memory budget, they all require additional PTQ on the fine-tuned weights for fast inference, which leads to noticeable performance degradation when the quan- tization wittt is low.
Multiplicative low-rank adaptation. The vanilla LoRA employs additive form W + AB for weight merge. How- ever, it is difficult to seamlessly fuse the quantized W and AB when they are quantized by independent quantizers. To this end, we propose Multiplicative Low-rank Adap- tation (MLA) to rewrite the form of the original LoRA into a maantzationn-friend]] multiplication form. Specifi- cally, denote the quant-dequan results as Q(W -R) s (W round 2), then the MLA can be derived follows:
4. Methodology
In this work, we aim to remove the additional PTQ of the merged weights by introducing integer-type low-rank pa- rameters during fine-tuning. In this way, both the AB and W are in the same arithmetic type, thus ensuring the merged W' is naturally already quantized. However, several tech- nical challenges arise when transferring LoRA to integer arithmetic. First, the AB in the original LoRA is zero- initialized to ensure the behavior of the model is similar to the pre-trained one at the beginning of training. Although helpful for mee-tuning, this initialization complicates the quantization process. For instance, the all-zero distribution
W = Q(W W - R) + R + AB
( round
2) + R +
AB
= I W round
+ (R+ AB) 0 (Wround round - 2),
(3)
where the task-specific adaptation term is trainable and will be quantized, and the pre-trained term is already in integer type and is shared across tasks. I an all-one matrix. The
3

---

IntLoRA: Integral Low-rank Adaptation of Quantized Diffusion Models
Before Tuning: preprocess the weights
After Tuning: direct weight merginn without P PQ
Pre-trained Weights
AQS
AQS Weights W - R (FP1 6)
MLA
W (Pe16)
Pre- Pre-trained Term W ound -z (NNT9)
TSPER
VMC A uxiliarr Matrix R (e21s)
X
Int-multiply
Quantized Merged Weights W (IN14)
A Auxiliary Matrix
or Bit-shifting
R (#P16)
A daptation Term O(R+AD)
Tak-1
Low-rank A dappaation W Weights
AQS
AQS A Adaptation Weights
MLA
AB (FP16)
AB + R (#226)
Figure 3: Before tuning, we propose the Adaptation Quantization Separation (AQS) to incorporate auxiliary matrix into pre-trained weights and low-rank weights for zero-initialized but quantization-freend) distribution. Then, the Multiplicative Low-rank Adaptation (MLA) is used to reformulate additive LoRA into the product of the "pre-training term" and the "adaptation term". At last, we introduce the Variance Matching Control (VMC) to adjust the distribution of the adaptation term by modulating the auxiliary matrix. After tuning, we use serdwarr-friendll integer multiplication or bit shifting to directly generate quantized merged weights without additional PTQ. The detailed algorithm is given in Appendix A.
operator @ denotes the Hadamard product of two matrices. The proposed MILA is mathematically equivalent to its addi- tive counterpart, while is more quantization-fiin frrindlly since avoids the shared quantizer of pre-trained and adaptation weights. It is noteworthy that the adaptation term is still in FP16 6 at this step, and we will detail its quantization strategies in Sec. 4.2.
converges to zero when OR approaches infinity, namely,
R W R
8 W W
E lim on
I
=E
E
lim On
0.
R
(5)
On the other hand, setting OR too large can also lead the Q(W R) uncorrelated the original W, ie, namely,
Variance matching control. One opportunity brought from the multiplicative form in Eq. (3) is that we can apply the log2-quantization on the adaptation term, thus allowing more efficient bit-shifting on the pre-trained term. How- ever, log2-quantization is notoriously more difficult than common uniform quantization (Nagel et al., 2021) and re- quires appropriate distribution properties, e.g., most values concentrated around zero to allow for the utilization of as many quantization bins as possible on the logarithmic scale. Here, we revisit the adaptation term in Eq. (3) aiming to find useful mathematical insights. Given the AB is orders of magnitude smaller than R (the justification shown in Ap- pendix F), we approximate the adaptation term in Eq. (3) by removing AB from it, namely,
lim (Q(WW R), W) lim OR
dw
0,
oR
+ R
(6)
where the p(.)-) denotes the correlation coefficient. Eq. (6) indicates that over-large OR makes it diffieult re- construct the original signal W through dequantizing the Q(W - R) due to the low correlation. In short, it is impor- tant to choose an appropriate OR to strike a balance between quantization difficulty and information retention. We also give the visualization of this choice dilemma Fig. 8. To this end, we propose the Variance Matching Control (VMC) mechanism. Specifically, we first multiply R. by the vari- sw. E Cous for rough alignment from R .BB
ance ratio to the scale of W. After that, we introduce scalar a as an exponent of r, i.e., r" to fine-grain the search for the optimal R* As a result, the trix can be denoted as R = p R, and we can use this to obtain the distribution suitable for eng--quanizztton. Since rd can be shared across tasks, it is only of negligible cost, In addition to the dR, observe the distribution shape of R also has an effect on performance, and we give a detailed discussion in Sec. 5,4. It should be noted that the R can be online generated during fine-tuning using the distribution statistics and fixed random seed, thus avoiding the need to store its FP 1P16 parameters.
R
R
8 II
round
8 W
round 2)
(4)
auxiliary ma-
R
W
W R
W
From this derivation, it follows that the auxiliary matrix R is crucial for controlling the distribution shape of the adapta- tion term. Unfortunately, we find there exists a dilemma in choosing an appropriate distribution for R. On one hand, it is desirable for the values in R to be larger. Formally, let d' be the variance of the element in R which is a random vari- able, it can be derived the expectation of the adaptation term

---

IntLoRA: Integral Low-rank Adaptation of Quantized Diffusion Models
Table 1: Quantitative comparison on subject-driven generation tasks. The notion "WxAy" represents the bit-widths of weights "W" and activations "A". The best results are bolded.
methods
nbits
DINO1
CLIP-It
CLIP-TY
LPIPS
LoRA (Hu et al., 2021)
W16A16
0.4828
0.6968
0.2954
0.8076
QLoRA (Dettmers et al., 2024)
W8A8
0.4153
0.6661
0.2824
0.8088
QA-LoRA (Xu al., 2023b)
W8A8
0.4156
0.6664
0.2834
0.8086
IR-QLoRA (Qin et al., 2024)
W8A8
0.4070
0.6630
0.2841
0.8110
SSHLF (Ours)
W8A8
0,4353
0.6842
0.2841
0.8257
IntLRAMuL (Ours)
W8A8
0.4498
0.6882
0.2858
0,8062
QLoRA (Dettmers et al., 2024)
W4A8
0.2136
0.6134
0.2510
0.8201
QA-LoRA (Xu et al., 2023b)
W4A8
0.4127
0.6897
0.2700
0,8281
IR-QLoRA (Qin al., 2024)
W4A8
0.3722
0.6719
0.2707
0.8186
IntLoRASHIFT (Ours)
W4A8
0.4039
0.6716
0.2709
0.8187
IntLRAAULL (Ours)
W4A8
0.4242
0.6913
0.2710
0,8181
4.2. Implementation IntLoRA
5. Experiments
Benefiting from the quantization frienndl weight distribu- tion, we can implement our IntLoRA with two variants according to different quantizers on the adaptation term. The first variant employs the uniform affine quantizer the adaptation term, thus enabling weight merge through integer-type multiplication. The second variant introduces the more hardware- -eruarrffienllyy log2 quantizer to achieve down- stream adaptation by bit-shifting the quantized pre-trained weights. More details are given below.
5.1. Experimental Setup
Datasets. We evaluate on multiple adaptation tasks, includ- ing subject-driven generation (Ruiz et al., 2023), control- lable generation (Zhang al., 2023), and style-customized image generation (Sohn et al., 2023). For the subject-driven generation,
subset which contains 15 text-subject pairs from the Dreambooth (Ruiz et al., 2023) dataset, for fast training and evaluation. For controllable generation, we consider three sub-tasks, i.e, Segmentation map to Image (S2I) on ADE20K dataset (Zhou et al., 2017). Landmark to Face (I.2F) on CelebA-HQ dataset (Karras, 2017), and the Canny edge to Image (C2I) on the COCO dataset (Lin et al., employ the StyleDrop (Sohn et al., 2023) dataset, which includes 18 style images, and we use 6 text prompts for each style to generate images with style similar to the style image and content aligned with the text prompt.
We employ uniform affine quantization on the adaptation term, with the scaling factor and zero-point denoted as s and z, and the quantized results as round then our IntLoRAMuL, can be formalized as:
Integer multiplication form.
2014). For the style-customized generation,
W' = 3 (U Uorund - 2) C (W round 2).
Bit-shifting form. Denote the adaptation term in Eq. (3) as V for clarity, we first compute the bit shift value as follows:
Implementation details. We employ the StableDiffu- sionV1.5 1.5 (Rombach et al., 2022) the pre-trained backbone for subject-driven generation and controllable generation. We further employ larger SDXL (Podell et al., 2023) the pre-trained model in the style-customized generation. We use uniform quantization (Nagel et al., 2021) to quantize the weights per-channel and activations per-tensor. Since previous methods mainly focus on efficient training, with the tuned weights still in FP16. 6, to make fair comparison, apply additional PTQ the merged weights for efficient inference. As for the training of the quantized adaptation term, we use the Straight Through Estimator (STE) to al- low back- propagation. For the proposed variance matching control, we employ the ratio the maxima of the sampled distributions as a fast estimator for the variance. Due the page limit, we provide more details Appendix
shift clip(1- log2 V11,0,2* -).
Then the weight adaptation with Int be rep- resented as:
w = sign( V) ④ 2 shift ( round 2)
= sign V) O (W round 2) > shift]
e sign V) (W round 2) < (N shift)), 2N
(9) where sVgn(V) E {-1,1} and N = 2 - Since the direct right-shifting on W round - may lead to truncation error, we thus use N shift with a scaling factor 2 to equivalently convert the left-shifting for error reduction.
5

---

IntLoRA: Integral Low-rank Adaptation of Quantized Diffusion Models
Text prompt: "a purple [V]"
Text prompt: "a [V] of the sidewalk in crowded street"
Text prompt: [V] the snow"
Text prompt: "a [V] on the beach*
Subject Images
LoRA-FP
QLoRA
QA-LoRA
IR-QLoRA
Ours-MUL
Ours-SHIFF
Figure 4: Qualitative comparison on subject-driven generation tasks. More results are provided in Appendix H.
5.2. Main Results
Table 2: Quantitative comparison of FID, score control- lable image generation.
Subject driven generation. Tab. gives the results of weight-activation quantization on subject-driven generation task. can be seen that the proposed method consistently outperforms other competitors under different bit-widths, For instance, the IntLoRAAmn suppresses the IR-QLoRA by even 0.0428 DINO score the W8A8 setup. Notably, the QLoRA and IR-QLoRA baselines, which use additional PTQ on the merged weights, suffer significant perfor- mance drop under the W4A8 setup. In contrast, even the challenging ogg2-quantization of our IntLORASHIFT works well under W4A8. We also give qualitative visualization in Fig. 4, where one can see that our IntLoRA can facilitate subject-faithful and photo-realistic image generation.
8-bitwidth L2F
4-bitwidth
methods
S2I
C21
S2I
L2F
C2I
LoRA(FP16)
31.39
37.50
16.05
31.39
37.50
16.05
QLoRA
31,09
38.88
15.34
71.75
117.37
62.49
QALoRA
31.32
38.88
15.34
31.51
43.09
16.73
IR IR-QLORA
31.81
36.30
15.70
35.83
39.63
18.30
IntLoRAsHIn
31.38
34.46
15.76
32.85
35.06
17.65
IntLoRAMUL.
31.08
37.52
15.26
30.87
33.62
16.32
5.3. Efficiency Comparison
We compare the training and inference efficiency of IntLoRA against other baselines in Tab. 3. As for training, both IntLoRA and OLo only need to load quantized pre- trained weights, which is more memory efficient compared to the vanilla LoRA fine-tuning. As a result, our IntLoRA and QLoRA have similar training speeds and memory costs. However, for the inference phase, our IntLoRA can naturally obtain the quantized merged weights without additional PTQ, thus streamlining the adaptation pipeline and avoiding potential performance degradation under low bit-width. In short, compared with existing methods which only focus on training efficiency, our IntLoRA presents a both training and inference efficient paradigm.
Controllable image generation. The results of controllable image generation are shown in Tab. 2. One can see that our IntLoRA continues to outperform existing strong baselines, e.g., our IntLoRAMUT, outperforms the IR-QLoRA by 4.96 FID on the 4-bit S2I setting. And it can be seen that QoRA struggles produce meaningful results at low 4 bit-width. We also give qualitative comparison in Fig. 5, and it can be seen that the images generated by the A-tunnd model are well-matched with the control signals.
Style customized generation. The results the style cus- tomized generation task are shown in Fig. 6. that our IntLoRA achieves a favorable balance between style images and text prompts, whereas some existing approaches fail. For instance, in the third row, both the QALoRA and IR- IR-QLoRA methods directly copy the original style image under the text prompt "The letter 'G' in [V] style""
be seen
5.4. Ablation Studies
Ablation on the smoothing factor. As discussed in Sec, 4.1, there is a dilemma in choosing an appropriate dR- For example, setting too large can lead information loss
6

---

IntLoRA: Integral Low-rank Adaptation of Quantized Diffusion Models
Table 3: Comparison of training and inference efficiency with other methods. We fine-tuning the StableDiffusion model on the Dreambooth task. The training speed is tested on one NVIDIA RTX 3090 GPU.
Training Stage
Inference Stage
method
nbits
training speed
model size
PTQ
CLIP-It
CLIP-TY
LoRA (Hu et al., 2021)
W32A32
0.68s/img
7700MB
0.6968
0.2954
QLoRA (Dettmers et al., 2024)
W8A8
0.85s/img
1925MB
V
0.6661
0.2824
InLLORASHPP (Ours)
W8A8
0.84s/img
1925MB
0.6842
0.2841
IntLoRAMULL (Ours)
W8A8
0.87s/img
1925MB
X
0.6882
0.2858
QLoRA (Dettmers al., 2024)
W4A8
0.85s/img
963. 1MB
V
0.6134
0.2510
IntLORASHIPT (Ours)
W4A8
0.84s/img
163
X
0.6716
0.2709
IntLoRAxLL (Ours)
W4A8
0.87s/img
963. 1MB
X
0.6913
0.2710
Origin
Control
QLoRA
QA-LoRA
IR-QLoRA
Ours-MUL
Ours- SHIFT
Segus-o to Image
Landnark Face
e
Carny- Carny-lb- to- Tres
Figure 5: Qualitative comparison on controllable generation tasks. More results are provided in Appendix H.
of the original weights, while a too-small one results in a large quantization error. To this end, we introduce 7 in the proposed VMC as the hyperparameter search for the task-oriented variance. We give the downstream task perfor- mance with varying a in Fig. 7. It can be seen that setting oR slightly smaller than can obtain better performance, indicating that the information loss has greater impact than the quantization eиror. In the implementation, we chose a moderate smoothing factor a = 1.5 for the trade-off.
Table 4: Ablation different distribution shape choices of the auxiliary matrix.
settings
DINOt
CLIP-It
CLIP-T1
LPIPS
Guassian
0.4135
0.6756
0.2492
0.8179
Cauchy
0.1367
0.5617
0.1870
0.8067
StudentT
0.2935
0.6420
0,2487
0.8048
Laplace
0.4492
0.6980
0.2588
0.8110
the light-tailed distribution performs empirically better than its heavy- counterparts.
Distribution selection for auxiliary matrix. this work, the auxiliary matrix plays crucial role in both AQS and VMC. Therefore, the distribution shape of R poten- tially influence the performance. To this end, we investigate different distribution shapes of R. through ablation exper- iments and give the results in Tab. 4. It can be seen that the Laplace distribution performs better than other options on most metrics. This because light-tailed distribution, such Laplace, clusters most samples around zero, which facilitates smaller errors for log2 uog2-quantzzation. Therefore,
6. Further Discussion
Results NLP tasks. In addition to fine- fine-tuning diffusion models for image generation, we further validate the effec- tiveness of the NLP tasks. Specifically, we fine-tune the Llama3-8B model (Dubey et al., 2024) and use the Meta- MathQA dataset (Yu et al., 2023) for training and GSM8K dataset (Cobbe et al., 2021) for testing. The comparison

---

IntLoRA: Integral Low-rank Adaptation of Quantized Diffusion Models
Style Images
LoRA-FP
QLoRA
QA-LoRA
IR-QUA IR-QLORA
Ours
Ge
Cii
Figure 6: Qualitative comparison style-customized generation. The text prompt is "A friendly robot in [V] style", "A panda eating bamboo in [V] style", and "The letter 'G' in [V] style", respectively. "Ours" denotes the InHLORASHIFT. More results are provided in Appendix H.
1.0
Table 5: Comparison the natural language task of mathe- matical answering. More qualitative results in Appendix H.
0.8
Methods
LoRA
QLoRA QA-LoRA OUrSSHIFT OursMUl
Downstren fask
DINOT T
nbits
W16A16 W8A8
W8A8
W8A8
W8A8
CLIP-11 f
accuracy 64.24%
64.06%
63.53%
64.10%
64.23%
CLIP-T T
0.4
LPIPS
0.2
Table 6: Comparison with EfficientDM W4A4 diffusion model quantization. We evaluate the ImageNet 256 256 image generation, and train with ddim step=20 on LDM-4 model with 500 training epochs.
0.6
0.8
1.0
1.2
1.4
1.6
1.8
2.0
The value of
Figure 7: The normalized performance under different a.
methods
ISt
FID
sFID
precision
EfficientDM
178.20
13.42
26.67
0.70
results shown in Tab. 5. It be seen that Int- LoRA maintains stable performance when transferring to natural language. For example, IntLoRAMur outperforms QLoRA by 0.17% QA accuracy under 8-bit quantization, demonstrating the generalization of our IntLoRA.
OurSSHIFT
116.50
20.20
26.79
0.63
OursMul
199.20
10.43
24.02
0.79
7. Conclusion
Difference from EfficientDM. EfficientDM (He et al., 2023) employs QAT-like LoRA fine-tuning on the FP16 diffusion weights for network quantization. Despite it is inference-efffcient as it can directly produce quantized merged weights, we would like to point out that it is not training-efficient Specifically, EfficientDM requires load FP16 pre-trained weights at the training stage, which is unacceptable for fine-tuning large-size models on consumer- level GPUs. By contrast, our IntLoRA is both training and inference efficient. Moreover, we also explore our IntLoRA on the diffusion quantization task. The results shown in Tab. 6. It can be seen that our Intl oRAMMT achieves even better performance than the EfficientDM. It should be noted that our IntLoRA only needs to load the quantized weights during calibration instead the Roating-pott weights in EfficientDM, thus reducing the training memory cost.
We propose IntLoRA, which employs integer-type low- rank parameters, remove the additional PTQ on the merged weights. Specifically, introduce the quantization- adaptation separation to allow the coexistence of zero- initialized gradient and quantization-friett friendll distribution. We further develop the multiplicative low-rank adaptation to achieve a decoupled quantizer of pre-trained and adaptation weights, accompanied by variance matching control to adjust the variance for accurate adaptation control. Benefit- ing from these elegant designs, we provide variants of IntLoRA, which either use int-multiplication or bit-shifting to adapt the quantized pre-trained models. Through trans- ferring the adaptation weights to the integer arithmetic, our IntLoRA demonstrates its effectiveness across different pre- trained models and various downstream tasks, while exhibit- ing impressive both training and inference efficiency.
8

---

IntLoRA: Integral Low-rank Adaptation of Quantized Diffusion Models
Acknowledges
Hu, E. J., Shen, Y., Wallis, P. Allen-Zhu, Z., Li, Y., Wang, S., Wang, I.,, and Chen, W. LoRA: Low-rank adaptation of large language models. arXiv preprint WiXi::210609685,, 2021.
This work supported part by the National Natural Science Foundation of China, under Grant (62302309, 62171248), Shenzhen Science and Technology Program
Huang, W., Qin, H., Liu, Y., Li, Y., Liu, X.. Benini, L.,
Magno, M., and Qi, X. SliM-LLM: Salience-driven pixeei-recisionn quantization for large language models. arXiv preprint arXiv:2405. 14917, 2024a.
Impact Statement
This work aims to achieve efficient fine-tuning of quantized diffusion models, reducing both training and inference costs without sacrificing performance. It has no ethical concerns and can lower the resource barrier for model customization, thus enabling broader access to generative AI.
Huang. Y., Gong, R., Liu, J., Chen, T., and Liu, X. TFMQ- DM: Temporal feature maintenance quantization for dif- fusion models. CVPR, pp. 7362-7371, 2024b.
Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A., Adam, H., and Kalenichenko, D. Quantization and training of neural networks for efficient integer- arithmetic-only inference. In CVPR, pp. 2704-2713, 2018.
References
BlackForestLabs. FLUX.
black-forest-labbr fwwp// 2024.
Chavan, A., Liu, Z., Gupta, D., Xing, E., and Shen, Z. One-for-All: Generalized LoRA for parameter-efficien fine-tuning, arXiv preprint arXiv:2306.0796; 2023.
Jia, M., Tang, L., Chen, B.-C., Cardie, C., Belongie, S., Hariharan, B., and Lim, S.-N. Visual prompt tuning. In ECCV, pp. 709-727. Springer, 2022.
Chen, S., Ge, C., Tong, Z., Wang, J., Song, Y., Wang. J., and Luo. P. AdaptFormer: Adapting vision transformers for scalable visual recognition. In NewrlPS, volume 35, pp. 16664-16678, 2022.
Jie, S. and Deng. Z.-H. FacT: factor-tuning for lightweight adaptation vision transformer. In AAAI, pp. 1060- 1068, 2023.
Karras, T. Progressive growing of GANs for im- proved quality, stability, and variation. arXiv preprint 17X1v:1710.00196 2017.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint 10.11168, 2021.
Li, X., Liu, Y., Lian, L., Yang, H., Dong, Z., Kang, D., Zhang, S., and Keutzer, K. Q-Diffusion: Quantizing diffusion models. In ICCV, pp. 17535-17545 2023.
Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. QLoRA: Efficient finetuning of quantized llms. In NeurlPS, volume 36, 2024.
Li, X. L. and Liang, P. Prefix-Tuning Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190, 2021.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang. A., Fan, A., et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21788 2024.
Li, Y., Gong, R., Tan, X., Yang, Y., Hu, P., Zhang, Q. Yu, F., Wang, W., and Gu, S. BRECQ: Pushing the limit of post-training quantization by block reconstruction. arXiv preprint arXiv:2 2021.
He, J., Zhou, C., Ma, X., Berg- Keng-Kirkpatrick, T., and Neubig, G. Towards unified view of parameter-efficient transfer learning, arXiv preprint 2rXiv:2110.043666 2021.
Li, Y., Xu, S., Zhang, B., Cao, X., Gao, P., and Guo, G. Q-ViT: Accurate and fully quantized low-bit vision trans- former. In NeurIPS, volume 35, pp. 34451-34463, 2022.
He, Y., Liu, J.. Wu, W. Zhou, H., and Zhuang, B. Effi- cientDM: Efficient quantization-awar fine-tuning of low- bit diffusion models. arXiv preprint arXiv:2310.03270, 2023.
Li, Y., Xu, S., Cao, X., Sun, X., and Zhang, B. Q-DM: An efficient low-bit quantized diffusion model. In NeurIPS, volume 36, 2024.
Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B.,
De Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S. Parameter-effician transfer learning for NLP. In JCML. pp. 2790-2799 PMIR, 2019.
Lian, D., Zhou, D., Feng, J., and Wang, X. Scaling & shifting your features: A new baseline for efficient model tuning. NeurIPS, volume 35, pp. 109-123, 2022.

---

IntLoRA: Integral Low-rank Adaptation of Quantized Diffusion Models
Lin, T.-Y., Maire, M., Belongie, S., Hays, Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. Microsoft COCO: Common objects in context. In ECCV, pp. 740- 755. Springer, 2014.
Wang, C., Wang, Z., Xu, X., Tang, Y., Zhou, J., and Lu, J. Towards accurate data-free quantization for diffusion models. arXiv preprint
2(5). 2023.
Wei, X., Gong, R., Li, Y., Liu, X., and Yu, QDrop: Ran- domly dropping quantization for extremely low-bit post- training quantization. arXiv preprint arXiv:2203.05744) 2022.
Liu, J., Niu, L.,, Yuan, Z., Yang, D., Wang, X., and Liu, W. PD-Quant: traittrainng quantization based on prediction difference metric. In CVPR, pp.
2023.
Nagel, M., Fournarakis, M., Amjad, R. A., Bondarenko, Y., Van Baalen, M., and Blankevoort, T. A white pa- per on neural network quantization. arXiv preprint arXiv:2106.08295, 2021.
Xu, S., Li, Y., Lin, M., Gao, P., Guo, G., Lüi, J., and Zhang, Q-DETR: An efficient low-bit quantized detection transformer. In CVPR, pp. 3842-3851, 2023a.
Xu, Y., Xie, L.,, Gu, X., Chen, X., Chang, H., Zhang, H., Chen, Z., Zhang, X., and Tian, Q. QA-LoRA: Quantization-aware low-rank adaptation of large lan- guage models, arXiv preprint arXiv:2309, 14717, 2023b.
Nahshan, Y., Chmiel, B., Baskin, C., Zheltonozhskii, E., Banner, R., Bronstein, M., and Mendelson, A. Loss aware post-training quantization. Machine Learning, 110 (11):3245-332 -2622, 2021.
Yu, L., Jiang, W., Shi, H., Yu, J., Liu, Z., Zhang, Y., Kwok, J. T., Li, Z., Weller, A., and Liu, MetaMath: boot- strap your own mathematical questions for large language models. arXiv preprint arXiv:2309. 12284, 2023.
Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., and Rombach, R. SDXL: Im- proving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023.
Zhang, L.,, Rao, A., and Agrawala, M. Adding conditional control to text-to-image diffusion models. ICCV, pp. 3836 1336-3847, 2023.
Qin, H., Ma, X., Zheng. X., Li, X., Zhang, Y., Liu, S., Luo, J., Liu, X., and Magno, M. Accurate LoRA-finetuning quantization of LLMs via information retention. arXiv preprint ar arXiv:2402.05445. 2024.
Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., and Torralba, A. Scene parsing through ADE20K dataset. In CVPR, pp. 633-641, 2017.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, High-resolution image synthesis with latent diffusion models. CVPR, pp. 10684-10695, 2022.
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., and Aberman, K. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR, pp. 22500-22510, 2023.
Saharia, C., Chan, W., Saxena, S., Li, L.., Whang, J., Denton, E. L., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., al. Photorealistic text-to-image dif- fusion models with deep language understanding. In NeurlPS, volume 35, pp. (4479-36492 2022.
Shang, Y., Yuan, Z., Xie, B., Wu, B., and Yan, Y. Post- training quantization on diffusion models. In CVPR, pp. 1972-1981, 2023.
Sohn, K., Ruiz, N., Lee, K., Chin, D. C., Blok, I., Chang, H., Barber, J., Jiang, L., Entis, G., Li, Y., et al. Styledrop: Text-to-image generation in any style. arXiv preprint (tt::233660098833 2023.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. CoRR, abs/1706.03762, 2017. URL
httpp://rxiv..o http:p//aaaavooro////////.6...7
10

---

IntLoRA: Integral Low-rank Adaptation of Quantized Diffusion Models
A. Summery of IntLoRA Algorithm
Before tuning, we pre-process the pre-trained weights Algo. 1, followed by the forward process of IntLoRAMUL, and IntLORASHIF in Algo. 2 and Algo. 3, respectively.
Algorithm 1 The weight pre- process of the linear layer in IntLoRA
Input:
Pre-trained wight W IR RoCoxt auxiliary matrix R Cout CCn, smooth factor a E R
Output: Quantitzed pre-trained weights W rounds scaling factor Sround, point Zround
sigma_R +- variance estimation of R
sigma W <- variance estimation of W
== (sigma_W W sigma
Ratar = r R
W process = W -Rtar
W round Sround: Zround < uniform quantizer W process)
Algorithm 2 The forward process of the linear layer in IntLORAMUL.
Input: Pre-processed quantized weights W round, scaling factor Sround, point 2round, auxiliary matrix Rytar
Rouu XCin, LoRA parameters A E TR. Xo
Xd B E R.Cue input tensor x E IR Ci. xL
Output: Output tensor y E RCoue
W adant = Sround I W Fround @ (Ratar + AB)
WINT INT
adapt: Sudapt, Padapt f- uniform.quantizee quantizer( W adaat)
X INE f- aua_quantizer(x)
merge adapt Indapt) 3 (W round 2round)
(WT TTT
3rond
W mergex INT
Algorithm 3 The forward process of the linear layer in IntLORASHIFT
Input: Pre-processed quantized weights W Woundd scaling factor Stound, zero point Zround, auxiliary matrix Ratar RCouxXC,, LoRA parameters A Coue x B E Xd input tensor R.nx desired bit-width b, pre-defined
max bit-width number N -32
Output: Output tensor y E XC
W adapt = Sround I
Retar + AB)
W round Around
shift clip([ |- logg W adapt 0,20
W merge 2R sign( W adapt) @ I(W round- 2) < (N shift)]
INN 8x f- act quaanttzizz)r(
W mergee"" TNT
B. Distribution Visualization of OR.
In Sec. 4.2, we have theoretically pointed out that there a choice dilemma for oR- Here we elaborate on its effect through distribution visualization. Specifically, we remove the VMC and use scaling scalar to generate too-large or too-small auxiliary variance, followed by the log2 quantization on the adaptation term. The results are shown in Fig. On the one hand, setting oR too large can lead
low correlation p pWW, W -R), which makes hard to reconstruct W from W W-B R using estimator W ~ Q(W R.) + R. On the other hand, too small oR prevents the expectation of adaptation term converging to zero, causing few log bins to be used. In experiments, we find that the training of both settings fails to converge. By contrast, the proposed VMC can precisely control to allow most values of the adaptation term to be neighbred,b,,,, facilitating more challenging log2 quantization. Moreover, it should be noted that the too-small OR can also be regarded as an approximation of direct quantization on the zero-inin initialized AB, and thus the experimental results also justify the AQS for rero-initialized AB.

---

IntLoRA: Integral Low-rank Adaptation of Quantized Diffusion Models
Origin pre-trained weight W
W i Q(W
R) R using VMC ox
Adaptation term using VMC On
Adaptation turm using VMC Cn.
small quanthation
Origin pre- me-trained weight W
@ R
using too large
A Adeptation term using too small
Bit bin usage too small or
large
MAHAAA Bandy
Tow log2 bins are Laad
00
C200
X01
Figure 8: The distribution visualization using Kernel Density Estimate (KDE) on different weight tensors. Left: the KDE plot of pre-trained weights and estimated weights under different dR. Right: the KDE plot of the adaptation term and the log2 bins usage with different dR-
DINO score
CLIP-1
CLIP-T
LPIPS
0.700
0.20
0.590
0.290
0.51
0.700
0.990
0.779
0772
0.91
0.602
0.340
0.53
0.032
0.625
0.79
10
Figure 9: Ablation experiments of different LoRA ranks.
C. More Implementation Details
For the subject-driven generation, we use the AdamW optimizer with a weight decay of le-2 and fine-tune the query, key, value, and output projection layer. The learning rate is set to 6e-5. The batch size is set to 1, and the number of training steps is 400. The rank of the LoRA is set to We adopt the prior preservation strategy as Dreambooth (Ruiz et al., 2023) to generate 200 class images. For the controllable generation, we fine-tune the model for 11 epochs for Canny-to-Ime Image tasks and 20 epochs for Landmark-to-Faan and Segmentationnto--naa optimizer. The LoRA rank is set to 4. The batch size is set to 8 and the image resolution is 512 x 512 for all three tasks. For the style-customized generation, we fine-tune the pre-trained model using the Adam W optimizer with a learning rate of 5e-5. Since it involves a larger SDXL, we chose a relatively large LoRA rank of 64 for all compared methods, since there is only one style image as well as the larger pre-trained parameters. We fine-tune for 500 steps with batch size 1. Similar to StyleDrop (Sohn et al., 2023), we only use one image as the style image and find it works well. The style images and text prompts for evaluation are given in Appendix H. The variance ratio in the variance matching control is surrogated as the value range ratio, i.e., r = max|] max WW], ]I.] main(W]]]
tasks. The learning rate is set le-5 using the Adam W
min[ max [R]]. min( We append the trainable low-rank parameters on the Query, Key, Value, and Out projection, in the attention layers (Vaswani et al., 2017) and keep all other layers frozen and quantized. The rank of LoRA is set to 4 for subject-driven generation and controllable generation, and 64 for style-customized generation.
D. A Additional Ablation Experiments
Ablation on the LoRA rank. The low-rank d in LoRA is trade-off between performance and efficiency. A larger rank improves the adaptation ability by training more parameters but comes with larger training and storage costs, and vice versa. Here, we give the impact of different rank setups on performance in Fig. 9, One can see that the performance generally improves as we increase the rank, but the rate of growth varies. For instance, the increased speed from rank=4 to rank=8 increases inferior to the one from rank=2 to rank=4. Moreover, increasing the rank to 16 can generally obtain better results than its lower counterpart, practice, considering the trade-off between performance and efficiency, we select a moderate rate rank=4.
12

---

IntLoRA: Integral Low-rank Adaptation of Quantized Diffusion Models
Caussian Distibution
Stadeff Distrbottoon
Luglace Distribuion
Cauchy Distibution
Figure 11: The shape of different distributions for initialing the auxiliary matrix.
The distribution of pre- trained weights W
The distribution of auxiliary matrix R
The distribution of LoRA weights AB
Figure 12: The distribution visualization of the original weights W, the auxiliary matrix R, and the learned low-rank
weights AB.
The effects of variance matching control for IntLoRAMuL- In this work, we propose the variance matching control to adjust the variance of R. so that allows the log2-quaniizttin of the adaptation term obtain the IntLoRASHIFT- In other words, the VMC is primarily introduced for IntLoRAsHIFT Despite we also apply the VMC to IntLoR given the IntLoRAMur does not require such strict constraints on the distribution shape of the adaptation term, it is interesting to investigate the influence of variance matching control on the performance of IntLoRAMuL- To this end, we adjust the smoothing factor a to adjust the strength of the VMC, e.g., setting a zero can lead the removal of the VMC. The results of the IntLoR AMMIL. under different VMC scales are shown in Fig. 10. As one can see, despite the VMC being initially proposed for the fog2-quantization. the well-structured distribution also facilitates uniform quantization. For example, when set the approaching zero, i.e., the VMC is close to being removed, and the performance of IntLoRA appears similar pattern as the when the a > 1.5. In short, the VMC can not only allow the log2-quantization to work but also improve the performance of the uniform quantization.
BINO CLIFIT
CLI-TT
LXXS
07
06
20
30
Figure 10: The effects of VMC for IntLoRAMUL-
which suffers a significant performance drop. Moreover, the performance gains gradually converge
Distribution shape for auxiliary matrix. Sec. 5.4, we provided different symmetric distributions including Gaussian, StudentT, Laplace, and Cauchy. Fig. 11 gives the results of sampling from different distributions. The Laplace distribution possesses light tails, and the shape of the distribution is convex, i.e., (z) > 0, x + 0. This unique property makes it easy to control the value of the adaptation term to produce distributions that are friendly to log2 quantization, i.e., most samples are clustered around the zero to use as many bins as possible. This analysis also verified by the experiments in Tab. 4, which shows that the Laplace distribution achieves the best performance.
E. Impacts from the Auxiliary Matrix
In Eq. (2) of the proposed AQS, introduce additional auxiliary matrix R to the original pre-trained weight Wa to achieve adaptation-qqantizzat
separation. However, this extra R. potentially introduces outliers and thus causes quantization error for W. Here, we point out that since the proposed VMC can control the range of R through the variance scaling factor r = ow / the introduction of R. in the AQS is ensured not result additional outliers. For validation, we also give the distribution visualization of the original W and the VMC re-scaled in Fig. 12. It can be seen that the range of R. effectively controlled within the range of W. thus effectively avoiding the detrimental effect of additional outliers.
13

---

IntLoRA: Integral Low-rank Adaptation of Quantized Diffusion Models
F. Justification for the Value Orders
key assumption in derivation VMC that the learned values low-rank parameters AB are orders of magnitude smaller than the auxiliary matrix R. Based on this assumption we ignore AB as an approxima- tion. Here, we give the specific evidence for this approximation. Specif- ically, we visualize the weights of the trained AB and the distribution of R. as shown in Fig. 12. It can be seen the range of AB is constrained to [-0.0004, 0.0004), while the range of R is [-0.08, 008]. Therefore, the ex- perimental visualization above confirms the soundness of our approximation. Since the AB LoRA zero-initialized, tends to be distributed around zero with learned small values aiming disturb the pre-training weights too much.
The darrd detilbutuo vastnce
175
300
15
50
Figure 13: The value distribution of the
channel-wise variance ratio r
G. Limitation and Future Works
Although the proposed IntLoRA can effectively improve the efficiency of diffusion model fine-tuning by allowing the adaptation parameters also on the integer arithmetic, the proposed framework can be further improved in the following aspects. First, although the trainable low-rank weights are quantized with STE, these quantized weights are still in FP type during tuning enable accurate gradient updates. Therefore, is promising specifically design integer-type propagation. Despite this seems challenging, it can further reduce the training cost and accelerate the adaptation process. Second, although we introduce feasible way that uses hyperparameter search of the smoothing factor to find a compatible as well as the appropriate distribution shape of R. it can be more elegant if we can use advanced mathematical analysis techniques, such as functional analysis, to find the statistical properties a suitable R. should satisfy. Third, this work mainly focuses on the efficient acceleration of LoRA due to its prevalence among the PEFT techniques. Applications to other PEFT methods for hurdware-efficiere adaptation could be interesting future work.
H. Additional Visualization Results
Fig. 14 gives more samples on the subject-driven generation tasks.
In Fig. 15, we give more samples of the gegmentation-to-imaee tasks.
In Fig, 16, we give more samples of the face landmark-to-face image tasks.
In Fig. 17, we give more samples of the canny edge-to-image tasks.
Fig. 18 provides more samples of the results of the style-accustomed generation.
In Fig. 19, we give the style images and the text prompts used for evaluation on the style customized generation tasks.
In Appendix H.I, we give some case studies the mathematical question-answerrig task using the fine-tuned Llama3-8B model.
14

---

IntLoRA: Integral Low-rank Adaptation of Quantized Diffusion Models
"a [V] in chef outfit"
"a [V] with mountain in the background"
Subject Images
QLoRA
DA-LoRA
IR-QLoRA
Ours-MUL MUL
Ours-SHIFE
"a [V] with a city in the background"
"a [V] in purple wizard outfit"
Subject Images
QLoRA
QA-LoRA
IR-QLoRA
Ours-MUL.
Ours- SHIFT
"a [V] on the beach"
"a [V] wearing rainbow scarf"
Subject Images
QLoRA
QA-LoRA
IR-QLoRA
Ours-MUL
Ours-SHIFT SHIFT
Ours-
Figure 14: More qualitative comparison results subject-driven generation.
15

---

IntLoRA: Integral Low-rank Adaptation of Quantized Diffusion Models
Origin
Control
QLoRA
QA-LORA
IR-QLoRA
Ours
Figure 15: More qualitative comparison results on segmentation image task. The 'Ours' denotes the IntLORASHIFT. Zoom in for better effects.
16

---

IntLoRA: Integral Low-rank Adaptation of Quantized Diffusion Models
Origin
Control
QLoRA
QA-LORA
IR-QLORA
Ours
2
Figure 16: More qualitative comparison results on landmark face task. The 'Ours' denotes the IntLoRASHIFT. Zoom in for better effects.
17

---

IntLoRA: Integral Low-rank Adaptation of Quantized Diffusion Models
Origin
Control
QLoRA
QA-LoRA
IR- RR-LLORA
Our
STOP
STPP
STOP
STOP
STOP
Figure 17: More qualitative comparison results canny image task. The 'Ours' denotes the Zoom in for better effects.
18

---

IntLoRA: Integral Low-rank Adaptation of Quantized Diffusion Models
"A man riding a snow board in [V] style"
"The letter 'G' in [V] style"
C
G
G
"A friendly robot in [V] style"
D&G
"The Golden Gate bridge in [V] style"
"A man riding snow board in [V] style"
HATA
"A friendly robot in [V] style"
Style Image
LoRA-FP
QLoRA
QALoRA
IR-QLoRA
Ours
Figure 18: More qualitative comparison results style-accustomed generation. The 'Ours' denotes the IntLoR
Zoom in for better effects.
19

---

IntLoRA: Integral Low-rank Adaptation of Quantized Diffusion Models
Figure 19: The style images used for training on the style-customized generation task. The text prompts are "A friendly robot in [V] style", "A fluffy baby sloth with a knitted hat trying to figure out a laptop in [V] style", "A panda eating bamboo [V] style", "The Golden Gate bridge in [V] style", "A man riding snowboard in [V] style", "The letter 'G' in [V] style".
20

---

IntLoRA: Integral Low-rank Adaptation of Quantized Diffusion Models
H.1. Cases of mathematical question answering.
Case Study
Janet's ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market? (Ground-truth answer is 18)
Question:
LoRA-FP Answer: Janet's ducks lay 16 eggs per day. She eats three for breakfast every morning, so she has 16 3 13 eggs left. She bakes muffins for her friends every day with four eggs, so she has 13 = 9 eggs left. She sells the remainder at the farmers' market daily for $2 per fresh duck egg, so she makes 9 $2 = $18. Therefore, Janet makes $18 every day at the farmers' market. The answer 18.
Janet's ducks lay 16 eggs per day. She eats three for breakfast every morning, so she has 16 13 eggs left. She bakes muffins for her friends every day with four eggs, so she has 13 eggs left. She sells the remainder at the farmers' market daily for $2 per fresh duck egg, she makes 9 $2 $18. Therefore, Janet makes $18 every day at the farmers' market. The answer is: 18
QLoRA Answer:
QA-LoRA Answer: Janet's ducks lay 16 eggs per day. She eats three for breakfast every morning, so she has 1 13 eggs left. She bakes muffins for her friends every day with four, so she has 13 = eggs left. She sells the remainder at the farmers' market daily for $2 per fresh duck egg, she makes $2 $18. Therefore, Janet makes $18 every day the farmers' market. The answer is: 18
QA-LoRA Answer:
Janet lays 16 eggs per day. She eats three for breakfast, so she has - 13 eggs left. She bakes muffins with four eggs, so she has 13 = 9 eggs left. She sells the remainder at the farmers' market for S2 per egg, so she makes $2 = $18. Therefore, Janet makes $18 every day at the farmers' market. The answer is: 18
Janet lays 16 eggs per day. She eats three for breakfast, so she has 16 3 13 eggs left. She bakes muffins with four eggs, so she has 13 4 eggs left. She sells the remainder the farmers' market for $2 per fresh duck egg, she makes 9 $2 $18. Therefore, Janet makes $18 every day at the farmers' market. The answer is: 18.
Our Answer:
21

---

IntLoRA: Integral Low-rank Adaptation of Quantized Diffusion Models
Case Study 2
Question: A robe takes 2 bolts of blue fiber and half that much white fiber. How many bolts in total does it take? (Ground-truth answer is 3)
LoRA-FP Answer:
We know that the robe takes 2 bolts of blue fiber. To find the amount of white fiber, we can use the ratio of blue fiber to white fiber, which is 2:1. So, the amount of white fiber is half of the blue fiber, which 2/2 bolt. Therefore, the total number of bolts needed is 2 + == bolts. The answer is
QLoRA Answer: The robe takes bolts of blue fiber. To find the amount of white fiber, Since the ratio is 1:2, for x, we get 1. Therefore, the robe takes 2 bolts of blue fiber and bolt of white fiber. In total, the robe takes 2 + 1= 3 bolts of fiber. The answer is: 3
the ratio of 1:2. set up the equation 1/2 * 2 where represents the amount of white fiber. Solving
QA-LoRA Answer: robe takes 2 bolts of blue fiber. The robe takes half that much white fiber, which is 2/2 bolt white fiber. So, in total, the robe takes 2 bolts of blue fiber bolt of white fiber bolts of fiber. Therefore, the robe takes 3 bolts of fiber in total. The answer is: 3
IR-QLoRA Answer: The robe takes 2 bolts of blue fiber. The robe takes half that much white fiber, which is 2/2 bolt of white fiber. So, the robe takes 2 bolts of blue fiber + 1 bolt of white fiber = 3 bolts in total. The answer is:
Our Answer: If robe takes bolts of blue fiber, then it takes 2/2 = bolt of white fiber. So, the total number of bolts needed bolts of blue fiber bolt of white fiber bolts total. The answer
22


Reviews:
[{"review_id": "wjpYM5uBuQ", "reviewer": "Reviewer_1Zzg", "summary": "This paper proposes IntLoRA that allows for seamless weight merging after efficient low-bit parameter-efficient fine-tuning (PEFT). The paper is motivated by the observation that existing low-bit PEFT (e.g., QLoRA) requires an additional round of PTQ due to a mismatch in precision between pre-trained and (low-rank) adapter weights. Towards this goal, authors proposed various techniques like task-agnostic auxiliary weights, variance matching control, and multiplicative/bit-shifting LoRA. Finally, authors demonstrate compute/memory efficiency of IntLoRA as well as the superior quality of images generated from diffusion models trained with IntLora against several popular baselines.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> ### The significance of weight merging\n\nGood comments!  In fact, the efficiency of INT-type matrix multiplication is extremely efficient in the highly optimized GEMM on modern GPUs, and we demonstrate below **the INT8 matmul is even faster than that of FP32 low-rank matrix multiplication**. \n\nGiven a weight matrix $\\mathbf{W} \\in \\mathbb{R}^{M\\times N}$, activation $\\mathbf{X} \\in \\mathbb{R}^{N \\times L}$ and the low-rank decomposition $\\mathbf{W = AB}$ ( where $\\mathbf{A} \\in \\mathbb{R}^{N \\times r},\\mathbf{B} \\in \\mathbb{R}^{r \\times M}$), we adopt the shapes from SD-1.5, i.e., $M = N = 320, L = 4096, r=4$. We use INT8 GEMM CUDA kernel to calculate $\\mathcal{Q}(\\mathbf{W})\\mathcal{Q}(\\mathbf{X})$ ($\\mathcal{Q}$ denotes the quantization operator), and use the common FP32 `torch.matmul` in Pytorch to compute the $\\mathbf{ABX}$. The experimental results on NVIDIA A5000 GPU are as follows:\n\n$$\n\\\\begin{array}{l|ccc}\n\\\\hline\n\\text{setups}      & \\text{runtime}  & \\text{GPU} \\text{costs} & \\text{FLOPs}   \\\\\\\\ \\\\hline\n\\text{INT8 GEMM}   & \\text{48.71 ms }& \\text{1.34MB}    & \\text{0.8388G} \\\\\\\\\n\\text{FP32 matmul} & \\text{52.16 ms }& \\text{5.01MB}    & \\text{0.0209G} \\\\\\\\ \\\\hline\n\\\\end{array}\n$$\n\nAlthough full-rank matrix multiplication has 40x higher FLOPs than its low-rank counterpart, the INT matrix multiplication shows even faster speeds and less GPU footprints due to quantized INT8 type on integer-optimized GEMM kernels.\n\nFrom the above analysis, **the low-rank FP32 matmul is not negligible than the INT8 pre-trained weights**. Therefore, it makes sense to apply the proposed IntLoRA to directly obtain the quantized merged weights.", "claims_and_evidence": "Technical claims/evidence regarding IntLoRA look impressive to me. However, I am not entirely sure if PTQ is necessary in e.g., QLoRA. Given that adapter weights are generally much smaller than pre-trained weights, I believe the additional compute cost from the adapter forward-pass at inference time can be completely hidden by overlapping its computation with the forward-pass from pre-trained weights. In theory, this of course incurs more FLOPs and GPU memory usage from loading adapter weights, but this can be quite marginal compared to memory/compute costs from pre-trained weights in practice. If authors can demonstrate the significance/necessity of weight-merging in general, it would make the paper stronger.", "methods_and_evaluation_criteria": "The proposed techniques--auxiliary weights, multiplicative/bit-shifting LoRAs, and VMC-- are all well-motivated. Furthermore, authors evaluated the effectiveness of IntLoRA in terms of compute/memory efficiency and image generation quality against reasonably chosen baselines.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes. They look good to me.", "relation_to_broader_scientific_literature": "N/A/", "essential_references_not_discussed": "I believe authors have cited major relevant work. However, I am not very familiar with quantization research, so I would count on other reviewers' opinions.", "other_strengths_and_weaknesses": "See \"Claims And Evidence.\"", "other_comments_or_suggestions": "See \"Claims And Evidence.\"", "questions_for_authors": "See \"Claims And Evidence.\"", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OIg1dVJdOL", "reviewer": "Reviewer_c6DN", "summary": "This work proposes a LoRA-based method to fine-tune the quantized weights of diffusion models that consists of Adaptation-Quantization Separation (AQS) for addressing the issue of zero-initialized weights in LoRA tuning for the quantized pre-trained model and Variance Matching Control (VMC) to determine an appropriate R to balance between quantization difficulty and information retention. After using AQS and VMC to preprocess the weights before tuning, the fine-tuning is performed for the adaptation term and then the result will be merged to the original pre-trained term by either Int-multiply or Bit-shifting to yield the quantized merged weights. The proposed method was tested on a number of diffusion-based tasks along with a couple of LLM tasks.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> ### Why Introduce Many LLM Related Works?\n\nThanks for your comment! Although there is a lot of work on Diffusion quantization, **very little work** explores the adaptation of quantized diffusion, which is also recognized by Reviewer wfKN. Therefore, we introduce related methods in quantized LLM adaptation to **ensure a comprehensive baseline comparison**. \n\n> ### Clarification of the Claims.\n\nConsistent with the title, this work mainly focuses on Diffusion models. The experiments on NLP tasks are **preliminary explorations** to answer potential concerns about the method generality. As suggested, we will make the relationship between Diffusion and LLM clearer to avoid any misleading or over-claims.\n\n> ### Efficiency of Different Stages.\n\nIn Tab.3, we give the training speed and memory usage during **fine-tuning stage**. Our IntLoRA is similar to that of QLoRA, but we can directly obtain the quantized merged weights.\n\nFor the pre-processing and post-processing stage, we give the time cost as follows. \n\n\\\\begin{array}{l|ccc}\n\\\\hline\n\\text{stage}  & \\text{pre-processing}  & \\text{fine-tuning} & \\text{post-processing}  \\\\\\\\ \\\\hline\n\\text{time}   & 28.8s & 128.2s & 0.5s  \\\\\\\\\n\\\\hline\n\\\\end{array}\n\nThe **pre-processing** accounts for 18% total time. However, this costs **can be shared** across different tasks. \n\nFor **weight merging**, the latency can be neglected.\n\n> ### Justification of Quantized Adaptation & Why Dreambooth needs Quantization & Comparison with Text-Inversion.\n\nOur quantized adaptation enjoys benefits in **both tuning and inference**.  Specifically,\n\n1) The low-bit adaptation **reduce GPU usage** during tuning. Although Dreambooth involves small samples, it still struggles when fine-tuning large models. For example, **loading FP16 FLUX consumes over 40GB, let alone fine-tuning with Text-Inversion or LoRA**. In contrast, our 4-bit quantized IntLoRA reduces the loading memory to <15GB, facilitating tuning on consumer-level GPUs. \n\n2) The quantized downstream model **boost inference efficiency**.  As shown in the `Response to Reviewer 1Zzg`, the INT GEMM is even more efficient than FP low-rank matmul, allowing for fast inference. As a comparison, the Text-Inversion has the same latency of the pre-trained large model. \n\nAs shown in Tab.1, our IntLoRA can achieve acceleration with negligible performance loss. We will include more discussion and related works in the revision! \n\n> ### Comparison with LQ-LoRA and GPTQ-LoRA.\n\nBoth **LQ-LoRA** and our IntLoRA share the idea of avoiding zero-initial low-rank weights. But they have several essential differences. 1) The **formulation is different**. LQ-LoRA uses the decomposition $\\mathbf{W'} = \\mathcal{Q}(\\mathbf{W}) + \\mathbf{AB}$ to achieve non-zero LoRA weights, our IntLoRA introduces the auxiliary matrix $\\mathbf{R}$ and use $\\mathbf{W'} = \\mathcal{Q}(\\mathbf{W + R) -R +AB}$ for this goal. 2) LQ-LoRA requires **multi-iterations** to search for the optimal approximation, while our IntLoRA does not. 3) We also **compare LQ-LoRA on Dreambooth**  as follows. One can see that our IntLoRA achieves better performance.\n\n\\\\begin{array}{l|ccccc}\n\\\\hline\n\\text{methods}  & \\text{nbits} & \\text{DINO}   & \\text{CLIP-I} & \\text{CLIP-T} & \\text{LPIPS}  \\\\\\\\ \\\\hline\n\\text{LQ-LoRA}  & W8A8  & 0.4056 & 0.6624 & 0.2824 & 0.8126 \\\\\\\\\n\\text{Ours-MUL} & W8A8  & 0.4498 & 0.6882 & 0.2858 & 0.8062 \\\\\\\\\n\\text{LQ-LoRA}  & W4A8  & 0.4022 & 0.6797 & 0.2680 & 0.8198 \\\\\\\\\n\\text{Ours-MUL} & W4A8  & 0.4242 & 0.6913 & 0.2710 & 0.8181 \\\\\\\\ \\\\hline\n\\\\end{array}\n\nAs for **GPTQ-LoRA**, it appends LoRA weights on the GPTQ quantized models, which needs additional PTQ during inference deployment. Although GPTQ-LoRA is not accepted and not open-source, we are happy to add related discussion in the revision.\n\n> ### The Cost of R & Ablation on sigma_R.\n\n- As stated in Line215, the $\\mathbf{R}$ can be **generated on-the-fly** under the same random seed, and **can be deleted once used**. The performance in all tables has included the effects from $\\mathbf{R}$. \n\n- Since $\\mathbf{R}$ is first normalized and then rescaled by $\\alpha$, the ablation of $\\alpha$ in Fig.7 can reflect the role of $\\sigma_R$. One can see **the performance is robust** when $\\alpha >1.4$.  We adopt a fixed $\\alpha = 1.5$, i.e., the same $\\sigma_R$,  in all experiments and the results are satisfactory.\n\n> ### Clarification on Fig.1 & Dtype of Adaptation-Term.\n\nSimilar to common QAT practice, the weights and activations (and Adaptation term) during training are \"simulatively quantizated\" with FP16 dtype, for accurate gradient backpropagation. However, **the inference is strictly in INT dtype (or 2^X format for SHIFT).** Fig.1 is the **post-tuning weight merging stage**, so it is correct since both W and A are INT. We will clarify this in revision.\n\n> ### Solution to Divide-by-Zero.\n\nBecause $\\mathbf{W}_{round}-z$ is ensured to be an integer, we replace the 0 with 1 and then zero mask the division result at corresponding slots.", "claims_and_evidence": "The claims are somewhat confusing and may be misleading. This work is mainly focusing on an adaptation method with pre-trained diffusion models. Only a couple of experiments with LLMs were presented, thus it is hard to justify the scope of this work beyond diffusion models due to lack of experiments and comparisons on LLMs. However, the introduction and the related work sections are very confusing since a lot of prior works were about LLMs with adaptations using LoRA variants or about quantized diffusion models without adaptations except for one work (EfficientDM, He et al., 2023). This confusion remains until the conclusion, thus makes the claims and conclusions very confusing as if this work have solved the problems in LLMs in general.\n\nThe proposed method was not fully investigated with LLMs in my view. However, the conclusion does not mention that this work is limited to diffusion models, thus may mislead that this work may work for all the cases including LLMs. The contributions of this work should be clearly described along with all supporting evidences.", "methods_and_evaluation_criteria": "This work reported performance in accuracy and what quantization was used. However, this work did not report computation time and memory usage during fine-tuning and pre-processing, which seem important to fully understand the capability of the proposed methods. It is also important to see these results not only for tuning stage, but also for the preprocessing stage as well as for the final merging stage. It seems that the proposed method has a quite heavy preprocessing stage with AQS and VMC.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "It is unclear why the selected experiments require quantization. Subject-driven generation and style-customized image generation do not seem to have large datasets (25 subjects, 18 styles), so it will be great if this work argues, justifies and reports the reasons to perform these tasks with quantized models and the clear, realistic evidences for them.", "relation_to_broader_scientific_literature": "This work was very confusing due to mixing references about LLMs with adaptation, diffusions without adaption and so on. It will be great if this work focuses on a single topic, justifies and investigates that with full experiments, and concrete conclusions without over-claims. Or this work should focus on LLMs like other related works on LoRA variants to clearly show the advantage of this work over prior works.", "essential_references_not_discussed": "1) It seems that this work missed the most important work by \nHan Guo et al., LQ-LORA: LOW-RANK PLUS QUANTIZED MATRIX DECOMPOSITION FOR EFFICIENT LANGUAGE MODEL FINETUNING, ICLR 2024\n\nThis work is not about LLMs, but addressed the issue of zero-initialization of LoRA by proposing a matrix decomposition, which might be related to the proposed method in this work. Comparing with this work seems critical for me to properly assess this work. There is also another work called GPTQ-LoRA by Chai et al. See the above work for more info. Thus, it seems very important to properly discuss and compare with this work.\n\n2) While the fundamental basis of this work is in diffusion models and their adaptations, this work failed to properly cite and discuss these works. Without proper justification of quantized adaptation, it will be difficult to appreciate the current work. For example, subject-driven generation does not have to fine-tune the LoRA weights. See the following work:\n\nRinon Gal et al., AN IMAGE IS WORTH ONE WORD: PERSONALIZING TEXT-TO-IMAGE GENERATION USING TEXTUAL INVERSION, ICLR 2023\n\nThis work achieved subject-driven generation by only fine-tuning a single token, which is much smaller than the LoRA weights. For the cases like this, it will be very hard to justify the necessity of using quantized models for fine-tuning. Thus, this work should search for literature on the tasks of adapting diffusion models to tasks, to see if the proposed method is indeed useful for many cases.", "other_strengths_and_weaknesses": "- Did R introduce additional memory? If so, then the goal of quantizing diffusion models may not be satisfied well due to increased memory usage. It seems that R looks quite large, actually the same size as W, so comparing with other methods in Table 1 and other tables may not  be fair. Table 1 simply wrote nbits, which can not explain this additional memory and thus, the proposed method may use more memory. Thus, please add the info about actual GPU memory usage, computation time of pre-training, fine-tuning and merging, and so on.\n\n- Sigma_R in Eq. (6) seems important to determine. How can we guarantee that the optimal one exists? How sensitive it is to other tasks and baseline pre-trained models? These properties do not seem to be well-investigated since similar baselines were used (SD and SDXL). Introducing additional parameter on top of the rank of LoRA seems undesirable.", "other_comments_or_suggestions": "- It is unclear how important it is to solve the problem of fine-tuning quantized diffusion models, so it will be important to justify it with concrete examples and applications in the introduction.\n\n- It is unclear if Figure 1 is accurate: A hat and B hat are INT4, but tuning with FP16 activations may require floating point operations. Figure 1 should be accurate to reflect these facts if they are true.\n\n- I strongly recommend to revise this work to clearly differentiate among LLMs works with adaptations, Diffusions without adaptations and so on and then to clearly indicate the contribution of this work.\n\n- In Figure 3, is Adaptation Term also Int4 or FP16? Is the fine-tuning operating on FP16 then?", "questions_for_authors": "- Was there any divided-by-zero issue in Eq. (3) for the practical cases?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E5aqWaYb5j", "reviewer": "Reviewer_wfKN", "summary": "The paper introduces IntLoRA, which uses integer-type LoRA weights to fine-tune directly on the quantized models, for both training and inference efficiencies. To achieve this, the authors propose three novel techniques. First, the authors propose the Adaptation-Quantization Separation (AQS), which allows for the coexistence of zero-initialized gradients and a quantization-friendly distribution. Then, the Variance Matching Control (VMC) mechanism is developed to fine-tune the channel-aware variance, ensuring favorable distribution shape for log2 quantization. Last, the Multiplicative Low-rank Adaptation (MLA) is used to allow independent optimization of pre-trained and adaptation terms, enlarging the parameter space of LoRA fine-tuning. Extensive experimental results indicate that IntLoRA achieves sota results on both efficiency and performance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> ### More discussions of Adaptation Quantization Separation (AQS).\n\nThe main motivation of the proposed AQS is to **address the effect of zero-initialized weights of LoRA during quantization**. Specifically, the zero-initialized weight leads to infinite results during quantization due to the divide-by-zero (please refer to Eq.1 in the original paper). In order to maintain the zero-initialized weights while achieving the quantization-friendly distribution, we propose the AQS, which decomposes the LoRA weights into a zero-initialized part that requires gradients (corresponding to the vanilla LoRA), and a non-zero part that does not require gradients (for ease of quantization) to facilitate quantization. In this way, AQS facilitates both model learning and quantization.\n\n> ### Correlation with Other Compression Methods, e.g., Knowledge Distillation.\n\nOur approach focuses primarily on quantization to reduce the memory footprint during fine-tuning large models. However, since quantization and other compression methods are orthogonal technically, **our approach can therefore potentially be combined with other methods such as distillation** to achieve further computational cost reduction. \n\nSpecifically, we can first use the large pre-trained model as a teacher model and first obtain a small student model. Afterwards, we can use the proposed IntLoRA on the student model to further obtain quantized downstream task weights, thus realizing further speedup.\n\nSince the knowledge distillation requires complete back-propagation of the model gradients, we cannot finish this process due to limited computational resources. However, given the promising performance of IntLoRA on existing small models (which can be viewed as distilled student models), combining IntLoRA and knowledge distillation is practical. We will give a more detailed discussion in the revision.\n\n> ### Responds to Other Questions.\n\n- We assume you refer to the rank in the LoRA. Following the common practice, we determine the LoRA ranks through ablation experiments in Fig.9 in the Appendix. Performance generally improves as we increase the rank, but the rate of growth varies. To balance a tradeoff, we choose $r=4$ in all of our experiments.\n \n- The motivation of VMC is to adjust the distribution shape of the adaptation term for effective log2 quantization. In detail, the log2 quantization requires that most of the values are distributed near zero to use as many log buckets as possible to reduce quantization error. As shown in Fig.8 in the Appendix, the proposed VMC can control the distribution shape to achieve sharp peaks and light tails, thus utilizing more buckets on the logarithmic scale to reduce the quantization error.", "claims_and_evidence": "The claims of this paper is well supported by evidence such as distribution visualization for quantization weights, the quantitative ablation results for method's design rationality.", "methods_and_evaluation_criteria": "•\tThe proposed method make sense. In the Method section, the motivation of each module is clearly stated before the technical details.\n\n•\tThe author also proposed the log2-quantized fine-tuning pipeline, which is hardware efficient. \n\n•\tThe evaluation metrics are commonly used in image generation tasks, such as FID, CLIPscroe, and the authors also gives sufficient qualitative results.", "theoretical_claims": "The process of quantization is well-justified with Adaptation-Quantization Separation, the Variance Matching Control, and Multiplicative Low-rank Adaptation.", "experimental_designs_or_analyses": "•\tThis work validate the effectiveness of IntLoRA on various diffusion generation tasks, including Dreambooth fine-tuning, ControlNet fine-tuning, and Style-customization, all tasks being widely adopted in previous diffusion fine-tuning works.\n\n•\tThe performance is noteworthy. The IntLoRA achieves sota performance on FID and CLIP on the Dreambooth fine-tuning, even under a more challenging integer-type low-rank weights.\n\n•\tThe experimental analysis is well supported by the empirical evidence.", "relation_to_broader_scientific_literature": "The integer-type Lora fine-tuning without additional PTQ is almost unexplored by previous work. The IntLoRA can inspire future works towards both training and inference efficient lora tuning pipeline.", "essential_references_not_discussed": "The authors give a detailed comparison with a highly related work, i.e., EfficientDM, with both functional analysis and solid experiments on the diffusion network quantization tasks.", "other_strengths_and_weaknesses": "**Strentgh:**\n1) The paper is well-motivated, and the sufficient experiments validate the effectivess of the proposed method.\n2) The presention is easy to follow and well-organized.\n\n**Weakness:**\n1) It is better to give more discussions for the dfiferent parts of the proposed method, such as e Adaptation Quantization Separation (AQS).\n2) The relationship with other model compression methods is not clear.", "other_comments_or_suggestions": "None", "questions_for_authors": "1) What is the effects of the rank of the parameter and how to determine it?\n\n2) What is the reason for using variance matching control (VMC) to adjust the distribution of the adaptation term?\n\n3) Does the proposed method can be combined with other model compression methods, such as knowledge distillation?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BKTHldnrXn", "reviewer": "Reviewer_cH9A", "summary": "This paper proposes the IntLoRA quantization method, which can adapt quantized diffusion models with integer-type low-rand parameters to include inference efficiency during training. The proposed IntLoRA enables the pre-trained weights to be quantized during training and the IntLoRA weights can be seamlessly merged into pre-trained weights to obtain the quantized downstream weights without PTQ.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> ### Results on More Diffusion Models.\n\nGood comments! As suggested, we evaluate our IntLoRA on the FLUX.1-dev model. Since FLUX is notoriously costly to fine-tune even using LoRA, due to limited computational resources, we only give the results of FP16 vanilla LoRA and our IntLoRA on 15 text-subject pairs of Dreambooth. The results are shown below.\n\n\\\\begin{array}{l|ccccc} \n\\\\hline\n\\text{methods}        & \\text{nbits}  & \\text{DINO}   & \\text{CLIP-I} & \\text{CLIP-T} & \\text{LPIPS}  \\\\\\\\ \\\\hline\n\\text{FLUX-LoRA(FP)}  & W16A16 & 0.3564 & 0.6490 & 0.2501 & 0.8383 \\\\\\\\\n\\text{FLUX-Ours(MUL)} & W8A8   & 0.3150 & 0.6272 & 0.2348 & 0.8372 \\\\\\\\ \\\\hline\n\\end{array}\n\nIt can be seen that the performance drop of our IntLoRA compared to the original LoRA is acceptable, while achieving inference speedup. This preliminary result shows the  potential generalization of the proposed method on other Diffusion model backbones.\n\n> ### Efficiency Comparison.\n\nIn Tab.3 of the paper, we give the forward time cost of MUL and SHIFT: 0.87s/img for MUL, and 0.84s/img for SHIFT. One can see that the SHIFT is relatively faster than the MUL. However, it is worth noting that the speedup of the log2 quantized model can be further accelerated by hardware-level optimizations. Since this work focuses primarily on the algorithmic level, we leave further log2 acceleration for future work.\n\n\n> ### Typos.\n\nWe will fix the types in the caption of Figures 4-6 in the revision. Thanks for your suggestion!\n\n\n> ### Responds to other Questions.\n\nIn weight merging of previous methods, i.e., $\\mathbf{W'} = \\mathcal{Q}(\\mathbf{W) + AB}$, the $\\mathcal{Q}(\\mathbf{W})$ is the quantized INT8 weights, but $\\mathbf{AB}$ is low-rank FP16 weights. This arithmetic inconsistency between FP16 and INT8 caused the $\\mathcal{Q}(\\mathbf{W})$ to have to upcast back to FP16 to allow addition with LoRA weights. As a result, the summed results $\\mathbf{W'}$ is in FP16 type and needs PTQ to accelerate inference.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "I have checked the proofs for the proposed IntLoRA quantization adaptation.", "experimental_designs_or_analyses": "I have checked the soundness/validity of the quantitative comparison on different generation tasks and the comparison of training and inference efficiency.", "relation_to_broader_scientific_literature": "The proposed method solves the problem of additional PTQ for the adaptation of quantized diffusion models.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths: this paper is well-written and easy to follow. The proposed IntLoRA is a novel method to solve the problem of additional PTQ for the adaptation of quantized diffusion models. Extensive experiments have been conducted to verify the effectiveness of the proposed method. Besides, the code is also provided in the supp.\n\nWeaknesses: It is better to validate the effectiveness of the IntLoRA on more diffusion models other than the StableDiffusion.", "other_comments_or_suggestions": "(1) It is better to also compare the inference speed of the SHIFT and MUL variants of IntLoRA.\n\n(2) Figures 4-6 should be termed as qualitative comparisons other than quantitative comparisons.", "questions_for_authors": "In Figure 1(a), why the quantized merged weights are in FP16?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces IntLoRA, a LoRA-based method for efficiently fine-tuning quantized models using integer-type LoRA weights. It proposes three key techniques: Adaptation-Quantization Separation (AQS) to support zero-initialized gradients within quantization-friendly distributions, Variance Matching Control (VMC) to align channel-wise variance for log2 quantization, and Multiplicative Low-rank Adaptation (MLA) for flexible parameter optimization. Fine-tuning is conducted on the adaptation term and merged via integer operations. Experiments on both diffusion and LLM tasks demonstrate strong performance and efficiency.

Three out of four reviewers recognized the soundness of the proposed techniques and the strength of the experimental results. One reviewer raised concerns about the paper’s clarity and focus, noting that although the work claims to target diffusion models, it includes substantial content and experiments on LLMs. Additionally, the reviewer pointed out a lack of discussion and comparison with important LLM baselines such as GPTQ-LoRA and LQ-LoRA. The authors have acknowledged these points and committed to addressing them in their revision.

Given the overall contributions and the authors’ willingness to improve the manuscript, we believe the strengths of this submission outweigh its limitations. We therefore recommend acceptance, and encourage the authors to revise the paper in accordance with the reviewers’ feedback and their rebuttal commitments.
Decision: Accept (poster)