CLoQ: Enhancing Fine-Tuning of Quantized LLMs via Calibrated LoRA Initialization
Authors: Yanxia Deng
Aozhong Zhang
Selcuk Gurses
Naigang Wang
Zi Yang
Penghang Yin
Venue: Accepted by TMLR ()

Abstract: Fine-tuning large language models (LLMs) using low-rank adaptation (LoRA) has become a highly efficient approach for downstream tasks, particularly in scenarios with limited computational resources. However, applying LoRA techniques to quantized LLMs poses unique challenges due to the reduced representational precision of quantized weights. In this paper, we introduce CLoQ (Calibrated LoRA initialization for Quantized LLMs), a simplistic initialization strategy designed to overcome these challenges. Our approach focuses on minimizing the layer-wise discrepancy between the original LLM and its quantized counterpart with LoRA components during initialization. By leveraging a small calibration dataset, CLoQ quantizes a pre-trained LLM and determines the optimal LoRA components for each layer, ensuring a strong foundation for subsequent fine-tuning.
A key contribution of this work is a novel theoretical result that enables the accurate and closed-form construction of these optimal LoRA components. We validate the efficacy of CLoQ across multiple tasks such as language generation, arithmetic reasoning, and commonsense reasoning, demonstrating that it consistently outperforms existing LoRA fine-tuning methods for quantized LLMs, especially at 2-bit.

Published in Transactions on Machine Learning Research (08/2025)
CLoQ: Enhancing Fine-Tuning of Quantized LLMs via Cali- brated LoRA Initialization
Yanxia Deng' Deportment of Mothematics and Statistics University at Albany, SUNY
ydeng5@albony. edu
Aozhong Zhang
azhang38allaaryyycmmm edu
Department Mathemalics and Statistics University at Albany, SUNY
Selcuk Gurses
gourses@ollanne edu
Department of Mathematics and Statistics University at Albang, SUNY
Naigang Wang
mwang himppm..s.mmmmmmmmmm com
IBM T. J. Watson Research Center
Zi Yang
zyangStalbony. edu
Deportment of Mathematics and Statistics University at Albang, SUNY
Penghang Yin
pyinnolbbany.coo edu
Department of Mathematics and Statistics University at Albany, SUNY
Reviewed on OpenReview: https: openreview. net/ forum? id= dZ.
Abstract
Fine-tuning large language models (LLMs) using low-rank adaptation (LoRA) has become a highly efficient approach for downstream tasks, particularly in scenarios with limited computational resources. However, applying LoRA techniques to quantized LLMs poses unique challenges due the reduced representational precision of quantized weights. In this paper, we introduce CLoQ (Calibrated LoRA initialization for Quantized LLMs), simplistic initialization strategy designed to overcome these challenges. Our approach focuses minimizing the biser-wise discrepancy between the original LLM and its quantized counterpart with LoRA components during initialization. By leveraging small calibration dataset, CLoQ quantizes a pre-trained LLM and determines the optimal LoRA components for each layer, ensuring a strong foundation for subsequent fine-tuning. A key contribution of this work novel thcorctical result that enables the accurate and closcd-form construction of these optimal LoRA components. We validate the efficacy of CLoQ across multiple tasks such as language generation, arithmetic reasoning, and commonsense reasoning, demonstrating that it consistently outperforms existing LoRA fine-tuning methods for quantized LLMs, especially at 2-bit. The code is available at https::// gi thub. com /ao//oonZZhang//Coo
"Equal contribution. "Corresponding pyindalbany.co edu.

---

Published in Transactions on Machine Learning Research (08/2025)
1 Introduction
Large language models (LLMs) Achiam et al. [0033] ;
Touvron et al. (2023 Jiang et al. (2023) ; Guo et al.
(2024a) have achioved remarkable success across a wide range of domains and applications. With ongoing advancements, the size and complexity of LLMs have grown exponentially, with some models now exceeding billions or even trillions of parameters. Although this scaling has unlocked unprecedented capabilities, it also introduces significant challenges, particularly in efficiently adapting these models to downstream tasks. Traditionally, full fine-tuning has been the dominant approach for adapting pre-trained models, involving updates to all model parameters. While effective in achieving state-of-the-arl results, full fine-tuning is resource-inte interrev,, requiring substantial GPU memory to store both model weights and optimizer states. These memory domands grow with the size of the model, making full fine-tuning incrcasingly impractical for large-scale models resource-constrained settings.
To address these challenges, parameter-effiiint fine-tuning (PEFT) Houlsby et al. (201 , such as Low-Rank Adaptation (LoRA) Hu et al. (2021) has emerged promising approach. PEFT updates only small subset of parameters while keeping the majority of the model unchanged, enabling resource-efficien fine-tuning of large-scale models. LoRA, for instance, introduces small, Icarnable low-rank matrices into the model's architecture. These matrices are optimized during fine-tuning while the original model weights remain frozen, significanlly reducing memory and computational requirements. This design leverages the insight that weight updates often reside in a low dimensional subspace, allowing LoRA to achieve efficient adaptation with minimal overhead.
In an orthogonal direction, model compression techniques notably quantization Rastegari et al 2016
Hubara et al. [2018): Choi et al. 2018): Wang et al. (2018): Yin et al. (2016) 2018 2019b Li et al. (202bb) Shao et al. (2023): Zhang et al. (2023): Frantar et al. (2022a) Zhang et al. (2024a). have been developed
to minimize GPU memory usage by converting high-
weights into low-precision representations Initially designed for inference in memory-limited environments, quantization methods have since been adapted to support fine-tuning. A notable advancement in this regard is QLoRA Dettmers ct al. (2023)
which combines LoRA with quantization techniques to reduce GPU memory requirements for fine-tuming significantly. By leveraging low-rank adaptation and quantized weights, QLoRA enables resource-efficient
fine-tuning of LLMs without compromising performance, making it powerful tool for adapting large-scale models to downstream tasks. However, extending LoRA to quantized LLMs introduces additional challenges, as the reduced representationa ional precision of quantized weights can disrupt standard initialization strategies, impacting task performance. Recent works Li ct al. (2023a) | Yao et al. (202)); Liao ct al. (2024); Guo et al.
(2024b) proposed minimizing quantization error through strategic initialization of the low-rank components
in LoRA, to align with the original weight states of the model. This strategy has demonstrated success in fine-tuning lower-bit quantized LLMs.
Contributions. In this paper, we introduce CLoQ (Calibrated LoRA for Quantized LLMs), an efficient layer-wise, data-driven initialization strategy specifically designed for quantized LLMs by leveraging a small calibration dataset. CLoQ consists of two main steps: a post-training quantization phase to obtain quantized weights and a generalized low-rank approximation phase under a linear transformation to compute the corresponding optimal adapters. We derive a novel closed-form solution to the low-rank approximation problem, which can be efficiently computed using just two singular value decompositions (SVDs).
Our CLoQ method requires no back-propagatiinn making it highly efficient LoRA initialization scheme for fine-tuning quantized models. We demonstrate the effectiveness of CL.oQ through extensive validation on multiple benchmark datasets. Our results show that CLoQ consistently outperforms existing LoRA methods for quantized LLMs, particularly at ultra-low bil-widths. For instance, the fine-tuning accuracy of INT2 CLoQ on the Llama2-132 model Touvron et al. (2023) surpasses that of INT4 QLoRA Dettmers et al. 2023 in the arithmetic reasoning tasks, as shown in Table
Notations. We clarify the mathematical notations that will be used throughout this paper: we denote vectors by bold small letters and matrices by bold capital ones. For any matrix X E gmxx X puxm is the transpose of X, and Tr(X) = Σm Xz,i denotes the trace of X when m = n. For any two matrices X.Y Y E RR (X,Y) Tx YY = EE XiyYis is the inner product. We denote the
2

---

In addition, for any diagonal matrix Published in Transactions on Machine Learning Research (08/2025)
QLoRA
LoftQ
CLoQ
LoRA
WikiText-2(7B)
GSM8K(7B)
Arithmetic Reasoning(7B)
(96
Accuracy (%6
Perplexity
30
20
WikiText-2 (13B)
GSMBK
Arithmetic Reasoning
(96)
Accuracy (%6)
Perplexity
20-
3
Bit
Figure 1: Fine-tuning results of Llama2-7B and Llama2-13B across various tasks. Left: the perplexity measured on WikiText-2. Middle: the accuracy achieved on GSM8K. Right: the average accuracy across multiple arithmetic reasoning tasks.
Spectral narm the initialization difference
Frobenius norm of the initialization difference
LoRQ
CLoQ
Lant
CLoQ
500
800
400
Discrepancy
300
500
200
100
100
21
68
175
211
21
68
175
211
Layerindex
Layer Index
Figure 2: The discrepancy | AB WIll between the LoRA initialization and the original pre-trained weight matrix, computed using the spectral norm and the Frobenius norm, respectively. The layer shown in the figures above is randomly selected from the Llama2-7B model. The initialization is derived using both CLoQ and LoftQ under INT2 quantization. Notably, CLoQ significantly reduces this discrepancy, demonstrating its offectivencss.
Frobenius norm of X by |X == / Tr x)
v E m
Σラ=1 x2
Σ diag((),.... Jn), we denote its square root. by Σ diag(o ......oo))
3

---

Published in Transactions on Machine Learning Research (08/2025)
2 Background
Integer Quantizer. Given a set of m weights w E RR",, the widely-used b-bit uniform integer (INT) quantizer
Choi et al. (2018) determines the float scaling factor = max(w) m and zero-point 2
min( (w)
2--1
where [- is nearest-to-round operation. The quantizer projects w onto the equally spaced grids {2 6, (2 1) -,.))........ .,.... (2+ (2 1)) 67 to obtain the quantized weights
w + +,, -)) = 8. (clip (1²
For channel-wise (or group-wise) quantization, the scaling factor ó is shared by the quantized weights within the same channel (or group, respecitively).
Post-Training Quantization. Post-training quantization (PTQ) has been cornerstone technique for compressing LLMs. PTQ methods directly identify low-precision representations of a model without requiring retraining, making them particularly well-suited for extremely large-scale AI models including LLMs. The simplest approach in this calegory is data-free quantization, commonly referred to as RTN, which involves rounding the pre-trained model weights to their nearest quantized states. More advanced PTQ algorithms, such as OPTQ Frantar et al. (2022b), leverage a small calibration dataset to solve least-squares problem under discrete constraints:
min XXQ XW I QEO
(1)
to calibrate the quantization layer by layer, where W is the weight matrix of the pre-trained model. In this formulation, X denotes the activation or feature matrix corresponding to a batch of calibration data for a fixed layer. This approach ensures that the quantization process preserves the layer's output by minimizing the discrepancy between the original and quantized representations on the calibration dataset. Several efficient pack-propagation-fiec algorithms Zhang et al. (2023) : Behdin et al. (2023); Zhang et al. (2024b) have boen proposed to address I
Low-Rank Adaptation. LoRA Hu et al. (2021) enables efficient fine-tuning of large pre-trained models by introducing two small, trainable matrices, A and B, which are added to the frozen weight matrix W. The weights of the fine-tuned model are then expressed as W + AB where A E Rmxr B E Rx and r < min(m, n). During fine-tuning, only A and B are updated, while W remains fixed, significantly reducing the number of trainable parameters and computational overhead. LoRA initializes its parameters as follows:
A A~ A~,(0002))
B 3=0, 0,
ensuring that the initialization maintains perfect alignment with the pre-trained weights W. Recent research has focused on developing variants of LoRA aimed at enhancing its performance Wang et al. (002a Liu
et al. 2024): Wang el al. (2024c); : Meng el. al. [2024]: Büyūkakyüz (2024).
3 Method
CLoQ designed to enhance the fine-tuning of quantized LLMs by incorporating calibration data, building upon OPTQ Frantar et al.) (2022a) with a specific focus on the initialization of LoRA adapters. It utilizes second-order information derived from input activations X to ensure that the low-rank adapter matrices A and B are initialized in manner that minimizes quantization error, particularly in relation to the data distribution. This alignment allows CLoQ to improve the fine-tuning process by making the low-rank adaptation more responsive to the model's behavior during calibration, thereby reducing the mismatch between the quantized model and its full-precision counterpart. Importantly, CLoQ uses the same calibration data as OPTQ, ensuring that the initialization of LoRA adapters both universal and effective across various downstream tasks. In our experiments, we use the Wikitext dataset for calibration, offering robust and generalizable foundation for fine-tuning across range of task domains.
4

---

Published in Transactions on Machine Learning Research (08/2025)
3.1 Calibrated Quantization and Low-Rank Initialization
Given the activation matrix X associated with the calibration data, CLoQ aims to solve the following optimization problem for LoRA initialization:
min
IX(Q+ AB' w)ll
(2)
QEQ.AERTXI BER BER
where X E (b-l) is the activation matrix associated with a calibration data set of b samples, each represented as an I x m sub-matrix and stacked along the row dimension. Q C R is an appropriate set of all feasible quantized weights. The objective of (2) is to ensure that the initialized weights for the LoRA model, Q + AB", closely approximate the pre-trained model weights W e gmxnn when applied to the activation matrix.
This problem can, in theory, be solved using alternating minimization (AltMin) method, whose 1-th iteration reads:
Q 4+ arg min QEQ
X(Q (B)) T W
A+ ++ B = arg min A,B
X(A + +Q -
In practice, observe that it suffices to just perform single iteration with the initialization (B)) 0. Therefore, the proposed CLoQ method comprises two steps: quantization step and low-rank approximation step under linear transformation, which we detail in the rest of this section.
After solving the above problem and obtaining Q, A, B, we fix the quantized weights Q and update only the low-rank components A, B during the subsequent fine-tuning stage, as per the LoRA approach.
3.1.1 Post Drainirann Quantization
With AB initialized to zero, the problem of finding Q simplifies to the standard layer-wise post-training quantization (PTQ):
min |X(Q W)
(3)
QEQ
an optimization problem that has been extensively studied in recent literature Frantar et al. (022b) Zhang
et al. (2023); Chee et. al. (2023); Xiao et al. (2023); Zhang et al. (2024b). For this, we adopt the widely- used OPTQ method Frantar et al. (2022a). To further enhance OPTQ's performance,
incorporate preprocessing technique called weight magnitude reduction (MagR) Zhang et al. (2024a). which modifies W by removing outliers prior to quantization. MagR preprocessing signilicantly improves OPTQ's effectiveness in the low-bit regime while introducing minimal additional time beyond the OPTQ process and incurring no computational or memory overhead during inference time.
3.1.2 Generalized Low-rank Approximation
After obtaining Q in the quantization step, we denote by
AW WV=WQQ
the residual of the quantized weights. To determine A and B, we solve the following low-rank approximation problem under the lincar transformation induced by X:
AERmxr, BeRex I(AB A.
min
(4)
It important to note that problem non-trivial due to the presence of the matrix X, and optimal solution not given by directly computing the low-rank approximation (or SVD) of AW. However, we demonstrate in the following result that the problem be solved accurately by performing just two SVDs:
5

---

Published in Transactions on Machine Learning Research (08/2025)
Theorem 3.1. Suppose the activation motric X E (b (b.1 m) is of full-ronk. Suppose the Gram
for Hession) matrix H X X E mm has the SVD H = =U and denote by R U the non-smmmetric root of H. Then any pair (A, B) satisfying
AB R R (RAW).
(5)
permits an optimal solution to problem the best vunk-r opprocimation of RAW.
Here LR, (RAW) arg min 1 RAW denotes
Proof. Firstly, observe that the objective in 4 has the following equivalent expression:
|X(AB* A AW
I
Tr((AB AW) X X(AB - AW))
=Tr((AB T H(A - -AW)
= =Tr((AB AW) R 'R R(AB AWW)
=||R(AB' -EAB"-AWW I
-|RAB" - RAW 1
where in the third equality, we used the identity: R R. Moreover, since X is full rank, H invertible,
and so is R.
Based on these facts, we interpret problem 44 as finding the standard best rank-r approximation of RAW, LR, (RAW), which can be obtained by performing the SVD of RAW and extracting the top-r principal components Eckart & Young 1936). Then, (A, B) is an optimal solution to
min AERmXr AERmxr,BERoxx BERaX
RAA RAWI
if and only if
RAB T (RAW)
Consequently, since R is invertible, any () fulfilling
AB R (RAW).
permits an optimal solution to problem
To apply Theorem 3.1 to solve problem we make the following observations:
One SVD is required to compute R and another is needed to determine LR, Given that R E gm and R&W Rmxn, while X E (b-)))mmm the computational complexity of SVDs required for solving is independent of b I, which is significantly larger than m or n in practice. Here I represents the context length, and denotes the size of the calibration dataset. We note that CLoQ requires fewer SVD computations than LoftQ Li et al. (2023a). which, by default, performs five AltMin ilerations, each involving one SVD. Although CLoQ uses the more expensive GPTQ for the quantization (versus the data-free RIN in LoftQ), their overall initialization runtimes are comparable, as shown in Table 10
Indeed, 5 admits infinitely many optimal solutions. Suppose (RAW) has the form E. V In our experiments, we consistently take A R E, and B = V.r, which empirically performs well. However, is clear that (A, B) is an optimal solution, then for any invertible pair (AC, B(C=l)TT also satisfies (5) and thus provides an optimal solution to d4. For example,
:r.
Rexr, the
B) (R-'U E., or (R U E V, In Section 5, our ablation study shows that the combination (R-'U. E.,, v.r) gives the best practical performance during the subsoquent LoRA tne-tuning process. A more comprehensive theoretical analysis of the impact of initialization like Hayou et al. (202)): Li et al. (2244) will be addressed in future work.

---

Published in Transactions on Machine Learning Research (08/2025)
When H is not invertible poorly conditioned, we propose adding small constant À to the diagonal elements. Specifically, we typically set 0.01 Tr(H) following a strategy similar to that used in the prior works Frantar et al. 2022b); Chec et al. (2023 This adjustment has consistently proven effective in miligating numerical issues and ensuring stability during computations.
In theory, even if X is rank-deficient, the optimality condition RAB" T (RAW) in the proof of Theorem 3.1 still holds. But in this case, since R is also rank-deficient, this optimality condition permits infinitely many solution for AB and we may simply take AB = (RAW), where R is a pseudo-inverse.
To summarize, our proposed CLoQ method is detailed in Algorithm
Algorithm CLoQ for initializing one linear layer
Input: Regularized Gram matrix of activations H X X + AI R m Pre-trained weight matrix
W E mmxx Rank < min 12).
Output: Quantized weight matrix Q E ERmx Low-rank components A MERT Rm B
1: Solve 3 to obtain the quantized weights Q.
2: Compute the residual of quantized weights AW = W Q E Rm
Perform SVD: H UUH
Evaluate: R U
Perform the SVD of RAW to find its best rank-r approximation: LR, (RAW) U, V
Compute the low-rank components:
A R- U,
B= V.
Return: Q,A,B
4 Experiment
In this section, we evaluate the effectiveness of CLoQ language modeling, arithmetic reasoning, and commonsense rcasoning tasks. The fine-tuning through CLoQ consists of two key stages: the initialization step and the fine-tuning step. In the initialization step, quantize the full-precision weight W into low-precision weight Q and find optimal low-rank matrices A and B that minimize the residual error. In the finetuning step, the quantized weight matrix Q is fixed in low-precision, while A and B are trained through back prop-propagation. The detailed hyperparameter settings for all our experiments are presented in the Appendix A
Models and Datasets. We test CLoQ on Llama2-7b. Llama2-13h Touvron et al. 20233 Llama3-8h Grattafiori et al. (2024) and Mistral-7bb-0.11 [iang et al. 2023) models. Following prior works Frantar et al. (2022a) we randomly sample 128 instances, each with context length of 2048 tokens, from the WikiText-2 dataset. Merity et al. (2016) to serve as the calibration set for quantization. Then, we fine-tune and evaluate the models on WikiText-2 for language modeling. For single arithmetic reasoning tasks, we fine-tune and evaluate on the GSMSK Cobbe et al. (2021). For multi arithmetic reasoning, we fine-tune the models on Math10K Hu et al. (2023) and then evaluate the test sets of AQuA Ling et al. (2017), GSMSK, MAWPS al. (2016) and SVAMP Patel et al. (2021). For commonsense reasoning tasks, we line-tune the models on Commonsensel 70K 100K Hu al.] (2023) and evaluate on eight representative tasks:
Konnel-Kedziorste
BoolQ Clark et al. (2019). PIQA Bisk et al. WinoGrande Sakaguchi et al. (001)), ARC-c, ARC-c Clark et al. (2018) and OBQA Mihaylov et al.
(020), SIQA (Sap et al. (2019). HellaSwag Zellers et al. (019
2018
Baselines. We compare with LoRA Hu et al. 2023), LoftQ Li et al. (2023a) and ApiQ Liao et al. (0244). LoRA is often considered as the benchmark for fino-tuning performance. QLoRA incorporates NF-quantization, with its low-rank initialization aligning with the standard LoRA method. GPTQ-LoRA integrates OPTQ for the base weights and fine-tunes the LoRA component while preserving its low-rank initialization in the standard LoRA, with the quantized
(0021), QLoRA Detimers et al. 2002)). GPTQ-LORA GPT
7

---

Published in Transactions on Machine Learning Research (08/2025)
weights kept frozen. In contrast, LoftQ and LQ-LoRA carefully initialize the quantized weight and low-rank matrices by solving some optimization problems to minimize the approximation error. Furthermore, ApiQ uses gradient-based block-wise optimization to specifically initialize the low-rank components during post-training quantization.
4.1 Implementation details
Quantization. We quantize the weights of all linear layers in the base model using MagR preprocessing Zhang et al. (2024a) followed by OPTQ Frantar et al. (2022a). The guantization scheme employs uniform (a.k.a. INT) and asymmetric quantization, with a default group size of 64. After quantization, we compute the LoRA components A and B, which are maintained in FP16 precision.
(2023a): Liao al. (2024). we fine-tune the models using the standard LoRA conliguration, with modifications the LoRA initialization and learning rates. The quantized weights remain fixed, and only the LoRAA adapter matrices are trainable during fine-tuning. The rank of the LoRA adapters is consistently set to 64 across all methods. For optimization, we use AdamW Loshchilov 2017). All experiments are conducted on NVIDIA A100 GPUs with 80GB of
Fine-tuning. Following prior works Dettmers al. (2023); Li
memory.
4.2 Fine-tuning Results
Language modeling. We evaluate the models by reporting the perplexity of language generation on WikiText-2. As shown in Table [ and 2 CLoQ consistently outperforms existing methods across most bit levels and model architectures. Notably, at INT2, CLoQ proves effective, achieving a perplexity reduction of 0.95 over ApiQ-Iw and 1.34 over LoftQ on Llama2-7B, and even surpasses ApiQ-bw by 0.1 perplexity. This result highlights CLoQ's ability to maintain superior performance even under ultra-low bit quantization constraints.
Table 1: Finetuning results of WikiText and GSMSK on Llama2-7B and Llama2-13B.
Llama2-7B
Llama22130
Method
Bit
Wiki (ppli)
GSM8K (aer")
Wiki (ppll)
GSM8K (acet)
3,88
36.9
QLattA
4
3.70
35.1
5.22
30.9
LoftQ
5.24
35.0
45.0
ApiQ-la
4
5.28
36.4
4.78
50.4
ApiQ bw
5.27
39.9
4.78
51.2
CLoQ
5.25
40.6
4.78
49.3
QLoRA
3
5.23
32.1
5.22
40.7
ApiQu tw
3
5.53
36.0
4,98
45.1
ApiQuhw
CLoQ
3
5.45
39.9
4.92
48.1
QLoRA
2
N.A.
N.A.
N.A.
N.A.
LoRQ
2
7.85
20.9
7.09
25.4
ApiQu
7:80
26.0
6.29
36.3
ApiQ-hw 2
33.55
5.70
CLoQ
2
6.51
33.7
5.73
41.7
Table 2: Finetuming results of WikiText and GSM8K on Llama3-8B and Mistral- 7B,
Llama3-8H
Mistral-718
Method
(ppl))
GSMSK (neer)
(ppli)
(acet)
LoRA
16
6.34
47.8
5.17
52.8
LoftQ
ApiQq tw
2
7.18
41.3
ApiQ--ow
CloQ
2
9.49
45.4
6.68
45.0
Arithmetic reasoning (single task). To evaluate the models on GSM8K, we extract numerical answers from the generated solutions and determine accuracy by analyzing these extracted values. As shown in Table and 2 CLoQ achieves better performance across different model types and quantization bit levels. At INT2,
8

---

Published in Transactions on Machine Learning Research (08/2025)
CLoQ reaches accuracy of 33.7% Llama2-7B, which CLoQ achieves 7.7% improvement in performance compared with ApiQ-Iw. Moreover, the Llama3-8B model, CLoQ even outperforms ApiQ-bw by 1.0% at INT2. More remarkably, CLoQ achieves comprehensive superiority over all existing methods at INT2 and INT3 on Llama2-13B. On the Mistral-7B model, the accuracy of INT2 CLoQ is better than ApiQ-Iw and is comparable with ApiQ-bw.
Arithmetic reasoning. To evaluate CLoQ's effectiveness across multiple arithmetic reasoning tasks, we fine-tuned models the Math10K dataset and assessed their performance four separate math reasoning benchmarks, demonstrating their adaptability to diverse mathematical challenges. As shown in Table 3 and 4 CLoQ consistently outperforms other methods across various model architectures and quantization levels, achieving superior accuracy both on average and on individual datasets. Notably, at INT2 quantization, CLoQ delivers substantial gains, surpassing ApiQ-bw bw by 1.9% on LLaMA2-7B and by 2.1% on both LLaMA3-SB and LLaMA2-133 13B. Even INT4, CLoQ surpasses both LoRA and QLoRA on Llama2-7B, Morcover, for complex reasoning tasks such GSM8K and SVAMP, CLoQ achieves 4.1% and 1.7% higher accuracy than ApiQ-bw at INT2 on Llamaa-8B, highlighting its robustness in challenging problem seltings. These results underscore CLoQ's remarkable potential in handling intricate reasoning tasks with enhanced accuracy, even under ultra-low-be quantization.
Table 3: Accuracy on four arithmetic reasoning tasks. The LoRA rank is 64 for all methods.
Llama2-7B 7B MAWPS
Llama2- 13B MAWPS
Method
GSMSK
SVAMP
AQuA
Avg.
GSM8K
AQuA
Avg.
LoRA
43.6
59.4
85.0
27.0
53.7
55.3
67.7
87.4
24.4
58.7
QLoRA
42.7
58.7
87.3
26.4
53.7
54.8
69.4
87.0
26.8
59.5
GPTQ-LoRA
LoftQ
41.7
56.0
86.3
25.3
52.3
54.9
66.5
87.7
23.9
58.3
ApiQ-bw bw CLoQ
43.2
59.0
85.7
26.0
53.5
55.3
67.4
87.8
25.6
59.0
43.6
60.3
87.0
27.6
54.6
55.4
66.9
88.7
27.2
59.5
QLoRA
1.4
6.2
GPTQ-LaRA
38.9
55.7
84.9
23.2
50.7
50.6
65.2
88.0
22.6
56.6
LoftQ
39.9
56.3
26.4
52.2
53.9
23.0
57.7
AprQ-bw
41.4
35.9
87.0
25.2
52.
51.5
67.1
88.5
25.6
58.3
CLoQ
42.2
58.9
89.1
24.0
53.6
52.5
66.1
89.1
28.0
58.9
QLoRA
0.9
1.5
0.8
5.1
2.1
0.5
0.7
0.1
0.9
0.6
GPTO-LoRA
21.7
39.0
22.1
39.9
49.6
23.6
46.9
LoftQ
29.5
45.8
83.6
23.2
45.6
37.0
55.9
87.7
21.7
50.6
ApiQ-bw bw
31.2
51.0
82.9
23.9
47.3
43.1
59.2
85.1
23.4
52.7
CLoQ
34.7
52.0
86.1
24.1
49.2
44.6
57.6
88.7
28.4
54.8
Table 4: Accuracy on four arithmetic reasoning tasks.
The LoRA rank r is 64 for all methods. CLoQ accuracies are averaged over random runs, with standard deviations reported.
Llama3-8m MAWPS
Method
GSMSK
SVAMP
AQuA
Avg.
LoRA
68.8
76.1
90.3
31.5
66.7
LoftQ
35.6
52.1
87.0
25.2
50.0
ApiQ-bw
88.2
CLoQ
50.7+0.35
67. 67.5+0.8
88.340.91
29.5+0.56
59.0=
Commonsense reasoning. We further evaluate the effectiveness of CLoQ on commonsense reasoning tasks by fine-tuning the models on the Commonsensel105s dataset and testing their performance reasoning benchmarks, presented in Table 5 Consistent with its performance arithmetic reasoning tasks, CLoQ outperforms other methods across different model sizes and quantization levels, achieving notable improvements both on average and within each dataset.
eight
INT2, CLoQ delivers a performance boost over ApiQ-bw, with an average accuracy improvement exceeding 0.7%, even approaching the performance levels typically observed with INT4 QLoRA. For instance, INT2 CLoQ exhibits only minimal average accuracy drop of 0.04% compared INT4 QLoRA. Additionally, INT4, CLoQ attains impressive accuracy, reducing the gap FP16 LoRA to just 0.4%. These results indicate
9

---

Published in Transactions on Machine Learning Research (08/2025)
that CLoQ significantly enhances LoRA's learning capacity, enabling it to better adapt to diverse range of tasks.
Table 5: Accuracy on eight commonsense reasoning tasks.
The LoRA rank r = 64 for all methods
Model
Method
Bit
BoolQ
PIQA
SIQA
HellaS.
WinoG.
ARC-e
ARC-c
OBQA
Avg.
LoRA
16
73.6
86.5
81.8
95.2
86.9
89.4
76.7
86.7
81.6
QLoRA
73.9
84.4
79.7
93.3
84.6
86.1
73.0
86.1
82.5
GPTQ-LORA
4
73.4
83.6
79.3
93.3
81.5
86.5
72.8
83.3
82. 12.1
LoftQ
86.2
ApiQ-bw
73.5
87.0
82.0
95.2
86.9
89.5
77.0
86.2
84.7
CLoQ
74.2
81.6
96.1
86.9
75.7
86.4
84.2
Llama2-7B
GPTQ-LoRA
3
71.8
82.7
79.3
92.1
82.8
84.2
70.6
83.4
80.8
LoftQ
3
74.0
86.6
81.0
94.3
85.6
88.
75.4
86.5
83.7
AplQ-bw
3
73.3
85.6
81.8
91.6
86.9
87.9
73.7
86.4
83.8
CLoQ
85.0
GPTQ-LoRA
33.3
49.4
27.6
LoftQ
2
62.4
70.5
73..4
78.8
71.0
66.3
50.8
62.3
67.0
ApiQ-bw
2
68.4
80.7
79.6
91.4
82.4
82.7
68.3
80.5
79.3
CLoQ
70.2
82.2
78.9
91.7
82.2
83.6
70.7
81.4
80.1
LoRAA
QLoRA
81.5
94.9
86.9
89.
77.1
87.2
CPTQ-LoRA
74.5
86.1
81.8
94.7
86.8
89.0
77.1
84.5
84.3
LoftQ
76.0
87.9
82.8
95.8
88.9
91.2
80.8
88.8
86.5
AplQ-bw
76.2
83.5
96.6
90.0
89.9
87.3
CLoQ
75.7
88.4
82.9
96.3
89.4
91.1
81.9
90.0
87.0
Llama2-13B
GITQQLoRRA
3
73.5
85.2
81.1
94.1
85.7
87.9
75.5
85.3
83.55
LoftQ
3
75.2
87.8
82.8
96.3
89.5
91.1
81.4
88.
86.5
ApiQ-bw
3
76.0
88.0
82.3
95.8
89.1
91.1
81.1
89.5
86.6
CL.Q
3
75.3
88.1
82.8
95.9
90.1
91.2
80.7
90.6
86.8
GPTQ-LoRA
2
62.2
50.1
34.0
25.1
49.6
25.0
22.7
27.6
37.
LoftQ
72.7
ApiQ-bw
2
73.1
86.2
82.3
94.4
86.2
88.2
74.9
85.9
83.8
CLoQ
2
73.9
85.5
81.6
94.8
87.3
89.5
7T.4
81.8
84.4
Finc-tuning mixed dataset. We evaluate the performance of CLoQ for LoRA fine-tuning on mixed dataset comprising Math10K and 5K samples from Commonsense datasel, and assess accuracy four arithmetic reasoning tasks. Interestingly, we find that fine-tuning on the mixed dataset leads to a drop in arithmotic reasoning accuracy compared to fine-tuning solely on the Math10K dataset, as shown by Table For example, for 4-bit Llama2-7B, the averaged accuracy of CLoQ fine-tuned on mixed dataset drops to 51.20 11.2% from 54.6% (in Table 3). However, CLoQ still consistently outperforms LoftQ in this setting across different bit-widths.
Table 6: Accuracy of Llama2-7B for four arithmetic reasoning finetuned on mixed dataset
Arithmetic reasoning MAWPS
Method
Bit
GSM8K
SVAMP
AQuA
Avg.
Loft() CLoQ
4
36.8
55.3
83.2
24.8
50.0
38.8
85.7
25.2
51.2
LoftQ CLoQ
2
21.8
40.7
74.4
24.0
40.2
27.2
24.4
44.7
4.3 Ablation study
LoRA initialization with different (A, B) combinations. Furthermore, we investigate the performance of various combinations of (A, B) in Algorithm I as shown in Table 7 We tried three different combinations of (A, B), including (R- V,, U E,) and the default choice (R UR-'U, V.r). Table shows that the default combination of initialized adapters gives the best performance the subsequent fine-tuning phase.
Calibration data size. We explore the sensitivity of CLoQ the size of the calibration dataset. As shown in Table 8 CLoQ exhibits strong robustness the calibration size. Across both INT4 and INT2, the overall
10

---

Published in Transactions on Machine Learning Research (08/2025)
Table Fine-tuning results of different combinations of (A, B) on WikiText-2 and GSMSK. The LoRA rank is 64.
Llama2-7B
Method
Bit
Wiki (ppli)
GSM8K (acef)
LoRA
5.08
36.9
(R U.. E.)
880.6
1.6
(R U,,E, V E.)
6.68
12.9
(R. UU,,, V.
6.51
33.7
performance remains consistently stable the calibration size varies from 32 to 256. A calibration dataset of 128 samples, commonly used as the default in PTQ, yields slightly better results. These findings indicate that CLoQ does not heavily depend on the specific choice of calibration set size, which enhances its practicality and ease of deployment in real-world scenarios.
Table 8: Accuracy different calibration dataset sizes for Llama2-7B.
Arithmetic reasoning
Calibration Size
Bit
Wiki
GSM8K
GSMSK
SVAMP
MAWPS
AQuA
Avg.
5.34
40.2
28.0
54.4
64
5.33
40.9
44.7
57.6
87.0
25.2
53.6
128 (default)
5.25
40.6
43.6
60.3
87.0
27.6
54.6
256
5.33
40.1
42.6
59.6
87.0
26.0
53.8
32.5
24.0
64
6.56
33.5
35.5
54.0
84.5
25.6
49.9
128
6.51
33.7
34.7
32.0
86.1
24.1
49.2
256
6.49
33.2
33,5
47.2
87.4
27.6
48.9
Sequence length. Additionally, we present the results for different sequence lengths when finetuning the 2-bit Llama2-7B model in Table g on arithmetic reasoning tasks. It shows that the fine-tuning accuracy nicely improves as the sequence length increases.
Table 9: Accuracy of finetuned 2-bit Llama2-7B for four arithmetic reasoning with different sequence lengths.
Arithmetic reasoning
Sequence length
GSMSK
SVAMP
MAWPS
AQuA
Avg.
512
34.7
52.0
86.1
24.1
49.2
34.1
87.8
2048
34.5
52.9
87.8
24.6
50.0
Initialization Cost and Pastency/Memoot Benefits. We compare the duration and GPU memory usage of Initialization between CLoQ and other baseline methods. As shown in Table [T] CLoQ achieves efficient and scalable initialization, offering notable reductions in both runtime and memory consumption. For instance, CLoQ requires only 0.7 hours and 9GB of memory LLaMA2-7B, outperforming ApiQ-Iw (4.1h) and bp (1.3h) by large margins. On the larger LJAMA2-13B, CLoQ remains competitive, requiring just 1.5 hours and 13GB of memory, compared to 6.5 hours (ApiQ-Iw) and 27GB (LoftQ). These results demonstrates CLoQ's strong practicality for largo-scale model under limited hardware budgets.
5 Related Work
When quantizing pre-trained models, QLoRA Detimers et al. (023) primarily emphasizes the quantization process, often overlooking the critical importance of subsequent LoRA fine-tuning. adopts the fixup initialization stratogy used in LoRA, attaching zero-initialized low-rank adapters the quantized pro-trainod model. However, the discrepancies introduced by quantization, especially in extremely fow-bit regimes, significantly impact the initialization of LoRA fine-tuning, ultimately affecting the overall fine-tuning
11

---

Published in Transactions on Machine Learning Research (08/2025)
Table 10: The duration and peak GPU memory used for Llama2.
Size
Method
Duration
GPU memory
LoftQ
0.6h
14GB
Llama2-7B
ApiQ-Iw
4.1h
6GB
ApiQ-bw
1.3b
12GB
CLoQ
0.7h
9GB
LoftQ
1.3h
27GB
Llama2-150
ApiQ-Iw
6.5h
9GB
ApiQ-bw
2.4h
17GB
CLoQ
1.5h
13GB
performance. LoftQ Li et al. (223a) jointly optimizes the quantized weights Q and the low-rank adapter matrices A and B by solving the following optimization problem:
min |Q + AB - - Q.A.B
(6)
This approach ensures that Q, A, and are initialized to minimize the reconstruction error between the quantized and pre-trained weights. By aligning the quantized model's initial state more closely with its pre-trained counterpart, LoftQ enhances fine-tuning performance without requiring calibration data.
LQ-LoRA Guo et al. (022bb) assigns importance weights to each parameter by evaluating its sensitivity to output variations, thereby guiding the decomposition process toward critical regions. Specifically, the (diagonal) Fisher matrix used to weight the reconstruction objective during decomposition. To solve the resulting weighted SVD problem, LQQLoRA assumes row and column homogeneity the Fisher matrix, allowing the use of standard SVD techniques the cost of theoretical precision. However, this approach has limitations. It relies on approximations rather than solving the weighted SVD problem exactly, which may lead to suboptimal results. Furthermore, computing the Fisher matrix requires propaaaooan through the pre-trained model, introducing additional computational overhead. Similar to our work, ApiQ Liao et al. (2024) also employs an activation-aware initialization strategy utilizing calibration data. However, it relies on two activation matrices- one obtained from the pre-trained model and the other from the quantized model, whereas CLoQ uses only single pre-trained activation matrix. ApiQ optimizes the discrepancy using standard back- propaaropag,tion, whereas CLoQ adopts a fully gradient-frot approach. By avoiding the computational overhead of propppropa,,tion, CLoQ enables faster adaptation while maintaining high performance across various tasks.
6 Concluding Remarks
In this work, we introduced CLoQ, an efficient and scalable method for fine-tuning quantized LLMs. By leveraging a small calibration dataset, CLoQ optimally initializes LoRA adapters through a novel layer-wise, data-driven approach, significantly improving the fine-tuning process without the need for pack-prroaggtton. The use of closod-form solution for low-rank approximation, computed via two SVDs, ensures that CLoQ is both computationally efficient and highly effective, particularly at ultra-low bit-widths. Our extensive experiments on multiple benchmark datasets demonstrate that CLoQ consistently outperforms existing LoRA-basee methods for quantized models, such QLoRA, in tasks requiring fine-grained precision. The results underscore the potential of CLoQ to enhance the performance of quantized models across a variety of downstream applications, including those that demand high accuracy, like arithmetic reasoning lasks. The simplicity and efficiency of CLoQ make a promising approach for fine-tuning large-scale quantized LLMs in resource-constrained environments. Future work could further investigate the thcoretical implications of different decompositions the adapter matrices in CLoQ, and how these variations influence the performance of subsequent fine-tuning.
12

---

Published in Transactions on Machine Learning Research (08/2025)
7 Acknowledgments
This work was partially supported by NSF grants DMS2208126, DMS-2110836, 11S-2110546, SUNY-IBM AI Research Alliance Grant, UAlbany-IBM CEAIS grant, and start-up grant from SUNY Albany. We would also like to thank SUNY Albany for providing access to the Nvidia DGX Cloud.
References
Gpiglora:
Efficient. finetuning of quantized Ilms with gplq. GitHub repository, 2023. URL https://githui gptqlora
commqqwopwopp00/gptqq/ Available at
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Kayhan Behdin, Ayan Acharya, Aman Gupta, Sathiya Keerthi, and Rahul Mazumder. Quantease: Optimization-base based quantization for language models-an efficient and intuitive algorithm. arXio preprint arXiv:2309.01885, 2023.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Kerim Büytikakytiz. Olora: Orthonormal low-rank adaptation of large language models. ar preprint arXiv:2406.01775, 2024.
Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher M De Sa. Quip: 2-bit quantization of large language models with guarantees. In Advances in Neural Information Processing Systems, 2023.
Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce T-Jen Chuang, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks, arXiv preprint Xiv: 0608, 2018.
Christopher Clark, Kenton Lee, Ming-Wo Heig-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXic:1905.100// 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge. arXie preprinl arXiv:1803.05./ 157,00050,, 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Retichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110. 14168, 2021.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.4331, 2023.
Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank. Psychometrika, (3):211-218, 1936.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210. 17323, 2022a.
Elias Frantar, Salch Ashkboos, Torsten Hocfler, and Dan Alistarh. Optq: Accurate quantization for generative pre-trained transformers. In The Eleventh International Conference on Learning Represenlations, 2022b.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama herd of models. arXiv preprint ar 2777:22477.11788 2024.
13

---

Published in Transactions on Machine Learning Research (08/2025)
Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Ti, et al. Deepseek-coder: When the large language model meets programming-uee the rise of code intelligence. arXiv preprint arXiv:2401.14196 2024a.
Han Guo, Philip Greengard, Eric Xing, and Yoon Kim. LQ-IoRA: Low-rank plus quantized matrix decom- position for efficient language model finetuning. In The Twelfth International Conference on Learning Representations, 2024b. URL httpp:://peenrevee...eee//e wttp:///eoorrm///0000000000
Soufiane Hayou, Nikhil Ghosh, and Bin Yu. The impact of initialization on lora finetuning dynamics. arXiv preprint arXiv:2406.088 08447, 2024.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficiet transfer learning for nlp. In International conference on machine learning, pp. 2790-2799. PMLR, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106, 09685, 2021.
Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Ka-Wei Lee. Lim-adapters: An adapter family for parameter-effiiint fine-tuning of large language models. arXiv preprint orXiv:2304.01983, 2023.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. Journel of Machine Learning Research, 18(18)):1300, 2018.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillamme Lample, Lucile Saulnier, et al. Mistral 7b. arxiv preprint arXiv:231000682 2023.
Rik Koncel- Kedziorsk,,,,,,iii Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. Mawps: A math word problem repository. In Proceedings of the 2016 conference of the north american chapter of the association for computational linguistics: human language technologies, pp. 1152-1157, 2016.
Bingcong Li, Liang Zhang, Aryan Mokhtari, and Niao Hc. On the crucial role of initialization for matrix factorization. arXiv preprint arXt:2240011966, 2024.
Yixiao Li, Yifan Yu, Chen Liang, Pengcheng He, Nikos Karampatziakis, Weizhu Chen, and Tho Zhao. Loftq: Lora-fine-tin fine- tuning-aware quantization for large language models. arXiv preprint arXiv:2310. 08659, 2023a.
Zhijian Li, Biao Yang, Penghang Yin, Yingyong Qi, and Jack Xin. Feature affinity assisted knowledge distillation and quantization of deep neural networks on label-free data. IEEE Access, 2023b.
Baohao Liao, Christian Herold, Shahram Khadivi, and Christof Monz. Apiq: Finetuning 2-bit quantized large language model. arxio preprint Xiw:2402.00147. 2024.
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale generation: Learning to solve and explain algebraic word problems. ar preprinl arXie: 105. 04 146, 2017.
Shih- Yang Liu, Chien-Ye Sioon-Yiio Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: Weight-docomposed low-rank adaptation. arXiv preprint arXin:2402.00955 2024.
Loshchilov. Decoupled weight decay regularization. arXie preprint 1rXX::1111.005 2017.
Fanxu Mong, Zhaohui Wang, and Muhan Zhang. Pissa: Principal singular values and singular vectors adaptation of large language models. Advonces in Neural Information Processing Systems, 37:121038- 121072, 2024.
14

---

Published in Transactions on Machine Learning Research (08/2025)
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arxiv preprinl Xrx::1609 609.07843, 2016.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit armor conduct electricity? a new dataset for open book question answering. arXiv preprint aix 1509. 809.02789, 2018.
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are nlp models really able to solve simple math word problems? arXiv preprint Xr::::2003.00 07191, 2021.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In Europeon conference on computer vision, pp. 525-542. Springer, 2016.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scalc. Communications of the ACM, 64(9):99-106, 2021.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Commonsense
reasoning about social interactions. arXiv preprint wrXiv:104.09998. 2019.
Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Ommidirectionally calibrated quantization for large language models. Xrxiv preprint Xix:2990.. 19137, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXie preprint 200000307,09288, 2023.
Haoyu Wang, Tianci Liu, Ruirui Li, Monica Cheng, Tuo Zhao, and Jing Gao. Roselora: Row and column-wise sparse low-rank adaptation of trained language model for knowledge editing and fine-tuning, arXiv preprint ar Xiv:24406. 10777, 2024a.
Naigang Wang, Jungwook Choi, Daniel Brand, Chia-Ye Yu Chen, and Kailash Gopalakrishnan. Training deep neural networks with 8-bit floating point numbers. Advances in neural information processing systems, 31, 2018.
Shaowen Wang, Linxi Yu, and Jian Li. Lora-ga: Low-rank adaptation with gradient approximation. ar preprint 055000447.00000, 2024b.
Zhengbo Wang, Jian Liang, Ran He, Zilei Wang, and Tieniu Tan. Lora-pro: Are low-rank adapters properly optimized? arXiv preprint 2rxi::2407. 18242, 2024c.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pp. -0087-38099. PMI.R, 2023.
Zhewei Yao, Xiaoxia Wu, Cheng Li, Stephen Youn, and Yuxiong He. Zeroquant-v2: Exploring post-training quantization in 1lms from comprehensive study to low rank compensation. arXiv preprint arXiv:2303.085 08302, 2023.
Penghang Yin, Shuai Zhang, Yingyong Qi, and Jack Xin. Quantization and training of low bit-width convolutional neural networks for object detection. arXiv prepriul 1xX::1612206052, 2016.
Penghang Yin, Shuai Zhang, Jiancheng Lyu, Stanley Osher, Yingyong Qi, and Jack Xin. Binaryrelax: A relaxation approach for training deep neural networks with quantized weights. STAM Journal on Imaging Sciences, 11 11(4):2055-2233, 2018.
Penghang Yin, Jiancheng Lyu, Shuai Zhang, Stanley Osher, Yingyong Qi, and Jack Xin. Understanding ttraught estimator in training activation quantized neural nets. In International Conference on Learning Representations, 2019a.
15

---

Published in Transactions on Machine Learning Research (08/2025)
Penghang Yin, Shuai Zhang, Jiancheng Lyu, Stanley Osher, Yingyong Qi, and Jack Xin. Blended coarse gradient descent for full quantization of deep neural networks. Research in the Malhemalical Sciences, 6: 1 1-23, 2019b.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint arXi::1905007800, 2019.
Aozhong Zhang, Naigang Wang, Yanxia Deng, Xin Li, Zi Yang, and Penghang Yin. Magr: Weight magnitude reduction for enhancing post-training quantization. In Advances in neural information processing systems, 2024a.
Aozhong Zhang, Zi Yang, Naigang Wang, Yingyong Qi, Jack Xin, Xin Li, and Penghang Yin. Comq: A quantization. arXiv prepriat arXiv:2403.07134 2024b.
backpropagation-fiee algorithm for
Jinjie Zhang, Yixuan Zhou, and Rayan Saab. Post-training quantization for neural net with provable guarantees. SIAM Journal on Mathematics of Data Science, 5(2):373 3999, 2023.
Appendix
A Experimental Details
A.1
Language modeling
To study the capability of CLoQ, we fine-tune quantized models on the WikiText-2 training and measure perplexity the validation set. The dyper-parametens used for fine-tuning provided in Table II and Table
12
We evaluate the models on the validation set at each epoch and report the lowest achieved perplexity.
A.2
Arithmetic reasoning
Single task (GSM8K)
To assess CLoQ's arithmetic reasoning capability, we fine-tune quantized models using the GSM8K training set and evaluate their accuracy on the test set. The hyperparameters used for fine-tuning are detailed in Table 11 and Table 12 Model performance evaluated at each epoch on the test set, and we report the highest recorded accuracy.
Multiple task
Following the framework proposed by Hu et al. (2023 adopt more integrated approach by training single model across multiple lasks. Specifically, we fine-tune Llama2-7B and Llama2-13B on Math10K, dataset that aggregates training samples from GSMSK, MAWPS, MAWPS-single, and AQuA. After finetuning, the models are tested on the evaluation sets of AQuA, GSMSK, MAWPS, and SVAMP. The hyper-parametors used for fine-tuning are detailed in Table I1 and Table 12 Moreover, instead of conducting evaluations at every epoch, we assess model performance only after the final epoch.
A.3 Commonsense reasoning
To evaluate the commonsense reasoning capabilities CLoQ, we consider eight key benchmark tasks: BoolQ, PIQA, SIQA, HellaSwag, WinoGrande, ARC-e, ARC-c, and OBQA. We adopt the framework proposed by Hu al. (2023) and fine-tune a single model across all these tasks instead of training separate models. We fine-ture Llama2-7B and Llama2-13B the merged training set and measure accuracy the corresponding test sets. The parpmtepraeeters used for fine-tuning are detailed in Table II and Table I2 For evaluation, we forgo per-epoch assessments and instead report the final model's performance after the last epoch.
16

---

Published in Transactions on Machine Learning Research (08/2025)
Table 11: plyper-parameter for the finetuning of Llama2.
Hyper-parameter
WikiText-2
GSMSK
Arithmetic reasoning
Commonsense reasoning
Optimizer
AdamW
AdamW
Weight decey
0.1
LR scheduler
cosine
linear
Warmup ratio
3%
10%
Epochs
3
6
Batch size
64
32
Max sequence length
1024
512
512
Table 12: Best learning rate for Llama2-7B and Llama2-13B on the WikiText-2, GSMSK, and multiple Arithmetic Reasoning tasks.
Llama2-7B
Llama2- 13B
Group size
Task
Bits
Bits
2 Bita
Bits
8 Bits
Bits
WikiText-2
GSM8K
3e-4
3o-4
3e-4
46-4
4e-4
36-4
Arithmetic reasoning
5e-1
9e-1
1e-1
26-1
1e-1
5e-1
Commonsense reasoning
8e45
Ic-4
4e-5
5e-5
76-5
5e-5
WikiText-2 GSM8K
5e-1
2e-1
7
5e-4
5e-4
4e-4
128
Arithmetic reasoming
6c-4
3c-4
Wix GSMSK
7
4e-4
4e-4
2e.4
2e-4
5e-4
4n-4
4n-4
4c-4
4n-4
5c-4
5e-4
per-channel
Arithmetic reasoning
6e-4
8e-4
3e-4
2e-4
4e-4
5e-4
17


Reviews:
[{"review_id": "2XmAZtA8qT", "reviewer": "Reviewer_xv2J", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength:**\nThis paper enhances existing LoRA initialization methods for quantized LLMs by incorporating features into the reconstruction error computation. This natural extension provides a clear and foreseeable path for further enhancing LoRA initialization quality in quantized LLMs, like LoftQ. In addition, the proposed method maintains similar computational costs compared to previous methods, and the problem can still be solved analytically. \n\n**Weakness:**\nIt is reasonable that adding feature maps to the reconstruction problem improves the performance. In the paper, different tasks use different datasets for fine-tuning, and this setting could be the most beneficial one since the samples are concentrated in a single direction. Under this setting, it could be easier to find a good LoRA initialization fitting the target task. The author could also consider a more challenging setting, where diverse datasets are mixed together for fine-tuning. Such a result would strengthen the paper and better understand the proposed method.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors introduced calibrated LoRA initialization for quantized LLMs to minimize the layer-wise difference between the original LLM and its quantized version with LoRA components during initialization", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**1.** Fine-tuning results with mixing more diverse datasets. For example, mixing WikiText-2, Math10K, and commonsense reasoning. \n\n**2.** Show the performance of the proposed method on Larger models like 30B or 70B models. Most people use quantized LoRA to tune very large models.\n\n**3.** Ablation study with different sequence lengths.\n\n**4.** The presentation of Tables 1 and 2 could be improved by separating the two tasks since these two tasks are fine-tuned on different datasets, and they'd better be separated.\n\n**1.** is critical to my recommendation. **2.** is better to have. **3.** and **4.** are less important but straightforward to address.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rXnS5PAy8H", "reviewer": "Reviewer_koA6", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n- The main idea of this paper is to initialize the LoRA parameters by minimizing the errors introduced during quantization. The method relies only on SVD and does not require backpropagation to initialize the parameters. The paper is also very clearly written. \n- Extensive experiments using Llama2 and LLama3 show improved finetuning performance on many benchmark tasks on arithmetic and commonsense reasoning, compared to competing methods like QLoRA, LoftQ, and ApiQ. \n- The authors also provide good ablation studies on size of calibration sets required for the initialization using SVD, and LoRA initialization.  \n\nWeaknesses: \n- The proposed algorithm works best for very low-bit quantization like 2-bit. For many tasks this quantization is too aggressive and there are significant drops in performance for all finetuning methods. This could limit the number of practical use cases for the algorithm. \n- For Table 3, why is the 16-bit LoRA performance lower than 4-bit and 3-bit finetuning? Is it related to overfitting?  \n- Although the authors explain the differences of their method compared to ApiQ, which is the improved efficiency without the use of backpropagation. However this does not explain the difference between the two methods in terms of finetuning performance. We see the proposed CloQ method is better on the arithmetic reasoning tasks in Table 3, but ApiQ is better (especially for 3 and 4 bit quantizations) for commonsense reasoning tasks in Table 4. Do the authors have any intuitions on the differences in performance for these two initialization methods?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a new algorithm for initializing LoRA parameters for quantized LLM models. Their main idea is to minimize the quantization errors with the LoRA parameters using a small calibration set. Their method is efficient to implement and only requires SVD and no backpropagation. Experiments show that their method provides better finetuning results with LoRA when compared against several state-of-art LoRA parameter initialization methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- The current paper is already quite polished. I would suggest the authors address the questions I raised in the weaknesses section above.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9HtuNeQ381", "reviewer": "Reviewer_JtkH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is clearly written. The claims are supported by evidence. The results look good. The topic is interesting and important for the community. The proposed algorithm might look like a straightforward and easy extension of existing methods, but I think this should not affect the overall contributions and should not be listed as weakness. My actual concerns are\n\n1. Some parts are not written very rigorously. See requested changes below for more details.\n\n2. I think all experimental results are just one single run? Correct me if I am wrong. It is not clear whether the improvement just comes from randomness. It is better to average over several random seeds and also report standard deviation, at least for some examples that are not too heavy to run.\n\nOther questions:\n\n1. In Figure 1, some quantized methods achieve even better performance than the baseline LoRA. What is the reason? Is it just some random observation?\n\n2. In Table 6, although each one is a valid solution of equation (4), the performance is significantly different (acc 1.6 v.s. 33.7). What could be the cause of this drastically different behavior?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper focuses on parameter efficient fine-tuning of quantized LLMs using LoRA. The specific problem considered is how LoRA should be initialized in this setting. The proposed solution, termed CLoQ, contains two stage. In the first stage, a small calibration dataset is used for pretrained model quantization using an existing gradient-free method for solving $\\min_Q\\Vert X(Q-W)\\Vert_F^2$, where $Q$ is the resulting quantized weight, $W$ is the pretrained weight, and $X$ is the activation when feeding the calibration data. In the second stage, LoRA initialization is obtained by solving $\\min_{A,B}\\Vert X(AB^\\top + Q - W)\\Vert_F^2$ using two SVD. Experiments show that CLoQ achieves better performance on several reasoning tasks, especially in the low-bit case.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would support the acceptance of the paper if the following adjustments are made.\n\nOn the writing side,\n\n1. In the second paragraph of contribution, the paper says \"Our CLoQ method requires no back-propagation, making it a highly efficient approach for fine-tuning quantized models\". This could be misleading as CLoQ is only an initialization scheme. Back-propagation is still required for fine-tuning. Changing it to something like \"making it a highly efficient approach for LoRA initialization\" makes more sense.\n\n2. It is better to introduce that W is the pretrained weights already in equation (1).\n\n3. Although CLoQ requires just 2 SVD while LoftQ requires 5, CLoQ also need to compute the costly matrix inversion or pseudo-inversion. This is not discussed at all in the paper. The experiments later show that CLoQ is a little slower than LoftQ, which I guess comes from the matrix inversion. When comparing with LoftQ at the beginning of page 6, it is not fair to just say the advantage of CLoQ. Actually, LoftQ is faster.\n\n4. The reference for Mistral 7b repeats twice (see page 13).\n\nOn the experimental side, it is better to run using different random seeds and report mean, std. This should be done at least for just 1 task, say Table 4, to show that the results are consistent. If the authors do not have time for these, I suggest report the current used random seed with details on how the results can be reproduced.\n\nThere are some missing related works studying initialization of LoRA, see [1,2,3]. Although these are not originally designed for quantized fine-tuning, it is also good to include them in the related works. Specifically, [2] also considers using SVD for initialization.\n\nReference:\n\n[1] OLoRA: Orthonormal low-rank adaptation of large language models. 2024.\n\n[2] PiSSA: Principal singular values and singular vectors adaptation of large language models. NeurIPS, 2024.\n\n[3] On the crucial role of initialization for matrix factorization. ICLR, 2025.", "broader_impact_concerns": "N/A. There are no immediate ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)