LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits
Authors: Zikai Zhou
Qizheng Zhang
Hermann Kumbong
Kunle Olukotun
Venue: ICML 2025 poster (2025)
Keywords: Quantization
LoRA
PEFT
Primary area: general_machine_learning->hardware_and_software
TL;DR: We introduce LowRA to enable LoRA fine-tuning below 2 bits per parameter with minimal performance loss, cutting memory use by up to 50%.

Abstract: Fine-tuning large language models (LLMs) is increasingly costly as models scale to hundreds of billions of parameters, and even parameter-efficient fine-tuning (PEFT) methods like LoRA remain resource-intensive.
We introduce LowRA, the first framework to enable LoRA fine-tuning below 2 bits per parameter with minimal performance loss.
LowRA optimizes fine-grained quantization—mapping, threshold selection, and precision assignment—while leveraging efficient CUDA kernels for scalable deployment.
Extensive evaluations across 4 LLMs and 4 datasets show that LowRA achieves a superior performance–precision trade-off above 2 bits and remains accurate down to 1.15 bits, reducing memory usage by up to 50\%. 
Our results highlight the potential of ultra-low-bit LoRA fine-tuning for resource-constrained environments.

LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under Bits
Zikai Zhou'
Qizheng Zhang'
Hermann Kumbong Kunle Olukotun
Abstract
GPU memory limits. Quantized LoRA approaches (e.g. QLoRA (Dettmers et al., 2024), LoftQ (Li et al., 2023)) alleviate this by quantizing base model weights without accuracy loss, enabling fine-tuning on standard GPUs and even mobile devices.
Fine-tuning large language models (LLMs) is in- creasingly costly as models scale to hundreds of billions of parameters, and even parameter- efficient fine-tuning (PEFT) methods like LoRA remain resource-intensive. We introduce LowRA, the first framework enable LoRA fine-tuning below 2 bits per parameter with minimal per- formance loss. LowRA optimizes fine-grained quantization- mapiin, threshold selection, and precision assignment-while leveraging efficient CUDA kernels for scalable deployment. Exten- sive evaluations across 4 LLMs and 4 datasets show that LowRA achieves a superior perfor- mance-precision trade-off above 2 bits and re- mains accurate down to 1.15 bits, reducing mem- ory usage by up to 50%. Our results highlight the potential of ultra-low-bil LoRA fine-tuning for resource-constraing environments.
Despite their success, existing quantized LoRA methods are confined to 2-4 bits per parameter (Wang al., 2024; Meng et al., 2024; Li et al., 2023), often failing ultra- low-bit settings (below 2 bits). Enabling such fine-tuning is critical for
environments, such as embedded systems (Shen et al.. 2023; Chai et al., 2025; Zhang et al., 2024) and mobile devices (Wang et al., 2025; Tan et al., 2024). However, current methods face three fundamental limitations:
L1: Focus exclusively on coarse- coarse-grained quantization of base model weights.
L2: Rely on quantization formats assuming fixed data distribution across the model weights.
L3: Depend on simulated quantization with no system- level support for efficient low-bit execution.
1. Introduction
unlock the full potential of quantized LoRA fine-tuning below 2 bits per parameter, we introduce LowRA, an accu- rate and efficient framework featuring: (1) a novel mapping and threshold search mechanism, (2) a fine- fine-grained preci- sion assignment strategy, and (3) CUDA- quantization primitives for efficient low-bit execution.
Fine-tuning large language models (LLMs) enhances task- specific performance (Wei et al., 2021; Wang et al., 2022: Ziegler et al., 2019) and mitigates undesired behaviors such as hallucinations (Hu et al., 2024; Liu et al., 2023) and harm- ful responses (Bai et al., 2022; Askell et al., 2021). However, model sizes grow-ew e-g. ILLMA 3.1 (405e)(Dubey et al., 2024) and DeepSeek-V V3 (67 et al., 2024a) fine- tuning costs escalate significantly (Hu et al., 2021).
Addressing L1 and L2 is particularly challenging, since LoRA base weights must remain compatible with multiple adapter sets in practical deployment scenarios (Sheng et al., 2024; Ostapenko et al., 2024; Chen et al., 2024). This neces- sitates task-agnostic, yet highly adaptive, quantization approach. Additionally, fine ine-grained precision assignment requires scalable, low-complexity methods to handle large parameter spaces. LowR A tackles these with hierarchical Integer Linear Programming (ILP)-based precision assigner and weighted Lloyd-Max quantization formulation for adaptive quantization mapping and thresholding.
caumette-efffcienn fine-tuning (PEFT) techniques (Zaken et al., 2021; Hu et al., 2021; Mao et al., 2021; Liu et al., 2022a) address this by freezing core model weights and in- troducing small trainable modules. LoRA (Hu al., 2021), widely adopted PEFT method, inserts low-rank adapters to reduce computation and memory overhead. Yet, even with LoRA, fine-tuning large models can exceed single-
Department of Computer Science, Stanford Univer- sity, Stanford, USA. Correspondence to: Zikai Zhou zikai @ stanford.edu>.
We extensively evaluate LowRA across 4 LLMs and 4 tasks, benchmarking against state-of-thh-art baselines. Re- sults show that LowRA: (1) achieves superior performance- precision trade-offs above 2 bits while being the first method to enable accurate LoRA fine-tuning below bits, (2)
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits
reduces memory footprint significantly during both fine- tuning and inference, and (3) introduces minimal overhead despite its additional components.
distributions., while LoftQ (Li al., 2023), PISSA (Meng et al., 2024), and ApiQ (Liao et al., 2024) jointly optimize quantized base weights and adapter initializations under unified objective. These advances unlock fine-tuning and deployment of L.MSs on low-resource platforms like em- bedded devices (Shen et al., 2023; Chai et al., 2025) and mobile phones (Wang et al., 2025; Tan et al., 2024).
In summary, we make the following contributions:
Identifying Limitations in Quantized LoRA: We pin- point three key bottlenecks in existing methods and pro- pose solutions leveraging fine-grained precision assign- ment and adaptive quantization mapping/hresholding.
2.3. Limitations of Existing Quantized LoRA Methods
Design and Implementation of LowRA: We develop an end-to-end framework, LowRA, incorporating map- ping/threshold learner, a precision assigner, and efficient quantization primitives for scalable fine-tuning,
Despite these advances, existing quantized LoRA ap- proaches face fundamental challenges that limit their ef- ficiency, particularly in ultra-low-bit regimes.
Improved Performaane-Prrecision Trade-Off: Our method outperforms baselines in performance-precte preccssinn trade-off, enabling an average 0.86-bit reduction per pa- rameter with negligible performance loss.
L1: Coarse-Grained Precision Assignment. Most meth- ods apply a uniform quantization precision across entire weight matrices or layers. For example, QLoRA uses uni- form 4-bit precisions for all base weights, while LoftQ adopts a layerwise mixed-precision scheme (e.g., higher precision for earlier layers, lower for later layers). How- ever, our findings ($6) suggest that unlocking ultra-low- bit fine-tuning requires a finer-grained precision assign- ment potentially at the sub-layer or even sub-matrix level.
Memory Efficiency Gains: LowRA reduces fine-tuning memory consumption by 30-50% and enables fine-tuning and deployment at as low as 1.15 bits per parameter.
Open-Source Release: We will open-source LowRA upon publication to foster further research in ultra-low-bit
LoRA fine-tuning.
This paper is organized follows: Section 2 covers LoRA fine-tuning and introduces three key limitations of existing quantized LoRA methods. Section 3 introduces LowRA's 's workflow, with Sections 4, 5, and 6 detailing its design choices, mapping/threshold search, and precision assign- ment. Finally, Section 7 presents our evaluation results and insights. In the Appendix, we provide in-depth details on kernel designs, ablation studies, memory usage, motivating sources, related algorithms, and supporting experiments.
L2: Discrepancy in Data Distribution. Quantized LoRA methods often assume a global data distribution when se- lecting quantization format. For instance, QLoRA's Nor- malFloat relies on a roughly normal distribution. However, Figure 2 shows that weight distributions vary significantly across layers and even within channels. Such discrepan- cies suggest that group-wiss or channel-wise quantization is necessary for improved accuracy.
L3: Lack of High- Performance Quantization Primitives. Most quantized LoRA implementations rely on simulated quantization (Li et al., 2023; Qin et al., 2024; Bai et al., 2020), as hardware support for sub-4-bit or mixed-precision operations remains limited. For example, LoftQ requires eight A100 GPUs even for smaller LLMs, making im- practical for real-world deployment. Moreover, no existing system provides optimized low-bit CUDA kernels tailored to LoRA, exacerbating inefffciencies in fine-tuning and in- ference (see Appendix J for related GitHub discussions).
2. Background and Motivation
2.1. Low-Rank Adaptation (LoRA) of LLMs
Fine-tuning large language models (LLMs) enables LLM adaptation to specific tasks or domains (Wei et al., 2021; Wang et al., 2022; Ziegler et al., 2019), but updating all parameters becomes prohibitively expensive as model sizes grow. Low-Ran Rank Adaptation (LoRA) (Hu et al., 2021) ad- dresses this by freezing base model weights and introducing trainable low-rank adapter matrices, significantly reducing memory and compute overhead. This method has become a cornerstone of ceaameter-efficieen fine-tuning.
3. The LowRA Framework
In this section, we provide an overview of the LowRA end- to-end workflow, illustrated in Figure 1. The process begins with the pretrained model weights (TI). We feed each layer of these weights into a dedicated mapping and thresholds learner (PI), which produces optimized per-output-channel mappings and thresholds, denoted (T2). These mappings and thresholds, along with the pretrained weights, are then processed by a two-step II. quantizer (P2) to determine the
2.2. Quantization for LoRA Fine-Tuning
Quantized LoRA fine-tuning further cuts memory usage by quantizing the base model weights without hurting per- formance. QLoRA (Dettmers et al., 2024) introduces a NormalFloat format designed to better capture parameter
2

---

LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits
User Defined Compression Ratão
Intelligently Initialized Low-rank (TS)
Final Low-rank Tensors (T6)
Pretrained Weight (T1)
Precision Assigner (P2)
Precs
Initializer (P4)
Output- channel-
wise
Mapping and Threshold Learner (PT)
Quantize
Output- channet- wise Doquantize
(T2)
Kernel (P3)
rer-output-chaanee Sixeed-reecisoo Downquant Residual (T4)
Programs
Tensors
Fine-grained Mappings and Thresholds
Kernel (P5.))
FPS2/7216/8P0
Further
Finetuning
(P5)
Figure End-to-end workflow LowRA. LowRA learns the mapping/threshold per output channel, assigns the precision to each output channel with precision assigner, and supports fine-tuning workflow with Quantize and Dequantize Kernels. See section for details. Limilation LI corresponds components and T3; L2 corresponds PI, T2, and corresponds to P3 and P5.1.
proj
pro]
pra)
4.1. Insights behind LowRA Design Choices
Channel #1214
Charnal #2082
Channel #723
Hormel 107
Per-Output-Chame Quantization In LLMs, linear lay- ers often exhibit substantially more variation across out- put channels than across input channels. In error-sensitive layers, weight magnitudes vary markedly from one output channel to the next, producing banded patterns along the input-channel dimension that reset at each output-channel boundary (see Appendix H). As result, grouping parame- ters by output channel and assigning a unique precision each group- -i.., per-output-chanie quantization more effectively captures their diverse distributions.
735
330
332
252
-2.
0.0
+10
-15
0.3
Figure Distributions normalized parameters in different out- put channels sampled from the first layer of LLaMA2-7b.
optimal precision assignments (T3) for each output channel.
Next, the output-channel quantize kernel (P3), which supports custom quantization thresholds, uses the derived thresholds (T2) and the assigned precision levels (T3) quantize the weights. We calculate the quantization errors arising from this step and apply low-rank tensor initializa- tion (P4). Techniques for intelligent low-rank initialization include LoftQ (Li et al., 2023) and PiSSa (Meng et al., 2024) (Appendix D) which generate low-rank tensors (T5) designed to absorb or reduce quantization errors as possi- ble during initialization. In our implementation, we opt for LoftQ (Li al., 2023) experiments show that they give better performance in the low-bit range.
Groupwise Normalization Each output channel may still exhibit significant internal variability even with per-output- channel quantization. To address this, groupwise normal- ization often used allow each group of elements share separate scale. We follow QLoRA's design of using 64- element normalization scaled by the absmmxx (i.c., maximum absolute value) in each group (Dettmers et al., 2024).
Data-Free Post- cest-Trainig Quantization Unlike approaches with quantization-awar training (QAT) (Esser et al., 2019; Yang al., 2021; 1 Jeon et al., 2024; Savarese al., 2022), our approach adds no overhead fine-tuning. By automati- cally searching for quantization mappings and thresholds, it frees users from manual tuning (Savarese et al., 2022; Zhou et al., 2023), saving both development and computation resources. Moreover, contrary to some methods that vary compression ratios over time (Savarese et al., 2022; Yang et al., 2021), LowRA maintains a consistent compression ra- tio, ensuring persistent memory savings during fine-tuning.
The mixed-precision weights (T4) and initialized low-rank tensors (T5) feed the fine-tuning module (P5), which uses an output-channel dequantize kernel (P5.1) to restore base weights. As in LoRA/QLORA, base weights (T4) stay frozen while only the low-rank tensors (T5) are trained. The mod- ule outputs updated low-rank tensors (T6) alongside the quantized base weights and their state (T4).
Per-Oueput-Channel Thresholds and Mappings Figure 3 visualizes the roles of thresholds and mappings in the process of quantization. Thresholds refer to the boundary points ("bin edges") that partition the continuous domain of normalized parameters into discrete intervals and thus
4. Discussion about Design Choices
In this section, we discuss various design choices the LowRA framework, as well as system and hardware support.
3

---

LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits
specific bitstring encodings. Mappings, on the other hand, specify the representative values assigned each encoded bitstring and thus the intervals. As discussed in $2.3, fine- grained designs of quantization mappings and thresholds could lead to significantly more accurate approximation and reconstruction of parameters. LowRA allows each output channel to adopt a different combination of mappings and thresholds for more precise fine-grained quantization.
convergence than LoftQ, our experiments consistently show LoftQ outperforming PiSSA. As illustrated by the sample data points in Table 1, PiSSA fails to achieve reasonable task performance at lower bit ranges. This aligns with our intuition that performing quantization rather than SVD first allows the low-rank tensors to better absorb quantization errors. Our findings also corroborate points raised in the LoftQ appendix. Since our main objective is to enable lower- precision tne-tuning and deployment, we opt use LoftQ as our low-rank initializer. Following the recommendation from LoftQ, we use five alternating steps for initialization.
-1.00 -0.76
0.63
+1.00
Original
Sctup
LLaMA-7b
LLaMA-
Quantized
Method
Dataset
2-bit
4-bit
2-bit
4-bit
-0.92
-0.31
0.46
0.78
PiSSA
1919.63
5.53
1825.68
5.05
Thresholds
Mappings
WikiText-2 (1)
LoftQ
8.63
5.26
7.27
4.79
Figure 3. Roles of mappings and thresholds in quantization.
Table I. Perplexities of PiSSA and LoftQ as initialization methods on WikiText-2. Quantization is performed at 2 bits or 4 bits per parameter. Lower values indicate better performance (L).
Data-Free One-Shot Post-Training Quantization Most quantization-awaa training (QAT) techniques achieve higher task performance by incurring additional training overhead and learning task-specific quantization parame- ters (Esser et al., 2019; Yang et al., 2021; Jeon et al., 2024; Savarese et al., 2022). Similarly, many post-training quan- tization methods require a calibration set for quantization scheme learning (Liao et al., 2024; Hubara et al., 2021). In contrast, LowRA uses data-free mhe- post-training quantization, enabling reusable quantization schemes and quantized parameters, minimal hyperparameter tuning, and negligible fine-tuning overhead. This design is particularly suited to LoRA fine-tuning because: (1) Task desendeenndnnt learning is confined to the adapters, (2) LoRA base weights are often shared across multiple adapters, and (3) LoRA primarily targets resource-constrained fine-tuning scenarios.
Adapting LowRA to Production Use Cases Many produc- tion cases, e.g., batched inference in data centers, require fixed quantization mappings (Li et al., 2024; Zhao et al., 2024). LowRA can be adapted to such scenarios by keeping only the thresholds learnable, which is shown to be useful in enhancing model performance (Liu et al., 2022b). To maxi- mize performance with rask-agnostic reusable base weight, LowRA can be extended use the same set of thresholds for multiple adapters but learn mappings for each down- stream task. In other words, each adapter can be connected to the base weight together with dedicated base weight decoding mapping for that downstream task. Nevertheless, this would demand higher development and computation costs. In our implementation and experiments, we adopt the same set of thresholds and mappings for ease of evaluation.
User-Defined Compression Ratios Quantized LoRA meth- ods see heavy use in tight resource settings e.g., limited- memory GPUs or on-device scenarios- where specifying a precise compression ratio is pivotal. By tailoring each pa- rameter's 's bit precision, thresholds, and mappings, LowRA directly aligns compression with real-world resource bud- gets, ensuring feasibility and efficiency even under strict constraints. Furthermore, because LowRA fixes the ratio in single pass, it obviates the extensive hyper- parameter tuning needed by alternative methods to find acceptable compression accuuuccy trade-offs (Savarese et al., 2022).
4.2. System and Hardware Support
Building on the limitations noted in $2.3, we implement practical LUDA-based primitives that support both low- bit and mixed-ppreision LoRA fine-tuning with maximum flexibility (details in Appendix A). Notably, the added kernel generalization incurs only negligible overhead in end-to- end inference, as quantizaiionddequntiaaii constitutes a minimal portion of the total compute cost.
Using LoftQ as Low-Rank Intializer Researchers have found that the initialization of low-rank tensors are crucial to the effectiveness of LoRA fine-tuning, especially when it comes to ultra-low-bit quantized base weight (Li et al., 2023; Meng et al., 2024; Wang et al., 2024). LoftQ (Li et al., 2023) and PiSSA (Meng et al., 2024) are two notable initialization techniques for quantized LoRA (see Appendix D for a detailed introduction). While PiSSA purports faster
5. Mapping and Threshold Learner
In this section, we introduce the mapping and threshold learner in LowRA. Because we want the final base weights to remain task-agnostic and thus reusable across multiple adapters, we adopt a simple approach that minimizes the mean squared error in each output channel. As discussed in 1, one could learn separate decoding mappings for
4

---

LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits
each downstream task (or adapter set), but a higher fine- tuning cost. We therefore propose efficient design for the mapping/threshold learner that avoids this expense.
formulation can serve an effective proxy retain more information for the harder-to-quantize channels in weights.
One can observe that finding the optimal mixed-precissee scheme (w.r.t SSE) can be formulated as an ILP. How- ever, due to the large number of output channels in L.LMs, a direct solver-based approach becomes computationally prohibitive. For instance, solving more than five layers of LLaMA-2-7B fails to finish within ten hours. To address this limitation, we propose two-level ILP workflow (Fig- ure 4, Algorithm 1) that retains the benefits of ILP-based methods while ensuring reasonable complexity.
Weighted Lloyd-Max Algorithm. We cast the problem of searching for the optimal quantization mappings and thresholds for each output channel to minimize MSE as Weighted Lloyd-Max Problem. A detailed description of this algorithm can be found in Appendix G.
Weighted Lloyd's for LoRA Quantization. As discussed in 4.1, we perform groupwise normalization to give more scale to quantization within each output channel. To recap, with groupwise normalization (Section 4. 1), each block of weights is scaled by the block-wise maximum absolute value (absmaz). Specifically, if we denote the set of original weights in a block by } and its maximum absolute value by absmax, then we treat absmar as a per-block weight in the Weighted Lloyd-Max algorithm. By assigning them proportionally larger weights, the algorithm "pays more attention" to those blocks and adjusts their bin thresholds and centroids accordingly. Consequently, blocks whose values have smaller magnitudes (and thus smaller absmax) are penalized less, striking balance across all blocks to minimize the overall quantization error in QLoRA.
Notation. Let N be the total number of output channels. Denote by {w(t),.... ,...., w(k) } the distinct parameter sizes (i.e., number of parameters) appearing across channels, and let T C (1,...,NN .N be the set of channel indices whose size is (k()) We further define Wx = Eier, w(k) and Wsum = ΣK WR. the per-group and total parameter counts that will be used when allocating the bit-budget.
Algorithm 1 Channelwise Precision Assignment
1: Input: N. distinct (1) ,..., (K) partition [7x), MSE(i.)) total budget Buaal
2: Step 1. Compute WR -EiEr (h)) then Wsam
Σ We
3: Step 2. BB + - x We for ks = L....,K
Wssn
4: for k 1 K do
In our application of the Weighted Lloyd's algorithm to LoRA base weight quantization, we initialize the thresholds as those used by NormalFloats (Dettmers et al., 2024) for 2- bit and bit precisions and use 0.0 as the initial threshold for --bi quantization" Then, at each iteration, we recompute the quantization mappings the weighted centroids of the assigned data and recompute the thresholds as the midpoints between consecutive mapping (centroid) values?. We output the last-computed quantization mappings and thresholds when max iteration is reached or the MSE stops going down.
5: Step Cluster channels in Ix (e.g. K-Means on MSE features) into K₂ clusters
6: Step 4. Cluster-Level ILP: decide how many channels in each cluster get each bitwidth, subject to Ba
7: Step 5. Intra-
Intra-Cluster Cluster ILP: within each cluster, assign spe- cific channels to bitwidths
8: end for
9: Step 6. Combine final bitwidths into bj b...... by
10: Output: (bi,...,bw) bw), total SSE, actual bits used
6.1. Preprocessing for the Pipeline (Step 1-3)
In our current implementation, we take the average of all thresholds preserve distribution and prevent instability in the interaction with the Low-Rank Initializer.
To preprocess channels for the hierarchical ILP pipeline, we first compute each channel's MSE under 1-, 2-, and b-t quantization. Next, we split channels by parameter count, which in LLMs typically yields two distinct sizes (e.g., 4096 and 1008 for L. Within each group, we then apply three-dimensional K-Means clustering (based on the three computed MSE values), forming 128 clusters per group in our implementation.
6. Mixed-Precision Quantization: Channelwise Precision Assignment
In this section, we present how mixed-precision quantization assignment is conducted in LowRA. In light of the afore- mentioned task-agnostic requirement, a simple yet effective objective for defining this problem is the minimization of the overall Summed Square Error (SSE) considering the reg- ular structure of transformer- based architectures (Waswani et al., 2017: Dubey et al., 2024; Touvron et al., 2023a). Such
6.2. Cluster-Level ILP (Step 4)
Formed clusters first go through the following cluster-level ILP be assigned budgets of 1-bit, 2-bit, and 4-bit channels.
Consider C' cluusters, each with S. channels (c= 1,... C). Let P = {1, 4] be the available bit- bit-precisions". For each
Seeparttly defined as NormalFloats lack 1-bit representation "See the initial values in Appendix F II our implementation, we set number of iterations to 2
*For ILLMA, restricting 2 and 4 bits outperformed including bit for hpp 22.0, so we adopt this configuration.
5

---

LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits
tization error by choosing, for each cluster c, how many of its channels yep are assigned each precision level p. The constraints ensure that every channel of cluster is allocated exactly once, the total bits used do not exceed the overall budget B, and that the decision variables remain non- negative integers and do not exceed the number of channels in their respective clusters.
Channelwise Quant Costs (MSEs)
E
Channelwise Param Counts
Budget
Split Param Counds
Channelwise Quant Conta (A)
Channelwise Param (A)
Channehwise Quant
Channelwise Puram Counts (X)
K-Means Clustering
K-Meams
6.3. Intra-Cluster ILP (Step 5)
Clustering
Once the cluster-level ILP decides how many channels {No.p} in each cluster c should be assigned to each bit preci- sion p, a second ILP distributes these assignments each channel within each cluster.
Clusters Group
Clusters
Group B
Chuster-level
ILP
Let Sp denote the total number of channels in cluster c. For channel i E {1....,
Per-Cluster
Precision
Sef [1. in cluster c. we define the pre-
Assigaments
computed mean-squared error at precision p as MSE(i, p) = (precomputed quantization error channel i at precision p). We define binary decision variables Ti,p = if channel i assigned bit precision p: otherwise, Ti.p
Intra-cluster
ILP
Channelwise Precision Assignment
S.
Figure 4. Two-step ILP--ased Workflow for Channelwise Preci- sion Assignment, with (1) channelwise clustering, (2) cluster-level ILP assignement, and (3) intra-cluster assignment, scquentially.
Minimize
MSR(i. Ti.p
PEP
subject to
Σ
Ti Etip = (l..,....,
pEP
cluster c and precision p E P, define:
S. Σ Ti.p
coste.:: mean quantization error (e.g., mean-mqaaaaatea reean-squrre er- ror) per channel in cluster e if all channels in that clus- ter are assigned to precision p. scaled by the number of weight parameters per channel that cluster
Ve.
VpeP,
Zip E {0,1} Vie (1....,,sck PEP.
This formulation constitutes the intra-cluster ILP. The objec- tive minimizes the total quantization error, where MSE(i, is the precomputed mean- seuune error for channel at bit precision p. The first constraint ensures that each channel precision. The second constraint
Ve.p E Zo decision variable representing the number of channels cluster e that will be assigned precision p.
We define global bit-budget B (i.e., total permissible bits across all clusters). Let (pp) be the bit-precision value (e.g., 3(1) = 1, (2)) = 2, ((4) = 4). To enforce the bit budget, we multiply 3(p) by the channel parameter count we for cluster c, and then by the number of channels Ye.p- As we do a bipartite splitting based on the number parameters per channel, each channel in cluster e shares the same We-
is assigned to exactly enforces that the number of channels assigned to each pre- cision matches the counts Ve.p determined by the cluster- level ILP. Finally, the binary constraint stipulates that each decision variable Ttp either 0 or
This intra-cluster ILP enforces that the required number of channels (from the cluster-level ILP) assigned each bit precision and minimizes local MSE within the cluster.
Minimize
Σ (costo.p) Dc.p
PEP
Efficiently Leveraging ILP Solvers. We collect the as- signed per-channel precisions in Step 6. By employing this two-step hicrarchical approach, we harness ILP solvers' strengths while keeping computational overhead low.
subject to
Σ vo. = Sc, c= c=.....,C
PEP
Ho. S B, Σ Σ (8(p)
PEP
7. Evaluation
Ve,p E Zzo,
0 03 Vep Sc.
We evaluate LowRA across four datasets spanning natu- ral language generation, multi-turn conversation, and long- context text summarization, demonstrating that: Better performance at the same precision: LowRA
This formulation seeks to minimize the total weighted quan-
*Contrary LoftQ's suggestion, per-layer weighting based on layer index proved suboptimal our experiments.
6

---

LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits
outperforms all baselines below 4-bit and matches their performance 4-bit ($7.3).
and low-rank adapter initialization. We match LoftQ's effec- tive batch sizes but observe discrepancies with its published results due to: (1) reproducibility constraints - the original authors did not release experiment seeds or mixed- precision quantization hyperparameters - (2) hardware differences Our experiments run on a single A100 GPU, whereas LoftQ was trained on 8 A 100 GPUs with greater data parallelism and (3) quantization implementation The original LofiQ experiments rely on simulated quantization, which intro- duces discrepancies in quantized model weights, as noted by the research community (Liao, 2023). In contrast, we em- ploy CUDA- AaA--kenel--aaae quantization and dequantization, ensuring more accurate and hardware-aligned results.
Same performance at lower precision: LowRA achieves comparable performance while reducing pre- cision by 0.86 bits per parameter on average ($7.3).
First method to fine-tune LoRA under 2 bits: LowRA enables fine-tuning down to 1.75 bits on LLaMA-2- 7B, LLaMA-2-13B, and BART-large, and 1.15 bits on LLaMA-33B ($7.4).
Substantial memory savings: LowRA reduces memory usage by 30-50% in fine-tuning and deployment, with minimal performance loss compared QLoRA ($7.5).
Minimal overhead: The additional one-time preprocess- ing costs in LowRA are negligible (Appendix E).
7.3. Analysis of Key Results
Real hardware gains: On one GPU, 1.5-bit LowRA loads about 29 % faster and, after TTFT, delivers up to 3.4x higher throughput than 4-bit QLoRA- -2.2 vs 9.4 tokens/s on LLaMA-7B and 1.4x on LLaMA- 13B (87.6).
Table 2 presents the performance comparison of LowRA against QLoRA and LofiQ.
Better performance at the same precision: LowRA outperforms QLoRA and LoftQ across all sub-4-bit pre- cision levels. In particular, at challenging 2-bit quanti- zation, LowRA achieves perplexity reduction of: 2.21 (WikiText-2)) / 1.45 (Open-Assistant) over QLoRA, and 1.76 (WikiText-2) / 1.12 (Open-Assistant) over LoftQ.
7.1. Evaluation Setup
Hardware Platform Experiments are conducted NVIDIA A A100 GPUs (80GB memory). Each LLaMA exper- iment runs on a single dedicated GPU. Each BART-large ex- periment runs two instances concurrently on a single GPU,
Same performance at lower precision: LowRA en- ables fine-tuning with 0.98 bits (QLoRA) / 0.76 bits (LoftQ) fewer per parameter (on average) without perfor- mance loss. For example, 2.5-bit LowRA on WikiTexx-2 (LLaMA-2-7B) matches b-t QLoRA; 1.9-bit LowRA on Open-Assistant (LLaMA-2-7B) matches 2.5-bit LoftQ.
Hyperparameters For a fair comparison, we use identi- cal hyperparameters across all methods, consistent with QLoRA (Dettees al., 2024) and LoftQ (Li et al., 2023). Details on selected hyperparameters are in Appendix I.
Language Models We evaluate LowRA on range of
LLMs: LLaMA-2-7B, LLaMA-2-13B (Touvron et al.,
7.4. Fine-Tuning LLMs with Ultra-Low Bits
2023b), BART-large (Lewis, 2019), and LLaMA-33B (Tou- vron et al., 2023a) (to assess ultra-low-bi scalability).
To investigate the limits of ultra-low bit LoRA fine tuning, we evaluate LowR A on two model scales: (i) L.LAMA 33B and (ii) LLAMA 65B. Table summarizes perplexities on Wikitext 2 and OpenAssistant when training with 1.25 and 1.15 effective bits per parameter.
Datasets and Evaluation Metries We use standard datasets across different NLP tasks: WikiText-2 (Mer- ity et al., 2016) (language modeling, perplexity), Open- Assistant (Köpf et al., 2024) (multi-turn conversation, per- plexity), XSUM (Narayan et al.. 2018) (summarization, ROUGE scores), and CNN/DailyMail (Hermann et al., 2015) (summarization, ROUGE scores). Each dataset is evaluated using the standard metrics used in prior work.
LowRA is the only approach that remains stable under these ultra-low precisions; QLoRA and LoftQ diverge at this regime. At 1.15 LLAMA 33B incurs merely a +0.54 absolute perplexity on WikiText 2 compared with the 1.25 bit setting, while retaining competitive quality on OpenAssistan (5.73). Strikingly, scaling the same recipe to the 65B model yields further gains-7, (.4s--7.4/4.97 perplex- ity illluttutaainng that LowRA scales gracefully with param- eter count despite the extreme quantization.
7.2. Baselines
QLoRA QLoRA originally employs a fixed 4-bit quanti- zation for pretrained LLMs and does not support fine-tuning below 4 bits. To enable sub-4-bit QLoRA experiments, we follow the adaptation introduced in LoftQ. Additionally, QLoRA directly quantizes the pretrained weights while pre- serving their original distribution, initializing the low-rank tensors with zeros and small Gaussian noise.
7.5. Memory Implications
Following the analysis methodology in QLORA (Dettmers et al., 2024), we evaluate the memory footprint of LowRA at different quantization precisions. Full visualizations are in Appendix C. Our results show that LowRA signifi- cantly reduces memory usage for both fine-tuning and in-
LoftQ LoftQ performs mixed preccison quantization (2- bit/4-bit) and jointly optimizes both quantized L.LM weights
7

---

LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits
LLaMA-2-7B
LLaMA-2-13B
BART-large
Method
Bit
WikiText-2
Open- Open-Assistant
WikiText-2
Open-Assistant
XSUM
CNN/DailyMail
ppl.1/ace.1
ppl.1
ppl.1/ace.1
ppl.
R
QLoRA
4.00
6.22 / 00.583
3.52
4.79/ 0.628
3.25
39.07 16.31 31.09
41.18 18.18/18.32// 27.58
LoftQ
4.00
5.26 / 0.613
3,48
4.79 0.628
3,23
40.34 / 17.06 31.92
41.12/ 18.29 27.54
LowRA
4.00
5.25 / 0.612
3,48
4.79 0.628
3.23
40.27 17.18 32.06
40.95 / 18.12 27.54
QLoRA
3.00
7.13 / 0.566
4.56
6.06 0.588
3.88
17.60 / 2.68 13.93
15.34 1.12 10.44
LoftQ
3.00
6.87 00.571
4.42
5.91 0.591
3.79
37.23 14.34 29.30
40.47 17.75 26.88
LowRA
3.00
5.84/ / 0.593
3.87
5.24 / 0.611
3,50
38.84 15.68 30.57
40.85 / /
QLoRA
2.50
8.05 / 0.546
5,17
6,84 0.568
4.36
15.33 / 1.97 12.55
13.68 1.04 / 99.99
LoftQ
2.50
7.72 / 0.552
4,98
6.70 0.572
4.21
34.48 12.26 / 27.05
39.81 17.19 26.57
LowRA
2.50
6,23 / 0.582
4.11
5,51 0.601
3.64
37.69 / 14.76 / 29.53
40.88 / 18.06 27.01
QLoRA
2.25
8.67 /0.534
5.59
7.31 0.588
4.64
16.37 2.22 12.84
11.90 / 1.32 10.25
LoftQ
2.25
8.22 / 0.543
5.24
6.96 0.564
4.46
32.71 10.94 25.37
39.36 / 16.87 26.29
LowRA
2.25
6.40 / 0.578
4.21
5.66 / 0.597
3.73
37.29 14.36 29.12
41.01 / 18.19 27.23
QLoRA
2.00
9.17 00.56
6.07
7.64 0.551
5.02
DNC 31.89/ 10.18 24.59
4.84 0.00 / /436
LoftQ
2.00
8.63 / 0.536
5.68
7.27 / 0.558
4.75
38.88/ / 16.49 25.85
LowRA
2.00
6.60 / 0.574
4.35
5.79 0.593
3.84
36.75 13.93 / 28.61
40.15 17.48 26.67
QLoRA
1.90
LoftQ
1.90
LowRA
1.90
7.13 / 0.562
4.94
6.16 0.583
4.22
34.05 11.74/ 26.49
39.19 16.84 26.35
QLoRA 1.80
LoftQ
1.80
LowRA
1.80
7.50/ / 0.553
5.24
6.48 / 0.575
4.59
33.29 11.19 25.85
39.20 / 16.69 26.07
QLoRA 1.75
LoftQ 1.75
LowRA
1.75
7.76 / 0.548
5,43
6.65 0.569
4.76
33.09 11.05 / 25.69
38.54 / 16.38 25.99
Table Performance comparison of different methods on LLaMA-2-7B, LLaMA-2-13B, and BART-large. "_" means this method fails support this level of precision. "DNC" means fine-tuning fails converge. LowRA (using PEFT) not only outperforms QLoRA trade-off, but also enables us to fine-tune LLMss in the sub-2-bit range. LoftQ results on
and LoftQ in terms of performance-precisten Bart-Large taken the best of two strategies: (1) layers are ordered based sheerly on layer-index and (2) encoder layers ordered before decoder layers. See Appendix K for detailed results. Also, see Appendix B for ablation analysis.
ference, making ultra-low-bit LoRA practical on resource- constrained hardware.
of LLMs significantly smaller devices. For instance, fine-tuned LLaMA-2-7B model can now be deployed a Raspberry Pi 4 Model B (4GB RAM) (Foundation, 2019), making on-device inference feasible even in extreme resource-constrann
For inference, reducing precision from 4-bit to 2-bit leads to 40% lower memory usage on LLaMA-2-13B and 30% on LLaMA-2-7B (Figures 8). Compressing LLaMA-33B to 1.15 or 1.25 bits achieves even greater savings, reducing the memory footprint by 50% (Figure 9).
settings. More strikingly, LowRA is the first method to enable LLaMAA33B fine-tuning on a sin- gle NVIDIA Tesla T4 (16GB VRAM) (Corporation, 2021), demonstrating its potential for democratizing large-scale LLM adaptation.
For fine-tuning, LowRA also achieves substantial reductions. Moving from 4-bit to 2-bit precision cuts memory consump- tion by 30% on LLaMA-2-13B and 25% on LLaMA-2-7B (Figures 7). On LLaMA-33B, reducing precision to 1.15 or 1.25 bits per parameter leads to an estimated 45% reduction in fine-tuning memory usage, making it feasible to train larger models under stricter memory constraints.
Model
Bits
Datasel
QLoRA
LeftQ
LowRA RA
WikiTex12
7.46
1.25
OpenAssistant
5.44
LLB
WikiTex12
8.00
OpenAssistant
5.73
WikiTex12
7.49
65B
1.15
OpenAssistant
4,97
These memory savings are particularly impactful given that 4-bit QLoRA models are already highly compressed. By pushing below 2-bit precision with minimal perfor- mance loss, LowRA enables fine-tuning and deployment
Table 3. Perplexity WikiText2 and OpenAssistant when fine tuning LLaMA-33B and LLaMA-65B with ultra low bits. "_" indicates that the method fails converge or exceeds memory limits at the corresponding precision.
8

---

LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits
7.6. Inference Implications
Table S. Throughput (tokens/s) for LLaMA-2-7B and LLaMA-2- 13B with on a single GPU. We experiment with prefill length of 100 decode length of 10. QLoRA 4.0 refers references results obtained with the original QloRA implementation.
In this section, we discuss the implications of LowRA on LLM inferences. The implications involve three primary aspects: loading latency, sime-to-firs-toker (TTFT), and end-to-end throughput. We discuss each aspect below.
Model
GPU
Bits
Tokens/s
1.5
32.16
1. Loading latency: the time to move model weights into GPU memory. Loading latency is crucial for latency-sensitive, bursty (i.e., not steady-state) workloads where cold-start delays directly impact user experience and SLA compliance. Results of LLaMA-2-7B
2.0
30.33
LLAMA 7B
RTX 3080
2.5
24.35
4.0
9.19
QLoRA 4.0
9.40
RTX A4000 are shown in Table 4.
1.5
16.11
2. Time-to momn-tooffrst-aee first-tottttn (TTFT): the latency from prompt submission to the first generated token, assuming the weights are already resident in memory. Sub-second TTFT is essential for interactive workloads because even brief stalls break the user's flow (Yao al., 2025). Smaller quantized models reduce VV-cache transfers slightly (Liu et al., 2024b), but the step is dominated by compute, so TTFT improvements from parameter compression are only modest.
2.0
14,46
LLAMA 13B RTX A4000
2.5
13,64
4.0
12.17
QLoRA 4.0
11.56
QLoRA. From our evaluation, LowRA adds at most 8.5 % runtime overhead and crucially, executes configurations that QLoRA cannot run due to out-of- -f-ooof-erryy limits.
Table 6. Benchmarked end-to-end training runtime (ms) of LowRA models, hardware, and precision configurations. Seq. refers to training sequence length, Batch refers to batch num- ber, and QLoRA refers the reference training lateney QLoRA (Dettmers et al., 2024). "_" means no data/not applicable. 00M "out-of-memory".
End-to-end throughput: the sustained generation rate-tokens per second or requests per second-afis after the model is resident and TTFT has passed. High throughput dominates cost and user-perceived speed in long generations, batched chat sessions, and API back- ends. It scales with effective GPU compute, memory bandwidth, and parallelization depth; quantization and smaller checkpoints help, but gains come chiefly from increased arithmetic intensity and optimized kernels. Ta- bles 5 summarize measured throughput for LLaMA-7B
Seq. Batch 1.5 bit 2.5 bit
4 bit
QLoRA
Max %
LLaMa 7B
RTX
A5000
256
1
782.7
789.4
782.7
754.3
4.65
512
1
1291.8
1295.8
1291.8
1268.5
2.15
1024
1
2398.1
2399.0
2398.1
2383.7
0.64
on an RTX 3080 and LLaMA-13B on RTX A4000.
256
2
1268.2
1274.0
1268.2
1247.2
2.15
512
2
2319.8
2322.0
2319.8
2304.8
0.75
Table Loading latency of LlaMA-2-7B on an RTX A4000. Us- ing LowRA to compress model to 1.5 bits per parameter reduces loading latency by close to 30%. LoRA tensors are kept in FP32.
LLAMA 13B
RTX
A5000
256
1
1494.7
1494.7
1501.2
1508.0
1444.2
4.42
512
1
2486.0
2486.0
00M
0OM
OOM
Framework Bits/Param
Latency (s)
256
2
2457.8
2457.8
OOM
0OM
OOM
QLoRA
4
220.73
LLAMA 7B
A100
80 GB
LowRA
4
218.17
LowRA
1.5
157.51 (-28.6%)
256
1
632.1
632.4
631.9
639.3
589.1
8.52
512
1073.7
1074.6
1072.9
1079.5
1030.8
4.72
256
2
1056.9
1056.2
1055.9
1063.5
1014.9
4.79
Across LLaMA-7B on an RTX 3080, 1.5- LowRA delivers a 3.42x throughput increase over QLoRA (32.16 16 9.40 tokens per second). On LLaMA-13B with an RTX A4000, the same 1.5-bit configuration still yields 1.39x speed-up. These results confirm that sub-2-bit quantization offers real-world throughput benefits beyond the documencddd memory savings.
Discussion.
8. Conclusion and Future Work
As LLMs grow, even parameter-effiieet schemes like LoRA strain compute and memory. LowRA overcomes these limits, enabling accurate LoRA fine-tuning <2 bits per weight through fine-grained precision assignment, adaptive quantization, and custom CUDA kernels. Across models, LowRA cuts memory by up to 50 % while matching or exceeding full full-precision accuracy down to 1.15 bits, unlock- ing fine-tuning on mobile, embedded, and other resource- constrained devices and charting a path toward ultra-low- bit L.LM training and deployment.
7.7. Training Overhead
We benchmarked LowRA at multiple bit-widths against QLoRA (4 bit) on an RTX A5000 (24 GB) and an A100 (80 GB). Runtime per iteration is shown milliseconds: the last column reports the largest LowRA overhead versus

---

LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits
Acknowledgements
Corporation, N. NVIDIA T4 Virtualization Datasheet.
www.:iiiiaaaaaiiaooooooooooooiao hotp:/iiaa..ccoocooootttttttata content/
We thank our anonymous reviewers for their invaluable feedback, which significantly enhanced the quality of this paper. We also thank Yilin (Clark) Xu, Hanchen Li, Geng- han Zhang, Yicheng Qian, Jungwoo Kim, and Yufei Jin for their insightful discussions. This research was supported in part by the National Science Foundation under Grant No. 2384 (Collaborative Research: CNS Core, Medium -A Stateful Switch Architecture for In-Network Compute) and by the Naval Surface Warfare Center under Agreement Cithery-Efffiieenn
en htt2:///obutttooneeeeesssvvaaaaa coluttiioo//essurrrs///ooooottls
Datasheet__WIDDrA____.ittt pdf, 2021. Accessed: 2025-01-29.
Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36, 2024.
No. N00164-23-9-G0335 and Scalable Al Hardware Systems through Heterogeneous Integration of Specialized Chiplets). The authors thank both sponsors for their generous support. Hermann Kumbong gratefully acknowledges support from the Stanford Knight- Hennessy Scholars Program". Zikai Zhou is supported by the Stanford School of Engineering Fellowship, generously endowed by the Enlight Foundation".
(Project
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21785 2024.
Esser, K., McKinstry, L.. Bablani, D., Appuswamy, R., and Modha, D. S. Learned step size quantization. arXiv preprint arXiv: vurXiv:1902..0 1902.08/32 2019.
Foundation, R. P. Raspberry Pi 4 Model B.
Impact Statement
https: / /www. raspberrypi.. wtt.yp..co////e.m wwt.coooo/pp...... products/////// raspberry pi- model-bl 2019. Accessed:
This paper contributes to the advancement of Machine Learning. While our work may have various societal impli- cations, none require specific emphasis. Additionally, this research does not raise any ethical concerns.
2025-01-29.
Hermann, K. M., Koeisky, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., and Blunsom, P. Teaching machines to read and comprehend. Advances in neural information processing systems, 28, 2015.
References
Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., DasSarma, N., al. A general language assistant a laboratory for alignment. arXiv preprint ar
Hu, E. J., Shen, Y., Wallis, P., Allen Z., Li, Y., Wang, S., Wang, I.., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint 116v6210..00665. 2021.
2021.
Bai, H., Zhang, W., Hou, L., Shang, L., Jin, J., Jiang, X., Liu, Q. Lyu, M., and King, I. Binarybert: Pushing the limit of bert quantization. arXiv preprint arXiv:2012.15701, 2020.
Hu, M., He, B., Wang, Y., Li, L., Ma, C., and King, I. Miti- gating large language model hallucination with faithful finetuning. arXiv preprint arXiv:2406. 1/677, 2024.
Hubara, I., Nahshan, Y., Hanani, Y., Banner, R., and Soudry, D. Accurate post training quantization with small cal- ibration sets. In International Conference Machine Learning, pp. 4466 4475, PMLR, 2021.
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training helpful and harmless assistant with rein- forcement learning from human feedback. arXiv preprint DrXiv:2204.05862. 2022.
Jeon, H., Kim, Y., and Kim, J.-j. L4q: Parameter efficient quantization-aware training on large language models via lora-wise Isq. arXiv preprint trXiv:240.009900 2024.
Chai, Y., Kwen, M., Brooks, D., and Wei, G.-Y. Flexquant: Elastic quantization framework for locally hosted Ilm on edge devices, 2025. URL htwph:tt////eee /ttss///axxvvoooo abs abs/2501.077 07139.
Köpf, A., Kilcher, Y., von Rütte, D., Anagnostidis, S.,
Tam, Z. R., Stevens, K., Barhoum, A., Nguyen, D., Stan- ley, O., Nagyfi, R., et al. Openassistant conversations- democratizing large language model alignment. Advances in Neural Information Processing Systems, 36, 2024.
Chen, L., Ye, Z., Wu, Y.. Zhuo, D., Ceze, L., and Krishna- murthy, A. Punica: Multi-tenant lora serving. Proceed- ings of Machine Learning and Systems, 6:1-13, 2024.
Lewis, M. Bart: Denoising sequence-to-sequee pre- training for natural language generation, translation, and comprehension, arXiv preprint arXiv: 11100.1346. 13461, 2019.
kumbong
httg://wwwwaaaighfffoaaaainnrnnr
10

---

LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits
Li, M., Lin, Y., Zhang, Z.,, Cai, T., Li, X., Guo, J., Xie, E., Meng, C., Zhu, J.-Y., and Han, S. Svdqunat: Absorb- ing outliers by low-rank components for 4-bit diffusion models. arXiv preprint arXiv:2411.0500
Meng, F,, Wang, Z., and Zhang, M. Pissa: Principal singular values and singular vectors adaptation of large language models. arXiv preprint arXiv:2404.02948. 2024.
2024.
Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. arXiv preprint arXiv: orXiv:6609.000 07843,, 2016.
Li, Y., Yu, Y., Liang, C., He, P., Karampatziakis, N., Chen, W. and Zhao, T. Loftq: Lora- fine-tuning--wwe quantization for large language models. arXiv preprint 1xxiv23110.08659, 2023.
Mitchell, S., OSullivan, M., and Dunning, 1. Pulp: lin- ear programming toolkit for python. The University of Auckland, Auckland, New Zealand, 65:25, 2011.
Liao, B. fake and true quantization don't match is- sue #7 (loftq), 2023. URL https: / thub. com/ 1xxi2123/Loft/issusee 28.
Accessed 2025-05-
Narayan, S., Cohen, S. B., and Lapata, M. Don't t give me the details, just the summary! topic-aware convolutional neu- ral networks for extreme summarization. arXiv preprint ww.Xivv808...rm 08745, 2018.
Liao, B., Herold, C., Khadivi, S., and Monz, C. Apiq: Finetuning of 2-bit quantized large language model. arXiv preprint 200:2402.05147, 2024.
Ostapenko, O., Su, Z., Ponti, E. M., Charlin, L., Roux, L., Percira, M., Caccia, L., and Sordoni, A. Towards modular Ilms by building and reusing a library of loras. arXiv preprint arXiv:2405.. 11177, 2024.
Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. Deepseek- v3 technical report. arXiv preprint wrXvv:2412.19437, 2024a.
Qin, H., Ma, X., Zheng. X., Li, X., Zhang, Y., Liu, S., Luo, J., Liu, X., and Magno, M. Accurate lora-finetuning quantization of llms via information retention. arXiv 2024.
Liu, F., Lin, K., Li, L., Wang, J., Yacoob, Y., and Wang. L. Mitigating hallucination in large multi-modal models via robust instruction tuning. In The Twelfth International Conference on Learning Representations, 2023.
preprint urXiv:2402.0544
Saltzman, M. J. Coin-or: an open-source library for opti- mization. Programming languages and systems in com-
Liu, H., Tam, D., Muqeeth, M., Mohta, J., Huang, T., Bansal, M., and Raffel, C. A. Few-shot parameter-efficient fine- tuning better and cheaper than in-context learning. Ad- vances in Neural Information P Processing Systems, 35: 1950-1965, 2022a.
putational economics and finance, pp. 3-32, 2002.
Savarese, P., Yuan, X., Li, Y., and Maire, M. Not all bits have equal value: Heterogeneous precisions via trainable noise. Advances in Neural Information Processing Systems, 35: 35782,, 2022.
Liu, Y., Li, H., Cheng, Y., Ray. S., Huang, Y., Zhang, Q., Du, K., Yao, J., Lu, S., Ananthanarayanan, G., et al. Cachegen: Kv cache compression and streaming for fast large language model serving. Proceedings of the ACM SIGCOMM 2024 Conference, pp. 38 38-56, 2024b.
Shen, X., Dong, P., Lu, L., Kong, Z., Li, Z., Lin, M., Wu, C., and Wang. Y. Agile-quant: Activation Activation-guided guided quantiza- tion for faster inference of llms on the edge, 2023. URL https: /tt///aaxvvv////////////e///aaaa wwwg/aa.sa..oooooooeeeeeoeoeeeee aaa//bb//233 /2112. 05693.
Liu, 7.,, Cheng, K.TT., Huang, D., Xing, E. P., and Shen, 7. quantization: Towards accurate quantization via generalized straight-through estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1944249522 2022b.
Sheng, Y., Cao, S., Li, D., Hooper, C., Lee, N., Yang, S., Chou, C., Zhu, B., Zheng, L., Keutzer, K., al. Slora: Scalable serving of thousands of lora adapters. Proceed- ings of Machine Learning and Systems, 6:296-311, 2024.
Tan, F., Lee, R., Lukasz Dudziak, Hu, S. X., Bhattacharya, S.. Hospedales, T., Tzimiropoulos, G., and Martinez. B. Mobilequant: Mobile-friendly quantization for on-device language models, 2024. URL peettps://axxiv.ooo abs 2440.448.11 13933.
Lloyd, S. Least squares quantization in pcm. IEEE transac- tions on information theory. 28(2):129-13) 1982.
Mao, Y., Mathias, L., Hou, R., Almahairi, A., Ma, H., Han, J., Yih, W.tt, and Khabsa, M. Unipelt: A uni- fied framework for parameter-efficien language model tuning. arXiv preprini arXiv:21 10.00577, 2021.
Touvron, H. Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models, arXiv preprint arXiv:2302 /3971, 2023a.
Max, J. Quantizing for minimum distortion. IRE Transac- tions on Information Theory, 6(1):7-12, 1960.

---

LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits
Touvron, H., Martin, L.,, Stone, K., Albert, P., Almahairi, A., Bahaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288 2023b.
Zhou, C., Richard, V., Savarese, P., Hassman, Z., Maire, M., DiBrino, M., and Li, Y. Sysmol: A hardware- software co-design framework for ultra-low and fine- grained mixed precisson neural networks. arXiv preprint arXiv:231 1. 2023.
Wang, S., Yu, I.,, and Li, J. Lora-ga: Low-rank adap- tation with gradient approximation. arXiv preprint 1rXiv:2407.05000 2024.
Ziegler, D. M., Stiennon, N., Wu, Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. Fine-tuning language models from human preferences, arXiv preprint arXiv: 1199999.0999. 2019.
Wang, X., Wang. P., Wang, B., Zhang, D., Zhou, Y., and Qiu, X. Bitstack: Any-size compression of large language models in variable memory environments, 2025. URL
https: http//axxvvvv.r/./ wwtp:oog/g/o/oooooo.ooooo wtwp/////....... 233188......
Wang, Y., Mishra, S., Alipoormolabashi, P., Kordi, Y., Mirzaci, A., Arunkumar, A., Ashok, A., Dhanasckaran, A. S., Naik, A., Stap, D., et al. Super- taperraattraliisttuucinns:: Generalization via declarative instructions on 1600+ nip tasks. arXiv preprint arXiv:2204.07705, 2022.
Waswani, A.. Shazeer, N., Parmar, N.. Uszkoreit, J., Jones, L.,, Gomez, A., Kaiser, L.,, and Polosukhin, I. Attention is all you need. In NIPS, 2017.
Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. Finetuned lan- guage models are zero-shot learners. arXiv preprint (XX.:2209.00152. 2021.
Yang. H., Duan, L., Chen, Y., and Li, H. Bsq: Explor- ing bit-level sparsity for mixeddpprecssie neural network quantization. arXiv preprint arXiv:2102.10466: 2021.
Yao, J., Li, H., Liu, Y., Ray, S., Cheng, Y., Zhang, Q.. Du, K., Lu, S., and Jiang, CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion. In Proceedings of the Twentieth European Conference Computer Systems, pp. 94 909 2025.
Zaken, E. B., Ravfogel, S., and Goldberg, Y. Bitfit: Simple parameter-efffcieet fine-tuning for transformer- based masked Linguage-modeh arXiv preprint arXiv:2106.10199 2021.
Zhang, Q. Imran, A., Bardhi, E., Swamy, T., Zhang, N., Shahbaz, M., and Olukotun, K. Caravan: Practical On- line Learning of In-Network MI. Models with Labeling Agents. In 18th USENIX Symposium on Operating Sys- tems Design and Implementation (OSDI 24), pp. 325 345, 2024.
Zhao, Y., Lin, C.-Y,, Zhu, K., Ye, Z.,, Chen, L.., Zheng, S., Ceze, L., Krishnamurthy, A., Chen, T., and Kasikci, B. Atom: Low-bit quantization for efficient and accurate Ilm serving. Proceedings of Machine Learning and Systems, 6:196-200 2024.
12

---

LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits
A. LowRA System Support for Low-Bit Fine-Grained LoRA Fine-tuning
In this appendix, we provide an overview of the CUDA-based system support we built for supporting low-bit fine-grained quantization and dequantization for LoRA fine-tuning. We specifically introduce the Quantize and Dequantize Kernels for low-bit fine-grained LoRA fine-tuning. We integrate this into the bitsandbytes library for usability.
Quant
Original Weight
Decision Trees
Quantized Weight
Precs
Prees
Feed
Write
to
absmax group
Write per-group abamax
normalize
absmax array
Figure 5. Overview Kemel Low-Bit Fine-Grained Quantization. Indexing logic omitted simplicity.
Dequant
Lookup
Precs
Quantized Weight
Original Weight
Tables
Precs determine
Index
Write
read size
into
2
scale
absmax group
normalize
per-group ahsmax
absmax array
Figure 6. Overview of Kernel for Low- Bit Dequantization. Indexing logic omitted for simplicity.
A.1. Quantization (Figure 5)
During the quantization process, within the kernel, each block of data from the weight tensor is loaded, the block's maximum magnitude is computed, stored, and used for normalization, then each value is quantized according to its channel's bit-precision (defined by precs) and channel-specific decision boundaries (in the form of arrays of decision trees), and finally the packed quantized results are stored in the output buffer at the corresponding offset.
A.2. Dequantization (Figure 6)
During the dequantization process, within the kernel, each thread calculates its offset into the quantized buffer A based on the channel's bit-precision (precs), then loads the relevant packed bytes (qvals). uses absmax for the current block to rescale the dequantized values, which are obtained by applying per-output-channel look-up table on each packed quantized index. Depending on the precision (1, 2, or 4 bits), the kernel unpacks the bits from qvals, looks up the corresponding float value in one of the channel-speciee look-up table, multiplies by the saved group- wise absmax, and finally writes the results back into the output tensor.
group-wise
13

---

LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits
B. Ablation Studies
In this appendix, we perform ablation studies on the different components of LowRA. In particular, we compare the fine-tuning results with QLoRA, LoftQ, ours with only the precision assigner, and ours with both the precision assigner and the mapping/threshold searcher. Table 7 reports results of Bart-Large on XSUM and CNN//aillMail.. Table 8 reports results of Llama-2-7B on Wikitex12.
XSUM
CNN/DailyMail
Technique
Metric
2
2.25
2.5
4
2
2.25
2.5
3
4
ROUGEIT
DNC
16.3727
15.3282
17.6011
39.0742
4.8420
11.8987
13.6767
15.3374
41.1846
QLoRA
ROUGE21
DNC
2.2212
1.9712
2.6801
16.3124
0.0040
1.3188
1.0364
1.1239
18.3249
ROUGEL1
DNC
12.8367
12.5547
13.9294
31.0891
4.3608
10.2535
9.9929
10,4375
27.5773
ROUGEI 31.8941
32.6153
33.7645
36.0222
40.3429
38.8866
39.3648
39.8138
40.4684
41.1247
LoftQ
ROUGE21 10.1775
10.7005
11.7335
13.4883
17.0615
16.4935
16.8711
17.1934
17.7529
18.2853
ROUGEL! 24.5908
25.1533
26.2388
28. 1558
31.9186 25.8534
26.2877
26.5726
26.8812
27.5372
ROUGEI1 31.8941 36.4856
37.2861
18.1543 40.3429
38.8866 40.3669 40.6187
41.1820
41.1247
Ours, PA Only
ROUGE21 10.1775 13.6310 14.3775
18.1856 17.0615 16.4935 17.6272 17.8600 18.3966 18.2853
ROUGEL! 24.5908
28.4593 29.2036
29.9965
31.9186 25.8534
26.7442 26.9107
27.4180 27.5372
ROUGEI 36.7454 37.2915 37.6897 38.8396
40.2669 40.1489
41,0133 40.8819 40.8470 40.9493
Ours, PA MTSearch ROUGE21 13.9324 14.3641 14.7587 15.6822 17.1800 17.4843 18.1898 18.0609 18.1191 18. 18.1223
ROUGEL1 28.6133 29.1175 29.5275 30.5712 32.0614 26.6717 27.2268 27.0113 27.2309 27.5392
Table 7. Comparison of ROUGE scores with Bart- on XSUM and CNN/DailyMail under different techniques and combinations of techniques. PA refers to the two-level precision assigner. MTSearch refers to mapping and threshold search.
Wikitext2
Technique
Metric
1.5
1.75
1.8
1.9
2
2.25
2.5
3
4
Perplexity
9.17
8.67
8.05
7.13
6.22
QLoRA
Accuracy
0.526
0.534
0,546
0.566
0.583
Perlexity
8.63
8.22
7.72
6.87
5.26
LoftQ
Accuracy
0.536
0.543
0.552
0.571
0.613
Perplexity
ND
ND
ND
ND
8.63
8.22
7.75
6.75
5.26
PA Only
Accuracy
ND ND
ND
ND
0.536
0.542
0.550
0.570
0.613
Perplexity
9.38
7.76
7.50
7.13
6.60
6.40
6.23
5.84
5.25
PA+MTSearch
Accuracy 0.521
0.548 0.553 0.562 0.574 0.578 0.582
0.593
0.6123
Table 8. Comparison of perplexity and accuracy of Llama2-7B on the Wikitext2 dataset under different techniques and combinations of techniques. "ND" indicates no data, *_" indicates the techniques fail support these precisions. PA refers the two-level precision assigner. MTSearch refers to mapping and threshold search.
We observe generally that the precision assigner gives a significant advantage for Bart-Large on summarization tasks (Table 7), whereas the mapping/threshold searcher yields sigmificant gains for Llama-2-7B on Wikitext2 (Table 8). Interestingly, the precision assigner only yields minimal advantage over LofiQ for Llama-2-7B on Wikitext2.
In terms of applying the mapping/threshold searcher to Bart-Large (Table 7), we see a less significant gain in the 2.5-4.0 precision range in comparison to the gains in the 2.0-2.5 precision range. This is in line with our intuition at higher precision there are more mappings and thresholds for capturing distributions during quantization and dequantization; thereby, the tolerance for a less accurate combination of mappings and thresholds is higher.
We encourage future works to build upon LowRA and study the optimal precision assignment and mapping/threshold search
14

---

LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits
algorithms for different types of architectures (i.e., encoder-decoder, encoder-only, decoder-only). We will provide easy integration new advances in open-sourced repository.
15

---

LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits
C. Memory Requirements
this appendix, we provide detailed breakdown of memory footprints. Figure shows the fine-tuning memory footprint decompositions for LLaMA-2-7B and LLaMA-2-13B, Figure 8 shows the inference memory decomposition for LLaMA-2- 7B and LLaMA-2-13B. Figure shows both inference and fine-tuning memory footprint decomposition for LLaMA-33B. For fine- fine-tuning, we follow QLoRA's setup of using batch size of I and sequence length of 512. Number labels on the bars are in Mega (MB). Estimations are linear layer only (not attention).
Model
LoRA
Weight Grad
Optimizer
Input Grad
Perplexity
Llama-2-78
Llama-2-138
1.0
1.0
Memory Usage (Normalizedd Max)
1152
7.5
Memory Usage (Normalizeed Max)
1800
6.55
1152
306
288
0.8
1800
450
6,25
0.6
1152
306
288
1152
306
288
1152
1152
288
306
306
288
6.5
Perplexity
0.6
1800
479
450
1800
1800
479
450
450
479
1800
450
479
479
450
450
479
-5.75
66.00
Perple expplexity
0.4
5046
8476
5550
4274
3425
3502
3695
3888
5148
5300
5451
5829
6207
5.25
0.2
0.2
5.5
5.00
-4.75
0.0
0.0
1.75
1.8
1.9
2.0
2.25
2.5
3.0
4,0
1.75
1.8
1.9
2.0
2.25
2.5
3.0
4.0
Bits per Parameter (BPP)
Bits per Parameter (BPP)
Figure 7. Decomposition of fine-tuning memory footprint for LLaMA-2 7B and 13B under different bits per parameter.
Model
Adapters
Llama-2-78
Llama-2-13B
1.0
306
479
Memoory Usage (Normalized to Max)
-7.5
Memory Usage (Normalized to Max)
6.50
0.8
0.8
306
6.25
306
306
306
306
-7.0
6.00
0.6
Perplexity
0.6
Perplexity
5.75
0.4
4274
0.4
5.50
3309
3348
3502
-6.0
5148
5300
5.25
0.2
0.2
5.5
5.00
4.75
0.0
0.0
1.75
1.8
1.9
2.0
2.25
2.5
3.0
4.0
1.75
1.8
1.9
2.0
2.25
2.5
3.0
4.0
Bits per Parameter (BPP)
Bits per Parameter (BPP)
Figure 8. Decomposition of inference memory footprint for LLaMA-2 7B and 13B under different hits per parameter.
16

---

LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits
Model
LoRA
Weight Grad
Optimizer
Input Grad
Inference Memory Usage
Finetune Memory Usage
Memory Usage Noomalieed Max)
Memory Usage (Normalize) menalizzed to Max)
3510
0.8
0.6
3510
3510
0.4
932
888
19302
0.2
8395
8778
0.2
8395
8778
0.0
1.i5
1.25 Bits per Parameter (BPP)
4.0
1.15
1.25 Bits per Parameter (BPP)
4.0
Figure 9. Decomposition of memory footprint for LLaMA-33B under different bits per parameter.
D. Low-Rank Initializers: LoftQ vs PiSSA
Both LoftQ (Li et al., 2023) and PiSSa (Meng et al., 2024) use an iterative two-step process to enhance LoRA fine-tuning by exploiting low-rank structure and quantizing the residual. each iteration:
1. Low-Rank Decomposition: A singular value decomposition (SVD) of the current weight (or updated version of it) is performed to factor out low-rank approximation.
2. Residual Quantization: The remaining component (i.e., the difference between the original weight and the low-rank approximation) quantized to preserve overall model capacity with fewer bits.
The key distinction lies in how they initialize this process:
LoftQ first quantizes the residual, which the beginning simply the full base weight. Thus, "initiaaizes"" treating the entire unmodified weight residual be quantized and only then proceeds with the low-rank factorization in subsequent iterations.
PiSSA starts by performing SVD on the unquantized base weight, extracting a low-rank representation before any quantization. Only after factoring out the low-rank component does PiSSa quantize the remaining residual.
Below (Table 9) are the experimental results covering the full 2.0-to-4.0 range comparing these two initialization techniques. We adopt LoftQ's mixed precision scheme for intermediate precision targets.
Sctup
LLaMA-7b
LLaMA-13b
Method Dataset
2.0
2.25
2.5
3.0
4.0
2.0
2.25
2.5
3.0
4.0
PiSSA LofiQ
1919.63
795.88
1938.33
1397.26
5.53
1825.68
1822.62
1769.34
1549.48
5.05
WikiText-2 (L)
8.63
8.22
7.72
6.87
5.26
7.27
6.96
5.7
5.91
4.79
Table Perplexities of PiSSA and LoftQ as initialization methods on WikiText-2 covering full range from 2.0 to 4.0. Lower values indicate better performance (L).
17

---

LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits
E. Overheads of Mapping/Thresholds Searcher and Precision Assigner
In this appendix, we report the overhead incurred by LowRA's newly added components. Note that LowRA's map- ping/threshold learner and precision learner can both be done offline. The former only needs to be rum for each model architecture, while the latter only needs be for each combination of model architecture and user-specified precision requirement.
HlappingTThreshlke Learner Overhead We timed the overhead of our mapping/threshold learner single NVIDIA A100 SXM GPU with GPU memory. For each precision, we ran 2 iterations of the Lloyd-Max algorithm, which we found sufficient to push down the MSE and enhance task performance. From the average 10 runs, each run on Bart-Large, LLaMA2-7B, and LLaMA2-153 takes 111.46, 309.87, and 464.92 seconds, respectively.
Solver Overhead We build our two-level ILP pipeline using the opensourced Coin-Or Branch and Cut (CBC) (Saltzman, 2002) solver via the Python-base bassd modeling library PuLP (Mitchell et al., 201 1). The experiments 2x Intel Xcon Gold 6342 CPUs. The solver uses 8 threads in parallel. We timed the overhead of rumning this two-level ILP pipeline for each combination of model architecture and precision requirement. From the average of 10 runs, each run on Bart-Large, LLaMA2-7B, and LLaMA2-13B takes 48.9 48.99 seconds, 319.13 seconds, and 665.93 533 seconds, respectively, Note that an exact (i.e., one-step) solver for the same problem would fail to finish in a reasonable amount of time.
server with
F. Initialization of the Mapping and Threshold Learner
We attach the mappings and thresholds use for weighted Lloyd-Max algorithm in Listing
mappings 4bit init forch.tenser
11-1.0, u, -0. 6961928, -0.5250731, -0. 39499755 -0.29444138, -0.18477343, -0.09105, 0.0, 0.0795803, 0.1609302, 0. 0.2461123, 0.33791524, 0.44070983, 0.562617, 0. 0.72295684, 1.0]], dtype-torch.float device=device repeat (nchannels, 1)
Brresholds_2bit___ init = torch.tenser
[[-0.5, 0.16895762, 0.668957621), dtype=torch. 32oat32, device=device repeat (nchannels, 1)
sappings_2bit_ing torch. tensor (
[[-1.0, 0.0, 0.3379, 1.011, syypestorch.float22. device=device repeat (nchannels,
thresholde_lbit_ing = torch.tenscri
[[0.0]],
device=device repeat (nchannels, 1)
mappings_ibbe lbit init
11-1.0, 1.011,
dtype-torch. float32, device=device repeat (nchannels, 1)
Listing 1: Initializations for bit bie-iircision mappings and thresholds.
18

---

LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits
G. Weighted Lloyd-Max Algorithm
this subsection, we briefly introduce the Weighted Lloyd-Max Algorithm (Lloyd, 1982; Max, 1960).
Let {= C. IRd be data points with corresponding weights fwi wy We seek find K cluster centers {ys 1 IRd minimizing the weighted mean-squared enror:
N
24 y0(1) 1|
min
wi
where c(i) E (I,...,K K} is the cluster index assigned to The weighted Lloyd's algorithm alternates between:
1. Assignment (E-step): Assign each data point to the cluster center closest in Euclidean distance:
c(i)
< arg
|| 10--0 y/ 11? 1SiSK
(E-step)
Update (M-step): Recompute each cluster center y/j as the weighted centroid of the points assigned it:
Exe(() W; 24
yj Σne(i) Wi
(M-step)
These steps are repeated until convergence or until a stopping criterion (c.g., a maximum number iterations) met.
19

---

LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits
H. Patterns of Parameter Values in Large Language Models
randomly sample 200x200 blocks and plot their distributions as heatmaps. In this appendix, we present representative parameter distributions within each type of tensor in Figure 10 (note that there are two parts). Generally, observe banded patterns across input channels and separated by output channels in Self-Attention K-Projection (Figure 10a), Self-Attention Q-Projection (Figure 10b), and Self-Attention V-Projection layers (Figure 10c), which have been shown to be more sensitive to quantization errors. For MLP Gate-Projection layers (Figure 10f), we observe some similar patterns, although not as significant. Self-Attention O-Projection layers (Figure 10d), MLP Up-Projection layers (Figure 10c), and MLP Down-Projection layers (Figure 10g) have distributions that are mostly evenly spread out.
Layer 6 Self-Attention, K-Projection
Layer 10 Self-Attention, Q-Projection
+0.075
+0.075
Output-channel index
+0.05
Output-channel index
+0.05
50
50
+0.025
+0.025
100
100
0
-0.025
-0.025
150
150
-0.05
-0.075
-0.075
50
150
100
150
Input-channel index
Input-channel index
(a) Layer 6 Sclf-Attention K projection.
(b) Layer 10 Self-Attention Q projection.
Layer O Self-Attention, V-Projection
Layer 16 Self-Attention, O-Projection
+0.04
+0.04
Output-channel index
Output-channel index
50
+0.02
50
+0.02
100
100
0
150
-0.02
150
-0.02
-0.04
-0.04
150
50
100
150
Input-channel index
Input-channel index
(c) Layer 0 Self-Attention V projection.
(d) Layer 16 Self-Attention O projection.
Figure Sampled 200x200 parameter value heatmaps from different tensors from different layers in LLaMA2-7B (1/2).
20

---

LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits
Layer 23
Layer 29
MLP, Up-Projection
MLP, Gate-Projection
0
+0.06
+0.04
Output-channer index
Output-channel index
+0.04
50
50
+0.02
+0.02
100
100
-0.02
-0.02
-0.04
-0.04
-0.06
50
100
150
50
100
150
Input-channel index -
Input-channel index -
(e) Layer MLP Up projection.
(i) Layer 29. MLP Gate projection.
Layer 27 MLP, Down-Projection
+0.04
Output-channel index
50
+0.02
100
-0.02
150
-0.04
50
100
150
Input-channel index -
(g) Layer MLP Down projection.
Figure 10. Sampled 200 parameter value healmaps from different tensors from different layers in LLaMA2-7B (2/2).
21

---

LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits
I. Experiment Hyperparamter Setup
this appendix, we lay out the hyperparameters used for different experiments.
Hyperparameter
Value
model
meta-Ilama/ meta-Ilama/Llmma222277 Llama-2-7--ht
data.seed
42
evaluatiionsttaaeey
steps
eval.dataset.size
1024
max.eval.samples
1000
per.dovicee.vaa..attc...zze
@atalooder..mm.orkeers
1ora
64
lora-alpha
64
modules
all
bf16
True
warmup-ratio
0.03
ir.scheduler.type
cosine
pradient.checkooittin
True
dataset
wikitext
dataset.config
wikitext-2-rawwv1
per.@erice.ttriinaatch...ce
16
gradien..ccommlatiinnssssss
max.steps
126
eval.steps
20
learning.rate
0.0003
adam.beta2
0.999
max.grad.norm
0.3
weight.decay
0.1
seed
0
block-size
1024
Table 10. Hyperparameters used for all LLaMA experiments on Wikitext-2.
22

---

LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits
Hyperparameter
Value
learning.rate seed dataset.name
le-4
11
enn enn dailymlllai
dataset.config
"3.0.0"
pad to max mex_te max.source.length num train epochs
True
512
15
per.@erice.traannbatch..iiee
8
per.cerrice.evvll..aatcccahaaa
32
gradiiett.ccumm]ltiionsseee
32
model_name.or.path
facebook/barttlarge epoch
evaluatiion.strateey predict.wilthh.enerate
True
Table 11. Hyperparameterss for fine-tuning Bart-Large CNN/DailyMaiil
Hyperparameter
Value
learning.rate
le-4
seed
11
dataset_name
Xsum
dataset.config
"3.0.0"
pad.to.max.length
True
max.source.lengthh
512
num.train.epochs
25
per.dervie.ttaainaaatccs.seee
4
per.deriice.evval.attchhssieee
32
gradieett.cccmm]aatiinnnseee
32
model__ammeor..path
facebook/bart-large epoch
evaluation.strateegy
Table 12. Hyperparameters for fine-tuning Bart-Large on XSUM.
23

---

LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits
Hyperparameter
Value
model name or path
Sata-IllamaLLlama 2---:-a--mmaa-a----a 2f
data.seed
42
evaluation strategy
steps
eval.dataset.size
1024
max eval samples
500
per..erice..eallaatcchh..e.
max mex.new.token.
dataloader dataloader..um.wookeee.. www.:rrneuaawoor
group.by.length
True
logging.stratesy
steps
renove.unused.colums
False
rora
lora-alpha
64
lora modules
all
bf16
True
warmup ratio ir.scheduler.type
0.03
constant
gradient checkpointing
True
dataset
oasstl
mource.max.1e.
target_max.len
512
per.device.traiinaatce
4
grailen..ccommlatiioutesss
max.steps
1875
eval steps rearning.rate adam beta2 max.grad.norm
200
0.0002
0.999
0.3
lora dropout
0.1
weight.decay
0.0
seed
Table 13. Hyperparameters used for all Llama experiments on Open Assistant (oasstl).
24

---

LowRA: Accurate and Efficient LoRA Fine-Tuning of LLMs under 2 Bits
J. Github Issues Related To The Lack of A Practical Quantization Primitive
http:///gghhaaa.occo/////l////tt The use of NF fake quantization LoftQ
Discrepancy between real model weights and ex-
pected model weights due to fake quantization in LoftQ
https//ssstttaaoooo////22/2/2//2 The use of NF fake quantization in LoftQ
Unrealized GPU memory saving due to fake quanti-
zation in LoftQ
httpaa/c.c///eaaaaa////////aa/// Unrealized LLM model size reduction due to fake quantization in PiSSA
K. Detailed LoftQ Results on Bart-Large
Setup
Bart-Large
Method
Dataset
Layer Index
92.62/10.70225 33.76/11.73/26.22 36.02/13.49/28.11
XSUM
31.8/10.18/24.99
40.34/17.06/1..22
Encoder First
32.72/10.94/25388 34.49/12.2627.05 37.2314.34/29.311
Layer Index
39.36/16.87/26.29 39.81/17.19/66.7 40.47/17.75/26.88
CNN/DailyMail 38.89/16.49/25.85
41.12/18.29/27.54
Encoder First
39.27/16.84/26.26
39.53/17.05/26.55 39.89/17.36/26.73
Table 14. Rouge-/RRouge-2Rougee- of Bart-Large fine-tuned with LoftQ. Higher scores indicate better task performance.
25


Reviews:
[{"review_id": "OOHTeGqutg", "reviewer": "Reviewer_7mTX", "summary": "This paper introduces **LowRA**, a novel framework for LoRA-based fine-tuning of LLMs in ultra-low bit (sub-2-bit) settings. LowRA is the first to enable LoRA fine-tuning at or below 2 bits with only minor accuracy/perplexity losses and achieves considerable memory savings (30–50%).\n\nThe authors observe that current quantized LoRA methods (e.g. QLoRA, LoftQ) typically fail or degrade significantly below 2-4 bits. To address this, LowRA proposes a fine-grained, mixed-precision quantization strategy that - (a) learns quantization thresholds and representative values (mappings) via a weighted Lloyd-Max procedure, (b) uses a two-level ILP solver to assign channel-wise precisions, and (c) includes optimized low-bit CUDA kernels tailored to LoRA.\n\nExperiments on Llama-2 (7B, 13B parameters) and BART-large, across tasks including WikiText-2, OpenAssistant, CNN/DailyMail, and XSum, with additional experiments on Llama 33B show that LowRA outperforms baselines in the performance-precision trade-off.\n\nNotably, LowRA is the first method to enable Llama-30B fine-tuning on a single NVIDIA Tesla T4 (16GB VRAM).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer 7mTX for their thorough and insightful review. We are glad that 7mTX finds our empirical gains convincing and our paper easy to read and follow. We address all feedback below and will incorporate it, along with new experimental findings, into the revised paper.\n\n---\n\n## Q1: Inference metrics on consumer-grade hardware\n\nOur experiments target GPUs that are relatively constrained for large language model (LLM) inference—an **NVIDIA RTX 3080** with **10GB VRAM** and an **NVIDIA RTX A4000** with **16GB VRAM**—to underscore how LowRA can unlock efficient inference without requiring high-end data-center hardware.\n\n\n### Llama 7B on RTX 3080 (unit: tokens / sec; float32) \n| Metric        | Prefill Length | Decode Length | 1.5 Bit | 2 Bit  | 2.5 Bit | 4 Bit  | QLoRA (4 Bit) |\n|---------|----------------|---------------|---------|--------|---------|--------|---------|\n| **Throughput**| 100            | 10            | 32.16   | 30.33  | 24.35   | 9.19   | 9.40          |\n| **Speedup**   | 100            | 10            | 3.42x    | 3.23x   | 2.59x    | 0.98x   | 1.00x          |\n\n - At **1.5 bits**, throughput reaches **32.16 tokens/sec**, yielding a **3.42× speedup** over QLoRA (9.40 tokens/sec).  \n - Even at **2 bits**, LowRA surpasses QLoRA by over **3×**, indicating that more aggressive quantization pays off on memory-limited hardware.\n\n\n\n### Llama 13B on A4000 (unit: tokens / sec; float32) \n| Metric        | Prefill Length | Decode Length | 1.5 Bit | 2 Bit  | 2.5 Bit | 4 Bit  | QLoRA (4 Bit) |\n|---------------|----------------|---------------|---------|--------|---------|--------|---------------|\n| **Throughput**|  100  |  10   |  16.11 | 14.46  | 13.64  |  12.17 |  11.56  |\n| **Speedup**   | 100           | 10            | 1.39x    | 1.25x   | 1.18x    | 1.05x   | 1.00x          |\n\n \n- For **Llama 13B**, 1.5-bit LowRA achieves **1.39×** speedup over QLoRA, and 2-bit still delivers a **1.25×** improvement.  \n - Though the gains are smaller than on the 7B model—likely due to higher overall computation overhead—LowRA still outperforms QLoRA across all tested bit-widths.\n\nThese results confirm that our sub-2-bit LowRA quantization provides real-world throughput gains beyond the documented memory savings.\n\n---\n\n## Q2: The tradeoff of model size and compression ratio.\nIn practice, users often start with a specific model, a fixed hardware memory budget, and particular tasks in mind. LowRA is designed to help maximize performance under those given constraints. \n\nWhile we do observe that ultra-low-bit quantization can make it feasible to run larger models within limited memory, our current study does not fully investigate the potential trade-off between choosing a bigger model at extremely low bits (e.g., 13B at 1.5 bits) and opting for a smaller model at higher bits (e.g., 7B at 2-4 bits). \n\nWe see further exploration of this bits-per-parameter versus performance trade-off - along with automatic selection of the optimal model-compression combination - as an important direction for future research. Moreover, our work serves as a stepstone for future works to further optimize the task performance of models in the lower bit range.\n\n\n---\n\n## Presentation Errors: Graphs, Text, and Algorithms\n\nThank you for your attention to details! We have fixed all of the presentation errors you pointed out. We will iterate through our draft rigorously to ensure our presentation is accurate.\n\n1. **Typographical and Spacing Fixes**  \n   - Corrected spacing for “Data-Free Post-Training Quantization.”  \n   - Fixed spelling errors (e.g., “Quantzation” → “Quantization,” “Summarization Summarization” → “Summarization”).  \n\n2. **Figure and Label Updates**  \n   - Corrected the mislabeled memory footprint for Llama 33B in Figure 9.  \n   - Standardized references to “Llama-33B” (instead of “Llama-30B”).  \n   - Adjusted the y-axis to a full-scale view in Figures 7 and 8, ensuring a clearer comparison of perplexity scores.  \n\n3. **Algorithm Clarifications**  \n\n    We have added clarifications to the algorithm. Specifically, $N$ refers to the total number of output channels in the model being quantized. The values $\\{ w^{(1)}, \\dots, w^{(K)} \\}$ represent the distinct parameter sizes (i.e., the number of parameters) across all output channels. The set $I_k$ contains the channel indices belonging to size group \\(k\\). Each channel \\(i \\in I_k\\) has a parameter count of $w^{(k)}$.\n\n    We define:\n\n    $$\n    W_k = \\sum_{i \\in I_k} w^{(k)},\n    $$\n\n    which is the sum of parameter counts for all channels in partition $I_k$. Finally:\n\n    $$\n    W_{\\text{sum}} = \\sum_{k=1}^K W_k\n    $$\n\n    is the total parameter count across all partitions (i.e., the total number of parameters in the model to be quantized).", "claims_and_evidence": "- **Claim A:** LowRA can fine-tune LLMs below 2 bits (down to ~1.15 bits in some cases) without catastrophic performance degradation.\n**Evidence:** Empirical results on Llama-2–7B and 13B (down to ~1.75 bits) [from Table 2] and Llama-33B (down to ~1.15 bits) [from Table 3] show perplexities close to baseline 2–4-bit methods.\n\n- **Claim B:** LowRA yields superior or on-par performance compared to existing sub-4-bit baselines (QLoRA, LoftQ) while using fewer bits.\n**Evidence:** On WikiText-2 and OpenAssistant, LowRA at ~2.5 bits approaches or exceeds the performance of QLoRA/LoftQ at 4 bits. [from Table 2]\n\n- **Claim C:** LowRA substantially reduces memory usage (30–50%), valuable for resource-constrained fine-tuning and on-device inference.\n**Evidence:** The authors detail memory estimates for Llama 2-7B,13B, & Llama-33B) in Appendix C.", "methods_and_evaluation_criteria": "1. WikiText-2 (language modelling) and OpenAssistant (multi-turn conversation) are evaluated via perplexity.\n2. XSUM and CNN/DailyMail (summarization tasks) are measured with ROUGE scores.\n\n*Datasets make sense for the problem at hand.*\n*Perplexity and ROUGE are among the most widely accepted metrics for these tasks. They ensure that both language generation quality (summarization) and text coherence/prediction (perplexity) are rigorously tested.*", "theoretical_claims": "The main theoretical component is the ILP-based assignment of channel-wise precisions. While no formal proof of optimality is given for the entire method’s synergy with LoRA - the authors do provide a well-defined ILP objective and a hierarchical solution scheme. The Weighted Lloyd-Max approach is grounded in standard quantization theory. Everything is consistent with known quantization methods; there are no obviously incorrect proofs. The approach is algorithmic rather than strictly theoretical, and the correctness of the optimization steps seems reasonable.", "experimental_designs_or_analyses": "The experimental designs and analyses seem fine.", "relation_to_broader_scientific_literature": "LowRA builds directly on the concept of parameter-efficient fine-tuning (PEFT), esp. LoRA (Hu et al., 2021), and extends quantized LoRA approaches like QLoRA and LoftQ. It adopts groupwise quantization ideas from prior work (e.g., GroupWise Normalization in QLoRA), but pushes further by combining a Weighted Lloyd-Max algorithm for adaptive thresholding with two-step ILP-based precision assignment, enabling sub-2-bit training. This unified approach addresses limitations observed in earlier methods (which were mostly restricted to 2-4 bits) and draws on established mixed precision and quantization techniques.", "essential_references_not_discussed": "None - to the best of my knowledge.", "other_strengths_and_weaknesses": "**Strengths**\n1. Empirical gains are convincing - particularly that performance remains stable even in ultra-low bit settings below 2 bits. \n2. Overheads of the framework components are discussed. (Appendix E)\n3. In the low bit range (2-4), the proposed framework outperforms LoftQ and QLora. (Table 2.)\n4. The paper is easy to read and follow. Good overall presentation. esp. sections 5 and 6 - the core components of the paper. \n\n**Weaknesses**\n1. Although the framework supports sub-2-bit fine-tuning, insights into how sub-2-bit quantization affects LLM behaviour (e.g., interpretability, generalization) are limited. Some fundamental insight would be valuable, not just system performance.\n\n2. While the paper makes progress in enabling ultra-low-bit LoRA fine-tuning and serving, it's not clear whether such ultra-low-bit quantized models outperform smaller models quantized at slightly higher bit widths. For instance, from Table 2, the **2-bit LowRA Llama-2-7B** (3808MB as from Figure 8) is comparable to the **1.8-bit LowRA Llama-2-13B** (5627MB as from Figure 8). Smaller models seem like a better choice than larger, more aggressively fine-tuned models esp. at lower bit widths scenarios. This raises an important question of whether it's always better to fine-tune larger models at lower precision, or whether smaller models at slightly higher precision might offer a better trade-off in both accuracy and throughput. A deeper discussion or analysis of this trade-off would strengthen the paper.\n\n3. The motivation of the paper is ultra-low-bit fine-tuning for resource-constrained environments. Yet, all experiments were done on high-end A100. It would be valuable to include at least some inference-time measurements (e.g., latency, throughput, memory footprint) on representative low-resource hardware. This would help contextualize the practical system-level benefits and potential overheads of the proposed CUDA kernels and fine-grained quantization approach.", "other_comments_or_suggestions": "Please see the word spacing for \"Data-Free Post-Training Quantization\". (~line 141)\n\nPlease fix the spelling error - \"Quantzation\". (Figure 5 caption)\n\nSummarization Summarization (~line 699)\n\nThe mislabeled memory footprint of Llama 33B. (Figure 9)\n\nPlease refer to Meta's first generation of mid-tier model as Llama-33B (as was officially announced). There are numerous references to Llama-30B.\n\nThe right y-axis for the perplexity score exaggerates the drop visually. A full-scale axis would be a good choice, esp. for readers assessing quality vs. compression trade-offs. (Figure 7/8.)\n\nIn Algorithm 1, it would help to clarify the definitions of inputs like N, w(1),…,w(K). W_k and W_sum.", "questions_for_authors": "In many real-world settings, the cumulative resources required for inference in deployment far exceed the one-time costs of fine-tuning.\n\n**Q1.** Do you have inference throughput benchmarks (e.g., tok/sec) for inference with and without LowRA’s sub-2-bit quantization? This would clarify real-world gains beyond memory savings.\n\n**Q2.** Can ultra-low-bit quantization with LowRA justify choosing a larger model (e.g., 13B @ 1.5 bits) over a smaller 7B model at 2–4 bits (with LoftQ)? Any guidance on this trade-off? Thanks.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9IICkAHPAt", "reviewer": "Reviewer_oGG6", "summary": "This paper introduces LowRA, a novel framework that enables LoRA fine-tuning below 2 bits per parameter while maintaining model performance. The work addresses three key limitations of existing quantized LoRA methods through innovative techniques: fine-grained precision assignment, adaptive quantization mapping/thresholding, and efficient CUDA kernels for low-bit execution. The authors demonstrate significant memory savings (up to 50%) while achieving comparable or better performance compared to state-of-the-art methods.\n\n## update after rebuttal\n\nThank you for the author's response. After referring to the author's response and the comments of other reviewers, I decided to keep my score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer oGG6 for their thorough review. We are glad that they find our work valuable overall. Below, we address the key concerns. We appreciate your feedback and will integrate it—along with additional experimental findings—into the revised manuscript.\n\n---\n\n## Training Overhead\n\nWe benchmarked **LowRA** at various bit widths against **QLORA (4 bits)** on both an **A5000 (24GB VRAM)** and **A100 (80GB VRAM)**.\n\n### Llama 7B A500, Runtime (ms)\n| Seq. Len. | Batch Size | 1.5 Bit  | 2.5 Bit  | 4 Bit    | QLORA (4 Bits) | Max % Overhead |\n|--------|-------|----------|-------|-------|------------|--------|\n| 256    | 1     | 782.67   | 789.37   | 782.67   | 754.27        | 4.65%          |\n| 512    | 1     | 1291.75  | 1295.82  | 1291.75  | 1268.50       | 2.15%          |\n| 1024   | 1     | 2398.10  | 2398.96  | 2398.10  | 2383.68       | 0.64%          |\n| 256    | 2     | 1268.17  | 1273.97  | 1268.17  | 1247.15       | 2.15%          |\n| 512    | 2     | 2319.83  | 2322.01  | 2319.83  | 2304.82       | 0.75%          |\n\n\n### Llama 13B A5000, Runtime (ms)\n| Seq. Len.  | Batch Size | 1.5-bit  | 2.5-bit  | 3-bit    | 4-bit    | QLORA (4-bit) | Max % Overhead |\n|-----|-------|------|------|------|------|------|--------|\n| 256    | 1     | 1494.69  | 1494.69  | 1501.24  | 1508.03  | 1444.20       | 4.42%          |\n| 512    | 1     | 2486.00  | 2486.00  | OOM      | OOM      | OOM           | N/A            |\n| 256    | 2     | 2457.75  | 2457.75  | OOM      | OOM      | OOM           | N/A            |\n\n### Llama 7B A100, Runtime (ms)\n| Seq. Len. | Batch Size | 1.5-bit  | 2.5-bit  | 3-bit    | 4-bit    | QLORA (4-bit) | Max % Overhead |\n|----|------|--------|-------|---|-----|-------|-----|\n| 256           | 1          | 632.09   | 632.36    | 631.85   | 639.31    | 589.12        | 8.52%          |\n| 512           | 1          | 1073.68   | 1074.60   | 1072.91  | 1079.48   | 1030.78       | 4.72%          |\n| 256           | 2          | 1056.94  | 1056.22   | 1055.86  | 1063.54   | 1014.91       | 4.79%          |\n\nLowRA introduces minimal overhead (with a maximum of 8.52%) and supports configurations **QLORA** cannot run without out-of-memory (OOM) errors.\n\n\n---\n\n## Results on Ultra-Large Models\nWe evaluated **LowRA** on **LLaMA-65B** (1.15 bpp), achieving perplexities of **7.49** (WikiText) and **4.97** (OpenAssistant), outperforming **LLaMA-33B’s** **8.00** and **5.73** under identical hyperparameters. This confirms **LowRA’s** scalability to ultra-large models.\n\n---\n\n## Variance Trends Across Different Layer Types\nWe will supplement our paper with fine-grained plots motivating output-channelwise quantization.\n\n---\n\n## Training Instability\nEven at 1.15 bits, we observed no instability or divergence, largely thanks to (1) LoftQ’s alternating SVD-based initialization of low-rank tensors and (2) freezing base weights while only updating low-rank tensors. We will include stability discussions in our paper.\n\n---\n\n## Inference implications\n\n**Inference implications involve three primary aspects: Loading Latency, Time to First Token (TTFT), and End-to-End Throughput.** We discuss each part here.\n\n1. **Loading Latency**  \n This is the time to load model weights into GPU memory. On an RTX A4000 (LoRA branch in float32), LowRA reduces loading latency significantly, as shown in the table:\n\n   | Framework | Bits/Param | Loading Latency             |\n   |-----|------|-------|\n   | QLoRA     | 4          |  220.73 s                    |\n   | LowRA     | 4          | 218.17 s                    |\n   | LowRA     | 1.5        | 157.51 s (28.6% reduction)  |\n\n2. **TTFT**  \nTTFT is the delay before the first token is produced. While smaller models transfer less data and slightly reduce TTFT, it's primarily compute-bound, so quantization alone provides modest gains.\n\n3. **End-to-End Throughput**  \n    Once the model is loaded and past TTFT, throughput (tokens or requests per second) becomes the key factor. \n\n    ### Llama 7B on 1 RTX 3080 (unit: tokens / sec; float32) \n    |Prefill Length | Decode Length | 1.5 Bit | 2 Bit  | 2.5 Bit | 4 Bit  | QLoRA (4 Bit) |\n    |------|----|---|----|-------|-------|-----|\n    | 100            | 10            | 32.16   | 30.33  | 24.35   | 9.19   | 9.40          |\n\n\n    ### Llama 13B on 1 A4000 (unit: tokens / sec; float32) \n    | Prefill Length | Decode Length | 1.5 Bit | 2 Bit  | 2.5 Bit | 4 Bit  | QLoRA (4 Bit) |\n    |----|-----------|------|----|------|--------|-----|\n    | 100  |  10   |  16.11 | 14.46  | 13.64  |  12.17 |  11.56  |\n\n    Across **Llama 7B** on an **RTX 3080**, 1.5-bit LowRA delivers a **3.42× throughput increase** over QLoRA (32.16 vs. 9.40 tokens/sec), while on Llama 13B using an **RTX A4000**, **1.5-bit LowRA** still achieves a **1.39× speedup**—demonstrating that sub-2-bit quantization offers clear performance gains across varying model sizes.\n\nThese results confirm that LowRA provides **real-world throughput gains beyond the documented memory savings**.", "claims_and_evidence": "The paper's claims are well-supported by comprehensive experimental results across multiple models (LLaMA-2-7B, 13B, 30B, BART-large) and tasks. The authors provide extensive ablation studies and detailed analysis of each component's contribution. The performance improvements and memory savings are clearly documented with quantitative metrics.", "methods_and_evaluation_criteria": "The proposed methods are technically sound and well-motivated. The evaluation methodology is comprehensive, including:\n\n1) Multiple model architectures and sizes\n2) Various downstream tasks\n3) Standard metrics (perplexity, ROUGE scores)\n4) Detailed ablation studies\n5) Memory usage analysis", "theoretical_claims": "The theoretical foundation is solid, with clear mathematical formulations for:\n\n1. Weighted Lloyd-Max quantization\n2. Two-level ILP-based precision assignment\n3. Channelwise precision optimization\n4. Memory footprint analysis", "experimental_designs_or_analyses": "The experimental evaluation is thorough and convincing:\n\n1. Comprehensive baseline comparisons\n2. Extensive ablation studies\n3. Clear performance benchmarks\n4. Detailed memory analysis\n5. Multiple model scales tested", "relation_to_broader_scientific_literature": "The work builds upon and advances existing research in:\n\n1. LoRA fine-tuning\n2. Model quantization: QLoRA, LoftQ\n3. Efficient LLM deployment\n4. Low-bit neural networks", "essential_references_not_discussed": "The paper adequately covers the relevant literature in LLM quantization and fine-tuning.", "other_strengths_and_weaknesses": "**Strengths:**\n\n1. First framework to enable LoRA fine-tuning below 2 bits. Achieved 1.75 bits on LLaMA-2-7B/13B and BART-large, and 1.15 bits on LLaMA-30B with competitive performance. No previous methods support sub-2-bit operation.\n2. Superior performance-precision trade-off. At 2-bit quantization, LowRA achieves perplexity reduction of 2.21 (WikiText-2) and 1.45 (Open-Assistant) over QLoRA, and 1.76 (WikiText-2) and 1.12 (Open-Assistant) over LoftQ.\n3. Significant memory savings. Reduces memory usage by 30-50% compared to 4-bit baselines:\n    - 40% lower memory for LLaMA-2-13B inference (2-bit vs 4-bit)\n    - 30% lower for LLaMA-2-7B inference\n    - 50% reduction for LLaMA-30B at 1.15 bits\n4. Theoretically sound precision assignment strategy. Two-level ILP formulation with proven convergence, validated by ablation studies showing effectiveness of precision assigner.\n5. Hardware-friendly design. Supports deployment on resource-constrained devices like Raspberry Pi 4 (4GB RAM) for LLaMA-2-7B and Tesla T4 (16GB VRAM) for LLaMA-30B.\n\n**Weaknesses:**\n\n1. Limited analysis of computational overhead during training. Paper focuses primarily on memory savings, with limited discussion of training time impacts in Section 7 and Appendix E.\n2. Incomplete exploration of ultra-large models. Experiments stop at LLaMA-30B, without testing on larger models like LLaMA-65B or GPT-3 (175B).\n3. Figure 10 shows variance analysis but lacks detailed explanation of impact on different layer types.\n4. While training curves are shown, there's limited discussion of potential instability issues during ultra-low-bit training.\n5. Impact on inference latency not thoroughly analyzed. Memory reduction is well-documented, but timing implications are not comprehensively measured and reported.", "other_comments_or_suggestions": "None", "questions_for_authors": "Please refer to Strengths And Weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AXwPtCecAk", "reviewer": "Reviewer_BPFc", "summary": "The authors of this paper tackle the important problem of LLM quantization, enabling fine-tuning below 2 bits per parameter with minimal performance loss. This is achieved through the proposed LowRA framework, which addresses three key challenges in quantized LoRA fine-tuning: coarse-grained precision assignment, discrepancy in data distribution, and lack of high-performance quantization primitives.\n\nThe LowRA framework effectively resolves these challenges via different techniques such as per-output-channel quantization, groupwise normalization, data-free post-training quantization, per-output-channel thresholds and mappings, data-free one-shot post-training quantization, and user-defined compression ratios. Furthermore, the authors provide practical CUDA-based primitives for seamless implementation.\n\nThe paper's experimental results demonstrate that LowRA outperforms baselines at the same precision and achieves similar performance to baselines at lower precisions. Furthermore LowRA results in substantial memory savings. This makes LowRA a promising solution for embedded systems and mobile devices.\n\n# Update after rebuttal\n\nAs mentioned in the comment below, I have decided not to update my scores.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer BPFc for their insightful feedback! Below, we address the comments from the **Weaknesses** section in detail. We will incorporate your valuable feedback into the revised manuscript.\n\n---\n\n## Weakness 1: Clarity on LowRA’s Components\n\n\n> **Reviewer Concern**: The paper would benefit from more explicit mapping between the three limitations tackled and the different components of the LowRA framework.\n\n**Our Response**: We will clarify how each component of the LowRA framework aligns with each of the three limitations. Specifically:\n\n1. **Limitation 1 (Coarse-Grained Precision Assignment)**  \n   - **P2: Precision Assigner**  \n   - **T3: Precs**\n\n2. **Limitation 2 (Discrepancy in Data Distribution)**  \n   - **P1: Mapping and Threshold Learner**  \n   - **T2: Fine-Grained Mappings and Thresholds**  \n   - **P4: Low-Rank Initializer** and **T5: Intelligently Initialized Low-Rank Tensors**  \n     (the low-rank initialization helps mitigate quantization errors)\n\n3. **Limitation 3 (Lack of High-Performance Quantization Primitives)**  \n   - **P3: Output-Channelwise Quantize Kernel**  \n   - **P5.1: Output-Channelwise Dequantize Kernel**\n\nWe will include both graphical annotations and supporting text in the paper to further illustrate these mappings.\n\n---\n## Weakness 2: Fine-Tuning with Data in Embedded Systems\n\n> **Reviewer Concern**: Many approaches favor data-free post-training for generalizability. However, in embedded systems, fine-tuning with data can be more practical. How can LowRA adapt to such scenarios?\n\n**Our Response**: We interpret your comment as referring to scenarios where there is some prior knowledge of the tasks for which the LoRA base weight will be used. In embedded systems, these base weights might be:\n\n\n### Case 1: Dedicated to a Single Task\n\n- **Using Task-Specific Data for Post-Training Quantization (PTQ).**  \n  If the base weights are used solely for one task, that task’s dataset can serve as a calibration set to learn precision assignments, thresholds, mappings, and rounding schemes.\n\n- **Quantization-Aware Training (QAT).**  \n  Another possibility is QAT. Preliminary experiments, however, showed that QAT did not outperform PTQ in terms of perplexity/accuracy—likely because LoRA’s low-rank adapters help compensate for quantization errors. Moreover, QAT demands more memory and introduces latency overhead.\n\n### Case 2: Shared Across Multiple Tasks\n\n- **Representative Calibration Set for PTQ.**  \n  If multiple tasks share certain characteristics, a representative calibration set may be synthesized for PTQ to learn shared precisions, thresholds, mappings, and rounding schemes.\n\n- **Possible QAT Approaches.**  \n  While still possible, QAT in multi-task scenarios faces the same drawbacks regarding memory and computational overhead. Moreover, the learned precisions, thresholds, mapping, and rounding schemes may overfit the calibration set and fail to generalize to each individual downstream task.\n\n### Other Scenarios and Future Directions\n\n- **Encoding vs. Decoding Schemes.**  \n  One strategy is to fix encoding (thresholds) globally and fine-tune only the decoding (mappings) per task. However, preliminary implementations showed significant latency overhead (e.g., scatter-add operations).\n\n- **Future Extensions.**  \n  We will continue exploring these ideas, focusing on deeper comparisons between PTQ and QAT under varying deployment constraints and use cases.\n\n\n---\n\n## Conclusion\n\nWe appreciate your feedback and have outlined our approach to making LowRA clearer and more applicable to embedded scenarios. Your comments have inspired further investigations and future directions, which we look forward to sharing in subsequent work.", "claims_and_evidence": "Yes, the claims made in this paper are supported through clear and convincing evidence. In particular the authors performed experiments using four LLMs and four tasks. Furthermore, the authors provided ablation studies of the different components in LowRA which is an important addition to the study.", "methods_and_evaluation_criteria": "Yes, the methods and evaluation criteria are ideal for the application in hand. The metrics used are clearly defined and are commonly used in this setting.", "theoretical_claims": "N/A. The paper does not make any theoretical claims.", "experimental_designs_or_analyses": "The main experimental results are highlighted in Table 2. I have the following comments/questions for the authors here:\n\n* LoftQ outperforms LowRA in terms of accuracy for 4 Bit LLaMA-2-7B in WikiText-2 and in terms of ROUGE1 for 4 Bit BART-large for XSUM. I might have missed this but am curious to know the authors thoughts in this regard.\n* 4Bit LLaMA-2-13B has the same performance for all quantized LoRA fine-tuning. I am curious regarding why this is the case?\n* The authors metnion PiSSA and ApiQ in the background section. Clear comparison against these methods (even in the supplementary section) would make the claims of this paper even stronger.", "relation_to_broader_scientific_literature": "The findings of this paper allows for LLMs to be adopted for systems with low memories such as embedded systems and mobile devices. This expands the use case of LLMs.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths\n* The problems of the current state of quantized LoRA fine tuning is clearly highlighted.\n* The LowRA framework is clearly defined\n* The ablation study is very helpful\n\nWeaknesses\n* It would help readability if the different components of the LowRA framework were explicitly mapped to the three limitations tackled in this paper.\n* In many cases authors use data-free post-training which helps with the generalizability of the approach. However, for embedded systems fine-tuning with data can be helpful in some cases. I would like to know the authors thoughts in this aspect.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A. I have already posed my questions in the previous sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: All three reviewers recommended to accept this paper in their reviews. After discussion, it was eventually agreed that this paper meets the ICML acceptance bar.
The authors are encouraged to address the comments of the reviewers to improve this work.
The author’s rebuttal has been carefully read and discussed. The author’s message has been carefully read and considered.
Decision: Accept (poster)